diff --git a/docker-compose.e2e.yml b/docker-compose.e2e.yml index e475873f..a03122c2 100644 --- a/docker-compose.e2e.yml +++ b/docker-compose.e2e.yml @@ -18,6 +18,12 @@ services: image: ped-ai-local:latest ports: - "127.0.0.1:3553:3000" + networks: + # Its own project network, plus the converter so PDF export is exercised + # here too. Without this the e2e stack could only reach Postgres and + # Redis, and a PDF download failed in a way production would not. + - default + - danvics_convert env_file: - .env environment: @@ -57,3 +63,7 @@ services: volumes: scribe-logs-e2e: + +networks: + danvics_convert: + external: true diff --git a/docs/CLINICAL_ASSISTANT.md b/docs/CLINICAL_ASSISTANT.md index e9923083..489c8f37 100644 --- a/docs/CLINICAL_ASSISTANT.md +++ b/docs/CLINICAL_ASSISTANT.md @@ -110,6 +110,9 @@ default in the right-hand column. | `clinical_assistant.image_model_roster` | Image models an admin added from Admin → Image Generation (**+ Add**). This is the pool the Image models tick-list offers; it is not itself an allowlist. Validated as up to 100 ids | | `clinical_assistant.search_limit` | Number of MCP results requested | | `clinical_assistant.context_chars` | Context characters requested from MCP | + +These are capped by `RERANKER_TOP_K` in the MCP deployment, which is the real +ceiling on every search. See [retrieval-tuning.md](retrieval-tuning.md). | `clinical_assistant.conversation_chars` | Input budget in UTF-16 code units. Empty means use `CLINICAL_ASSISTANT_CONVERSATION_CHARS`; a value must be 1000-1000000 | | `clinical_assistant.show_sources` | `true`/`false`. Display only: hides the Sources panel and the citation markers. The prompt, the retrieval and the stored answer are byte-for-byte identical either way, so it cannot bias an answer; turning it back on restores the citations | | `clinical_assistant.preview_enabled` | `true`/`false`. Lets signed-out visitors try the assistant read-only; anything needing an account asks them to sign in | diff --git a/docs/configuration.md b/docs/configuration.md index 5564d99e..d8e6583d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -198,3 +198,9 @@ OpenAI-compatible gateway — LiteLLM, Bifrost, or other proxies. | Bifrost | `provider/model` | Virtual keys, semantic caching, MCP gateway | | LiteLLM | Custom aliases | Requires PostgreSQL + Redis | | Any OpenAI-compatible | Varies | Must serve `/v1/chat/completions`, `/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1/embeddings` | + +## Retrieval sizing + +How many corpus excerpts the Clinical Assistant, the Learning Hub and My +Resources each receive, and the reranker cap that overrides all three: +[retrieval-tuning.md](retrieval-tuning.md). diff --git a/docs/learning-hub.md b/docs/learning-hub.md index a82afd29..253b22f7 100644 --- a/docs/learning-hub.md +++ b/docs/learning-hub.md @@ -85,3 +85,9 @@ available. | `learning_questions` | Quiz question prompts (FK to content) | | `learning_options` | Answer options (FK to question) | | `learning_progress` | Per-user attempt history | + +## Retrieval sizing + +How many corpus excerpts the Clinical Assistant, the Learning Hub and My +Resources each receive, and the reranker cap that overrides all three: +[retrieval-tuning.md](retrieval-tuning.md). diff --git a/docs/retrieval-tuning.md b/docs/retrieval-tuning.md new file mode 100644 index 00000000..fb1ca673 --- /dev/null +++ b/docs/retrieval-tuning.md @@ -0,0 +1,141 @@ +# Retrieval tuning — how many excerpts each feature gets + +Three features read from the same clinical corpus, and each takes a different +amount of it. This is where the numbers live and what actually changes them. + +Everything here is a Milvus collection called `mcp_bge_m3_1024`, embedded with +`openrouter-bge-m3` at 1024 dimensions, searched through the clinical MCP +(`clinical-assist-query`, deployed from `clinical-assist-deploy/`). There is one +corpus. Only the budgets differ. + +## The one number that caps everything + +`RERANKER_TOP_K`, in `clinical-assist-deploy/docker-compose.yml`. + +A search runs in two stages. Milvus returns a wide set of candidates by vector +similarity, then a reranker (`cohere-rerank-v4.0-pro`) scores each against the +query and keeps the best. `rerank_results()` takes `min(reranker_top_k, limit)`, +so **this value is the ceiling on every search, regardless of what the caller +asks for**. With it at 12, an app requesting 30 excerpts receives 12. + +This caused real confusion before it was written down: it was a library default +with no mention in any config file, so nothing explained where 12 came from. + +```yaml +# clinical-assist-deploy/docker-compose.yml — set on both mcp and mcp-indexer +- RERANKER_TOP_K=${RERANKER_TOP_K:-12} +- RERANKER_FETCH_MULTIPLIER=${RERANKER_FETCH_MULTIPLIER:-5} +``` + +To change it: + +```bash +cd /home/danvics/docker/clinical-assist-deploy +# either edit the default in docker-compose.yml, or set it in .env +echo 'RERANKER_TOP_K=20' >> .env +docker compose up -d mcp mcp-indexer +docker inspect mcp-server-mcp-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep RERANKER_TOP_K +``` + +`RERANKER_FETCH_MULTIPLIER` decides how many candidates the reranker sees: +`candidate_limit = max(limit, limit × multiplier)`. Raising it gives the +reranker more to choose from at the cost of a larger Milvus query and a larger +rerank call. 5 is the default and has not needed changing. + +### Is 12 enough? + +It is what the clinical assistant has always answered from, and 12 reranked +excerpts at 2500 characters is roughly 23,000 characters of closely matched +material — enough that a generated teaching resource reads with textbook +specificity (bilirubin production rates, conjugation timelines, thresholds in +mg/dL, all traceable to the indexed books). + +Raising it to 30 was tried and reverted. The reranker exists precisely to +discard near-misses; asking for more of what it already rejected adds length, +not signal. Raise it if a topic is genuinely broad and the output feels thin — +not by default. + +## Per-feature budgets + +These live in the `app_settings` table, are read live (2-minute cache), and are +clamped on read so a bad value cannot break a search. + +| Feature | Keys | Default | Clamp | +|---|---|---|---| +| Clinical Assistant | `clinical_assistant.search_limit`, `clinical_assistant.context_chars` | 8, 1400 | 3–20, 300–4000 | +| Learning Hub | `learning.search_limit`, `learning.context_chars` | 30, 2500 | 3–60, 300–8000 | +| My Resources | *the same `learning.*` keys* | 30, 2500 | 3–60, 300–8000 | + +`search_limit` is how many excerpts to request; `context_chars` is how much text +to pull around each one. + +**My Resources shares the Learning budget deliberately.** Both generate a whole +teaching resource from a topic, so they want the same shape of context. If they +ever need to diverge, `src/utils/learningRetrieval.js` is the single place that +reads these keys. + +Why the assistant is so much smaller: a chat answer is a paragraph and the +reader is waiting. A teaching resource synthesises an entire topic. Tuning one +must never move the other, which is why they are separate keys rather than one +shared pair. + +To change one: + +```sql +-- from the postgres container +INSERT INTO app_settings (key, value) VALUES ('learning.search_limit', '20') + ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value; +``` + +Remember the ceiling: setting `learning.search_limit` above `RERANKER_TOP_K` +changes nothing. Raise the reranker cap first. + +## Reading what actually happened + +The MCP logs every search and what survived reranking: + +```bash +docker logs mcp-server-mcp-1 --since 10m 2>&1 | grep -E "reranked search|before reranking|unverified" +# Milvus reranked search: user=..., limit=60, score_threshold=0.0, doc_type=file +# Milvus candidate retrieval returned 600 results before reranking +# Returning 12 unverified reranked results +``` + +Note `limit=60` for a request of 30: `semantic.py` asks the algorithm for +`limit × 2` and trims after verification. + +Generation responses carry the same fact, so a caller never has to guess whether +a resource was grounded: + +```json +"grounding": { "used": true, "count": 12, "reason": null } +``` + +`used: false` with a `reason` means the resource was written from the model +alone — retrieval never fails a generation, because ungrounded material is a far +better outcome than an error page. The Learning screen and My Resources both +show this, so ungrounded output is never presented as grounded. + +## A caution on raising these + +Context is not free and more is not automatically better. + +* The prompt has to fit the model's window. 12 excerpts at 2500 characters is + about 23k characters (~6k tokens); 30 at 2500 is about 57k (~14k). Overflow + does not error — it truncates, and truncation lands in the middle of the + excerpt block, which is the worst place to lose source material. If a resource + starts ignoring obvious material, lower `context_chars` before suspecting the + model. +* Every excerpt past the reranker's confident set is a near-miss. Ten strong + excerpts beat thirty mediocre ones for a model trying to write accurately. +* The reranker is billed per call and scales with candidates, not results. + `RERANKER_FETCH_MULTIPLIER` is the cost lever, not `RERANKER_TOP_K`. + +## Where each number is read + +| Number | Read by | File | +|---|---|---| +| `RERANKER_TOP_K` | clinical-assist | `clinical_assist/search/reranker.py` | +| `RERANKER_FETCH_MULTIPLIER` | clinical-assist | `clinical_assist/search/milvus_reranked.py` | +| `clinical_assistant.*` | ped-ai | `src/routes/clinicalAssistant.js` | +| `learning.*` | ped-ai | `src/utils/learningRetrieval.js` | diff --git a/public/components/my-resources.html b/public/components/my-resources.html new file mode 100644 index 00000000..1a681af2 --- /dev/null +++ b/public/components/my-resources.html @@ -0,0 +1,69 @@ +
+
+

My Resources

+ Private to you +
+
+

+ Teaching material you generate for yourself — a deck for tomorrow's session, a + handout, a summary. Nobody else sees these. Published Learning Hub content is separate + and stays with the moderators. +

+ +
+ + +
+ +
+ +
+ + + +
+
+ +
+ Clinical library +
+ +

+ Searches the corpus for your topic and writes from those excerpts, preferring them + over the model's own recall, and ends with a References section. Turn this off for a + topic the library does not cover. +

+
+
+ +
+ + +
+ +
+ + +
+
+
+ +
+
+

Library

+ +
+
+
diff --git a/public/index.html b/public/index.html index 85eedc46..4da90009 100644 --- a/public/index.html +++ b/public/index.html @@ -266,6 +266,10 @@ Diagrams +