pediatric-ai-scribe-v3/docs/retrieval-tuning.md

Retrieval tuning — how many excerpts each feature gets

Three features read from the same clinical corpus, and each takes a different amount of it. This is where the numbers live and what actually changes them.

Everything here is a Milvus collection called mcp_bge_m3_1024, embedded with openrouter-bge-m3 at 1024 dimensions, searched through the clinical MCP (clinical-assist-query, deployed from clinical-assist-deploy/). There is one corpus. Only the budgets differ.

One knob per feature

Each feature decides how many excerpts it gets, in the app_settings table. Nothing else overrides it.

That was not always true, and the history is worth knowing because the trap is easy to recreate. A search runs in two stages: Milvus returns candidates by vector similarity, then a reranker (cohere-rerank-v4.0-pro) scores each against the query and keeps the best. The reranking step computes:

top_k = min(settings.reranker_top_k or limit, limit)

RERANKER_TOP_K used to default to 12, so a caller asking for 30 excerpts received 12, and nothing in any configuration file explained why. Changing one number meant changing two settings in two repositories that had to agree — which is not a design anyone chose, it is two codebases each assuming it owned the decision.

It is now 0, which the expression reads as "however many the caller asked for". The app setting is the only knob.

# clinical-assist-deploy/docker-compose.yml — on both mcp and mcp-indexer
- RERANKER_TOP_K=${RERANKER_TOP_K:-0}
- RERANKER_FETCH_MULTIPLIER=${RERANKER_FETCH_MULTIPLIER:-5}

Zero costs nothing extra. The reranker is billed on the documents sent, which is candidate_limit and unchanged; top_n only decides how many come back. Set RERANKER_TOP_K to a real number only if you want a hard ceiling across every feature regardless of what each asks for.

RERANKER_FETCH_MULTIPLIER decides how many candidates the reranker sees: candidate_limit = max(limit, limit × multiplier). This is the cost lever. 5 is the default and has not needed changing.

Per-feature budgets

These live in the app_settings table, are read live (2-minute cache), and are clamped on read so a bad value cannot break a search.

Feature	Keys	Default	Clamp
Clinical Assistant	clinical_assistant.search_limit, clinical_assistant.context_chars	8, 0	3–20, 0–4000 (0 = excerpt only)
My Resources	learning.search_limit, learning.context_chars	30, 2500	3–60, 300–8000

The learning.* names are historical: they were the Learning Hub's, and My Resources inherited the retrieval code when that was removed. Renaming the keys would orphan whatever an administrator has already set, so they keep the old names.

search_limit is how many excerpts to request; context_chars is how much extra text around each excerpt to fetch from the source at query time — 0, the default, sends the stored excerpt as it is, which already carries the page, its tables and figure captions; any other value makes the search service download and re-extract the source document per hit (about a second each, cached for 15 minutes). It is how much text to pull around each one.

See my-resources.md for the rest of that feature — its sources, the deck renderer and illustrations.

My Resources shares the Learning budget deliberately. Both generate a whole teaching resource from a topic, so they want the same shape of context. If they ever need to diverge, src/utils/learningRetrieval.js is the single place that reads these keys.

Why the assistant is so much smaller: a chat answer is a paragraph and the reader is waiting. A teaching resource synthesises an entire topic. Tuning one must never move the other, which is why they are separate keys rather than one shared pair.

To change one:

-- from the postgres container
INSERT INTO app_settings (key, value) VALUES ('learning.search_limit', '20')
  ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;

That is the only change needed. RERANKER_TOP_K is 0, so the MCP honours whatever the app asks for — verified: Learning requests 30 and receives 30, the assistant requests 8 and receives 8.

Reading what actually happened

The MCP logs every search and what survived reranking:

docker logs mcp-server-mcp-1 --since 10m 2>&1 | grep -E "reranked search|before reranking|unverified"
#   Milvus reranked search: user=..., limit=60, score_threshold=0.0, doc_type=file
#   Milvus candidate retrieval returned 600 results before reranking
#   Returning 12 unverified reranked results

Note limit=60 for a request of 30: semantic.py asks the algorithm for limit × 2 and trims after verification.

Generation responses carry the same fact, so a caller never has to guess whether a resource was grounded:

"grounding": { "used": true, "count": 12, "reason": null }

used: false with a reason means the resource was written from the model alone — retrieval never fails a generation, because ungrounded material is a far better outcome than an error page. The Learning screen and My Resources both show this, so ungrounded output is never presented as grounded.

A caution on raising these

Context is not free and more is not automatically better.

• The prompt has to fit the model's window. 12 excerpts at 2500 characters is about 23k characters (~6k tokens); 30 at 2500 is about 57k (~14k). Overflow does not error — it truncates, and truncation lands in the middle of the excerpt block, which is the worst place to lose source material. If a resource starts ignoring obvious material, lower context_chars before suspecting the model.
• Every excerpt past the reranker's confident set is a near-miss. Ten strong excerpts beat thirty mediocre ones for a model trying to write accurately.
• The reranker is billed per call and scales with candidates, not results. RERANKER_FETCH_MULTIPLIER is the cost lever, not RERANKER_TOP_K.

Where each number is read

Number	Read by	File
RERANKER_TOP_K	clinical-assist	clinical_assist/search/reranker.py
RERANKER_FETCH_MULTIPLIER	clinical-assist	clinical_assist/search/milvus_reranked.py
clinical_assistant.*	ped-ai	src/routes/clinicalAssistant.js
learning.*	ped-ai	src/utils/learningRetrieval.js