pediatric-ai-scribe-v3/docs/retrieval-tuning.md

Retrieval tuning — how many excerpts each feature gets

Three features read from the same clinical corpus, and each takes a different amount of it. This is where the numbers live and what actually changes them.

Everything here is a Milvus collection called mcp_bge_m3_1024, embedded with openrouter-bge-m3 at 1024 dimensions, searched through the clinical MCP (clinical-assist-query, deployed from clinical-assist-deploy/). There is one corpus. Only the budgets differ.

The one number that caps everything

RERANKER_TOP_K, in clinical-assist-deploy/docker-compose.yml.

A search runs in two stages. Milvus returns a wide set of candidates by vector similarity, then a reranker (cohere-rerank-v4.0-pro) scores each against the query and keeps the best. rerank_results() takes min(reranker_top_k, limit), so this value is the ceiling on every search, regardless of what the caller asks for. With it at 12, an app requesting 30 excerpts receives 12.

This caused real confusion before it was written down: it was a library default with no mention in any config file, so nothing explained where 12 came from.

# clinical-assist-deploy/docker-compose.yml — set on both mcp and mcp-indexer
- RERANKER_TOP_K=${RERANKER_TOP_K:-12}
- RERANKER_FETCH_MULTIPLIER=${RERANKER_FETCH_MULTIPLIER:-5}

To change it:

cd /home/danvics/docker/clinical-assist-deploy
# either edit the default in docker-compose.yml, or set it in .env
echo 'RERANKER_TOP_K=20' >> .env
docker compose up -d mcp mcp-indexer
docker inspect mcp-server-mcp-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep RERANKER_TOP_K

RERANKER_FETCH_MULTIPLIER decides how many candidates the reranker sees: candidate_limit = max(limit, limit × multiplier). Raising it gives the reranker more to choose from at the cost of a larger Milvus query and a larger rerank call. 5 is the default and has not needed changing.

Is 12 enough?

It is what the clinical assistant has always answered from, and 12 reranked excerpts at 2500 characters is roughly 23,000 characters of closely matched material — enough that a generated teaching resource reads with textbook specificity (bilirubin production rates, conjugation timelines, thresholds in mg/dL, all traceable to the indexed books).

Raising it to 30 was tried and reverted. The reranker exists precisely to discard near-misses; asking for more of what it already rejected adds length, not signal. Raise it if a topic is genuinely broad and the output feels thin — not by default.

Per-feature budgets

These live in the app_settings table, are read live (2-minute cache), and are clamped on read so a bad value cannot break a search.

Feature	Keys	Default	Clamp
Clinical Assistant	clinical_assistant.search_limit, clinical_assistant.context_chars	8, 1400	3–20, 300–4000
Learning Hub	learning.search_limit, learning.context_chars	30, 2500	3–60, 300–8000
My Resources	the same learning.* keys	30, 2500	3–60, 300–8000

search_limit is how many excerpts to request; context_chars is how much text to pull around each one.

My Resources shares the Learning budget deliberately. Both generate a whole teaching resource from a topic, so they want the same shape of context. If they ever need to diverge, src/utils/learningRetrieval.js is the single place that reads these keys.

Why the assistant is so much smaller: a chat answer is a paragraph and the reader is waiting. A teaching resource synthesises an entire topic. Tuning one must never move the other, which is why they are separate keys rather than one shared pair.

To change one:

-- from the postgres container
INSERT INTO app_settings (key, value) VALUES ('learning.search_limit', '20')
  ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;

Remember the ceiling: setting learning.search_limit above RERANKER_TOP_K changes nothing. Raise the reranker cap first.

Reading what actually happened

The MCP logs every search and what survived reranking:

docker logs mcp-server-mcp-1 --since 10m 2>&1 | grep -E "reranked search|before reranking|unverified"
#   Milvus reranked search: user=..., limit=60, score_threshold=0.0, doc_type=file
#   Milvus candidate retrieval returned 600 results before reranking
#   Returning 12 unverified reranked results

Note limit=60 for a request of 30: semantic.py asks the algorithm for limit × 2 and trims after verification.

Generation responses carry the same fact, so a caller never has to guess whether a resource was grounded:

"grounding": { "used": true, "count": 12, "reason": null }

used: false with a reason means the resource was written from the model alone — retrieval never fails a generation, because ungrounded material is a far better outcome than an error page. The Learning screen and My Resources both show this, so ungrounded output is never presented as grounded.

A caution on raising these

Context is not free and more is not automatically better.

• The prompt has to fit the model's window. 12 excerpts at 2500 characters is about 23k characters (~6k tokens); 30 at 2500 is about 57k (~14k). Overflow does not error — it truncates, and truncation lands in the middle of the excerpt block, which is the worst place to lose source material. If a resource starts ignoring obvious material, lower context_chars before suspecting the model.
• Every excerpt past the reranker's confident set is a near-miss. Ten strong excerpts beat thirty mediocre ones for a model trying to write accurately.
• The reranker is billed per call and scales with candidates, not results. RERANKER_FETCH_MULTIPLIER is the cost lever, not RERANKER_TOP_K.

Where each number is read

Number	Read by	File
RERANKER_TOP_K	clinical-assist	clinical_assist/search/reranker.py
RERANKER_FETCH_MULTIPLIER	clinical-assist	clinical_assist/search/milvus_reranked.py
clinical_assistant.*	ped-ai	src/routes/clinicalAssistant.js
learning.*	ped-ai	src/utils/learningRetrieval.js