pediatric-ai-scribe-v3/docs/retrieval-tuning.md

# Retrieval tuning — how many excerpts each feature gets

Three features read from the same clinical corpus, and each takes a different
amount of it. This is where the numbers live and what actually changes them.

Everything here is a Milvus collection called `mcp_bge_m3_1024`, embedded with
`openrouter-bge-m3` at 1024 dimensions, searched through the clinical MCP
(`clinical-assist-query`, deployed from `clinical-assist-deploy/`). There is one
corpus. Only the budgets differ.

## The one number that caps everything

`RERANKER_TOP_K`, in `clinical-assist-deploy/docker-compose.yml`.

A search runs in two stages. Milvus returns a wide set of candidates by vector
similarity, then a reranker (`cohere-rerank-v4.0-pro`) scores each against the
query and keeps the best. `rerank_results()` takes `min(reranker_top_k, limit)`,
so **this value is the ceiling on every search, regardless of what the caller
asks for**. With it at 12, an app requesting 30 excerpts receives 12.

This caused real confusion before it was written down: it was a library default
with no mention in any config file, so nothing explained where 12 came from.

```yaml
# clinical-assist-deploy/docker-compose.yml — set on both mcp and mcp-indexer
- RERANKER_TOP_K=${RERANKER_TOP_K:-12}
- RERANKER_FETCH_MULTIPLIER=${RERANKER_FETCH_MULTIPLIER:-5}
```

To change it:

```bash
cd /home/danvics/docker/clinical-assist-deploy
# either edit the default in docker-compose.yml, or set it in .env
echo 'RERANKER_TOP_K=20' >> .env
docker compose up -d mcp mcp-indexer
docker inspect mcp-server-mcp-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep RERANKER_TOP_K
```

`RERANKER_FETCH_MULTIPLIER` decides how many candidates the reranker sees:
`candidate_limit = max(limit, limit × multiplier)`. Raising it gives the
reranker more to choose from at the cost of a larger Milvus query and a larger
rerank call. 5 is the default and has not needed changing.

### Is 12 enough?

It is what the clinical assistant has always answered from, and 12 reranked
excerpts at 2500 characters is roughly 23,000 characters of closely matched
material — enough that a generated teaching resource reads with textbook
specificity (bilirubin production rates, conjugation timelines, thresholds in
mg/dL, all traceable to the indexed books).

Raising it to 30 was tried and reverted. The reranker exists precisely to
discard near-misses; asking for more of what it already rejected adds length,
not signal. Raise it if a topic is genuinely broad and the output feels thin —
not by default.

## Per-feature budgets

These live in the `app_settings` table, are read live (2-minute cache), and are
clamped on read so a bad value cannot break a search.

| Feature | Keys | Default | Clamp |
|---|---|---|---|
| Clinical Assistant | `clinical_assistant.search_limit`, `clinical_assistant.context_chars` | 8, 1400 | 3–20, 300–4000 |
| Learning Hub | `learning.search_limit`, `learning.context_chars` | 30, 2500 | 3–60, 300–8000 |
| My Resources | *the same `learning.*` keys* | 30, 2500 | 3–60, 300–8000 |

`search_limit` is how many excerpts to request; `context_chars` is how much text
to pull around each one.

**My Resources shares the Learning budget deliberately.** Both generate a whole
teaching resource from a topic, so they want the same shape of context. If they
ever need to diverge, `src/utils/learningRetrieval.js` is the single place that
reads these keys.

Why the assistant is so much smaller: a chat answer is a paragraph and the
reader is waiting. A teaching resource synthesises an entire topic. Tuning one
must never move the other, which is why they are separate keys rather than one
shared pair.

To change one:

```sql
-- from the postgres container
INSERT INTO app_settings (key, value) VALUES ('learning.search_limit', '20')
  ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;
```

Remember the ceiling: setting `learning.search_limit` above `RERANKER_TOP_K`
changes nothing. Raise the reranker cap first.

## Reading what actually happened

The MCP logs every search and what survived reranking:

```bash
docker logs mcp-server-mcp-1 --since 10m 2>&1 | grep -E "reranked search|before reranking|unverified"
#   Milvus reranked search: user=..., limit=60, score_threshold=0.0, doc_type=file
#   Milvus candidate retrieval returned 600 results before reranking
#   Returning 12 unverified reranked results
```

Note `limit=60` for a request of 30: `semantic.py` asks the algorithm for
`limit × 2` and trims after verification.

Generation responses carry the same fact, so a caller never has to guess whether
a resource was grounded:

```json
"grounding": { "used": true, "count": 12, "reason": null }
```

`used: false` with a `reason` means the resource was written from the model
alone — retrieval never fails a generation, because ungrounded material is a far
better outcome than an error page. The Learning screen and My Resources both
show this, so ungrounded output is never presented as grounded.

## A caution on raising these

Context is not free and more is not automatically better.

* The prompt has to fit the model's window. 12 excerpts at 2500 characters is
  about 23k characters (~6k tokens); 30 at 2500 is about 57k (~14k). Overflow
  does not error — it truncates, and truncation lands in the middle of the
  excerpt block, which is the worst place to lose source material. If a resource
  starts ignoring obvious material, lower `context_chars` before suspecting the
  model.
* Every excerpt past the reranker's confident set is a near-miss. Ten strong
  excerpts beat thirty mediocre ones for a model trying to write accurately.
* The reranker is billed per call and scales with candidates, not results.
  `RERANKER_FETCH_MULTIPLIER` is the cost lever, not `RERANKER_TOP_K`.

## Where each number is read

| Number | Read by | File |
|---|---|---|
| `RERANKER_TOP_K` | clinical-assist | `clinical_assist/search/reranker.py` |
| `RERANKER_FETCH_MULTIPLIER` | clinical-assist | `clinical_assist/search/milvus_reranked.py` |
| `clinical_assistant.*` | ped-ai | `src/routes/clinicalAssistant.js` |
| `learning.*` | ped-ai | `src/utils/learningRetrieval.js` |