pediatric-ai-scribe-v3/docs/retrieval-tuning.md

# Retrieval tuning — how many excerpts each feature gets

Three features read from the same clinical corpus, and each takes a different
amount of it. This is where the numbers live and what actually changes them.

Everything here is a Milvus collection called `mcp_bge_m3_1024`, embedded with
`openrouter-bge-m3` at 1024 dimensions, searched through the clinical MCP
(`clinical-assist-query`, deployed from `clinical-assist-deploy/`). There is one
corpus. Only the budgets differ.

## One knob per feature

Each feature decides how many excerpts it gets, in the `app_settings` table.
Nothing else overrides it.

That was not always true, and the history is worth knowing because the trap is
easy to recreate. A search runs in two stages: Milvus returns candidates by
vector similarity, then a reranker (`cohere-rerank-v4.0-pro`) scores each
against the query and keeps the best. The reranking step computes:

```python
top_k = min(settings.reranker_top_k or limit, limit)
```

`RERANKER_TOP_K` used to default to 12, so **a caller asking for 30 excerpts
received 12**, and nothing in any configuration file explained why. Changing one
number meant changing two settings in two repositories that had to agree — which
is not a design anyone chose, it is two codebases each assuming it owned the
decision.

It is now `0`, which the expression reads as "however many the caller asked
for". The app setting is the only knob.

```yaml
# clinical-assist-deploy/docker-compose.yml — on both mcp and mcp-indexer
- RERANKER_TOP_K=${RERANKER_TOP_K:-0}
- RERANKER_FETCH_MULTIPLIER=${RERANKER_FETCH_MULTIPLIER:-5}
```

Zero costs nothing extra. The reranker is billed on the documents **sent**,
which is `candidate_limit` and unchanged; `top_n` only decides how many come
back. Set `RERANKER_TOP_K` to a real number only if you want a hard ceiling
across every feature regardless of what each asks for.

`RERANKER_FETCH_MULTIPLIER` decides how many candidates the reranker sees:
`candidate_limit = max(limit, limit × multiplier)`. This is the cost lever.
5 is the default and has not needed changing.

## Per-feature budgets

These live in the `app_settings` table, are read live (2-minute cache), and are
clamped on read so a bad value cannot break a search.

| Feature | Keys | Default | Clamp |
|---|---|---|---|
| Clinical Assistant | `clinical_assistant.search_limit`, `clinical_assistant.context_chars` | 8, 1400 | 3–20, 0–4000 (0 = excerpt only) |
| My Resources | `learning.search_limit`, `learning.context_chars` | 30, 2500 | 3–60, 300–8000 |

The `learning.*` names are historical: they were the Learning Hub's, and My
Resources inherited the retrieval code when that was removed. Renaming the keys
would orphan whatever an administrator has already set, so they keep the old
names.

`search_limit` is how many excerpts to request; `context_chars` is how much text around each excerpt to add — read from the neighbouring chunks already stored in Milvus, never by re-reading the source document; 0 sends the excerpt alone. It is how much text
to pull around each one.

See [my-resources.md](my-resources.md) for the rest of that feature — its
sources, the deck renderer and illustrations.

**My Resources shares the Learning budget deliberately.** Both generate a whole
teaching resource from a topic, so they want the same shape of context. If they
ever need to diverge, `src/utils/learningRetrieval.js` is the single place that
reads these keys.

Why the assistant is so much smaller: a chat answer is a paragraph and the
reader is waiting. A teaching resource synthesises an entire topic. Tuning one
must never move the other, which is why they are separate keys rather than one
shared pair.

To change one:

```sql
-- from the postgres container
INSERT INTO app_settings (key, value) VALUES ('learning.search_limit', '20')
  ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;
```

That is the only change needed. `RERANKER_TOP_K` is 0, so the MCP honours
whatever the app asks for — verified: Learning requests 30 and receives 30, the
assistant requests 8 and receives 8.

## Reading what actually happened

The MCP logs every search and what survived reranking:

```bash
docker logs mcp-server-mcp-1 --since 10m 2>&1 | grep -E "reranked search|before reranking|unverified"
#   Milvus reranked search: user=..., limit=60, score_threshold=0.0, doc_type=file
#   Milvus candidate retrieval returned 600 results before reranking
#   Returning 12 unverified reranked results
```

Note `limit=60` for a request of 30: `semantic.py` asks the algorithm for
`limit × 2` and trims after verification.

Generation responses carry the same fact, so a caller never has to guess whether
a resource was grounded:

```json
"grounding": { "used": true, "count": 12, "reason": null }
```

`used: false` with a `reason` means the resource was written from the model
alone — retrieval never fails a generation, because ungrounded material is a far
better outcome than an error page. The Learning screen and My Resources both
show this, so ungrounded output is never presented as grounded.

## A caution on raising these

Context is not free and more is not automatically better.

* The prompt has to fit the model's window. 12 excerpts at 2500 characters is
  about 23k characters (~6k tokens); 30 at 2500 is about 57k (~14k). Overflow
  does not error — it truncates, and truncation lands in the middle of the
  excerpt block, which is the worst place to lose source material. If a resource
  starts ignoring obvious material, lower `context_chars` before suspecting the
  model.
* Every excerpt past the reranker's confident set is a near-miss. Ten strong
  excerpts beat thirty mediocre ones for a model trying to write accurately.
* The reranker is billed per call and scales with candidates, not results.
  `RERANKER_FETCH_MULTIPLIER` is the cost lever, not `RERANKER_TOP_K`.

## Where each number is read

| Number | Read by | File |
|---|---|---|
| `RERANKER_TOP_K` | clinical-assist | `clinical_assist/search/reranker.py` |
| `RERANKER_FETCH_MULTIPLIER` | clinical-assist | `clinical_assist/search/milvus_reranked.py` |
| `clinical_assistant.*` | ped-ai | `src/routes/clinicalAssistant.js` |
| `learning.*` | ped-ai | `src/utils/learningRetrieval.js` |