pediatric-ai-scribe-v3/docs/retrieval-tuning.md

# Retrieval tuning — how many excerpts each feature gets

Three features read from the same clinical corpus, and each takes a different
amount of it. This is where the numbers live and what actually changes them.

Everything here is a Milvus collection called `mcp_bge_m3_1024`, embedded with
`openrouter-bge-m3` at 1024 dimensions, searched through the clinical MCP
(`clinical-assist-query`, deployed from `clinical-assist-deploy/`). There is one
corpus. Only the budgets differ.

## One knob per feature

Each feature decides how many excerpts it gets, in the `app_settings` table.
Nothing else overrides it.

That was not always true, and the history is worth knowing because the trap is
easy to recreate. A search runs in two stages: Milvus returns candidates by
vector similarity, then a reranker (`cohere-rerank-v4.0-pro`) scores each
against the query and keeps the best. The reranking step computes:

```python
top_k = min(settings.reranker_top_k or limit, limit)
```

`RERANKER_TOP_K` used to default to 12, so **a caller asking for 30 excerpts
received 12**, and nothing in any configuration file explained why. Changing one
number meant changing two settings in two repositories that had to agree — which
is not a design anyone chose, it is two codebases each assuming it owned the
decision.

It is now `0`, which the expression reads as "however many the caller asked
for". The app setting is the only knob.

```yaml
# clinical-assist-deploy/docker-compose.yml — on both mcp and mcp-indexer
- RERANKER_TOP_K=${RERANKER_TOP_K:-0}
- RERANKER_FETCH_MULTIPLIER=${RERANKER_FETCH_MULTIPLIER:-5}
```

Zero costs nothing extra. The reranker is billed on the documents **sent**,
which is `candidate_limit` and unchanged; `top_n` only decides how many come
back. Set `RERANKER_TOP_K` to a real number only if you want a hard ceiling
across every feature regardless of what each asks for.

`RERANKER_FETCH_MULTIPLIER` decides how many candidates the reranker sees:
`candidate_limit = max(limit, limit × multiplier)`. This is the cost lever.
5 is the default and has not needed changing.

## Per-feature budgets

These live in the `app_settings` table, are read live (2-minute cache), and are
clamped on read so a bad value cannot break a search.

| Feature | Keys | Default | Clamp |
|---|---|---|---|
| Clinical Assistant | `clinical_assistant.search_limit`, `clinical_assistant.context_chars` | 8, 1400 | 3–20, 300–4000 |
| Learning Hub | `learning.search_limit`, `learning.context_chars` | 30, 2500 | 3–60, 300–8000 |
| My Resources | *the same `learning.*` keys* | 30, 2500 | 3–60, 300–8000 |

`search_limit` is how many excerpts to request; `context_chars` is how much text
to pull around each one.

**My Resources shares the Learning budget deliberately.** Both generate a whole
teaching resource from a topic, so they want the same shape of context. If they
ever need to diverge, `src/utils/learningRetrieval.js` is the single place that
reads these keys.

Why the assistant is so much smaller: a chat answer is a paragraph and the
reader is waiting. A teaching resource synthesises an entire topic. Tuning one
must never move the other, which is why they are separate keys rather than one
shared pair.

To change one:

```sql
-- from the postgres container
INSERT INTO app_settings (key, value) VALUES ('learning.search_limit', '20')
  ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;
```

That is the only change needed. `RERANKER_TOP_K` is 0, so the MCP honours
whatever the app asks for — verified: Learning requests 30 and receives 30, the
assistant requests 8 and receives 8.

## Reading what actually happened

The MCP logs every search and what survived reranking:

```bash
docker logs mcp-server-mcp-1 --since 10m 2>&1 | grep -E "reranked search|before reranking|unverified"
#   Milvus reranked search: user=..., limit=60, score_threshold=0.0, doc_type=file
#   Milvus candidate retrieval returned 600 results before reranking
#   Returning 12 unverified reranked results
```

Note `limit=60` for a request of 30: `semantic.py` asks the algorithm for
`limit × 2` and trims after verification.

Generation responses carry the same fact, so a caller never has to guess whether
a resource was grounded:

```json
"grounding": { "used": true, "count": 12, "reason": null }
```

`used: false` with a `reason` means the resource was written from the model
alone — retrieval never fails a generation, because ungrounded material is a far
better outcome than an error page. The Learning screen and My Resources both
show this, so ungrounded output is never presented as grounded.

## A caution on raising these

Context is not free and more is not automatically better.

* The prompt has to fit the model's window. 12 excerpts at 2500 characters is
  about 23k characters (~6k tokens); 30 at 2500 is about 57k (~14k). Overflow
  does not error — it truncates, and truncation lands in the middle of the
  excerpt block, which is the worst place to lose source material. If a resource
  starts ignoring obvious material, lower `context_chars` before suspecting the
  model.
* Every excerpt past the reranker's confident set is a near-miss. Ten strong
  excerpts beat thirty mediocre ones for a model trying to write accurately.
* The reranker is billed per call and scales with candidates, not results.
  `RERANKER_FETCH_MULTIPLIER` is the cost lever, not `RERANKER_TOP_K`.

## Where each number is read

| Number | Read by | File |
|---|---|---|
| `RERANKER_TOP_K` | clinical-assist | `clinical_assist/search/reranker.py` |
| `RERANKER_FETCH_MULTIPLIER` | clinical-assist | `clinical_assist/search/milvus_reranked.py` |
| `clinical_assistant.*` | ped-ai | `src/routes/clinicalAssistant.js` |
| `learning.*` | ped-ai | `src/utils/learningRetrieval.js` |