diff --git a/docs/retrieval-tuning.md b/docs/retrieval-tuning.md index fb1ca673..9799e269 100644 --- a/docs/retrieval-tuning.md +++ b/docs/retrieval-tuning.md @@ -8,52 +8,43 @@ Everything here is a Milvus collection called `mcp_bge_m3_1024`, embedded with (`clinical-assist-query`, deployed from `clinical-assist-deploy/`). There is one corpus. Only the budgets differ. -## The one number that caps everything +## One knob per feature -`RERANKER_TOP_K`, in `clinical-assist-deploy/docker-compose.yml`. +Each feature decides how many excerpts it gets, in the `app_settings` table. +Nothing else overrides it. -A search runs in two stages. Milvus returns a wide set of candidates by vector -similarity, then a reranker (`cohere-rerank-v4.0-pro`) scores each against the -query and keeps the best. `rerank_results()` takes `min(reranker_top_k, limit)`, -so **this value is the ceiling on every search, regardless of what the caller -asks for**. With it at 12, an app requesting 30 excerpts receives 12. +That was not always true, and the history is worth knowing because the trap is +easy to recreate. A search runs in two stages: Milvus returns candidates by +vector similarity, then a reranker (`cohere-rerank-v4.0-pro`) scores each +against the query and keeps the best. The reranking step computes: -This caused real confusion before it was written down: it was a library default -with no mention in any config file, so nothing explained where 12 came from. +```python +top_k = min(settings.reranker_top_k or limit, limit) +``` + +`RERANKER_TOP_K` used to default to 12, so **a caller asking for 30 excerpts +received 12**, and nothing in any configuration file explained why. Changing one +number meant changing two settings in two repositories that had to agree — which +is not a design anyone chose, it is two codebases each assuming it owned the +decision. + +It is now `0`, which the expression reads as "however many the caller asked +for". The app setting is the only knob. ```yaml -# clinical-assist-deploy/docker-compose.yml — set on both mcp and mcp-indexer -- RERANKER_TOP_K=${RERANKER_TOP_K:-12} +# clinical-assist-deploy/docker-compose.yml — on both mcp and mcp-indexer +- RERANKER_TOP_K=${RERANKER_TOP_K:-0} - RERANKER_FETCH_MULTIPLIER=${RERANKER_FETCH_MULTIPLIER:-5} ``` -To change it: - -```bash -cd /home/danvics/docker/clinical-assist-deploy -# either edit the default in docker-compose.yml, or set it in .env -echo 'RERANKER_TOP_K=20' >> .env -docker compose up -d mcp mcp-indexer -docker inspect mcp-server-mcp-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep RERANKER_TOP_K -``` +Zero costs nothing extra. The reranker is billed on the documents **sent**, +which is `candidate_limit` and unchanged; `top_n` only decides how many come +back. Set `RERANKER_TOP_K` to a real number only if you want a hard ceiling +across every feature regardless of what each asks for. `RERANKER_FETCH_MULTIPLIER` decides how many candidates the reranker sees: -`candidate_limit = max(limit, limit × multiplier)`. Raising it gives the -reranker more to choose from at the cost of a larger Milvus query and a larger -rerank call. 5 is the default and has not needed changing. - -### Is 12 enough? - -It is what the clinical assistant has always answered from, and 12 reranked -excerpts at 2500 characters is roughly 23,000 characters of closely matched -material — enough that a generated teaching resource reads with textbook -specificity (bilirubin production rates, conjugation timelines, thresholds in -mg/dL, all traceable to the indexed books). - -Raising it to 30 was tried and reverted. The reranker exists precisely to -discard near-misses; asking for more of what it already rejected adds length, -not signal. Raise it if a topic is genuinely broad and the output feels thin — -not by default. +`candidate_limit = max(limit, limit × multiplier)`. This is the cost lever. +5 is the default and has not needed changing. ## Per-feature budgets @@ -87,8 +78,9 @@ INSERT INTO app_settings (key, value) VALUES ('learning.search_limit', '20') ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value; ``` -Remember the ceiling: setting `learning.search_limit` above `RERANKER_TOP_K` -changes nothing. Raise the reranker cap first. +That is the only change needed. `RERANKER_TOP_K` is 0, so the MCP honours +whatever the app asks for — verified: Learning requests 30 and receives 30, the +assistant requests 8 and receives 8. ## Reading what actually happened