031de53034
Retrieval fused a bi-encoder and BM25 by reciprocal rank. A bi-encoder embeds a document long before the question exists, so the two never meet: it is good at "same topic" and mediocre at "answers this". A cross-encoder reads the pair. The proxy already serves three — `cohere-rerank-v4.0-pro` is the default and measurably better than the fast variant. Query text goes exactly where the embeddings already go, and nothing new was signed up for. It found a defect nobody was looking for. In AI Mode each finder scored `1/(1+rank)` *within its own corpus*, so the best article, section, question and card all scored 1.0 and the shortlist was a meaningless round-robin. A cross-encoder is the first thing in this system that can compare a question with a section. Candidates per kind widened so it can select rather than merely reorder. Measured against labels neither ranker produced. Questions, 60 disease tags: precision@3 0.394 → 0.483. Sections, 60 article titles: 0.772 → 0.833. "Management of bronchiolitis" led with influenza transmission and a pregnancy question; "when do you image a first febrile seizure" returned the definition rather than the sentence saying imaging is unnecessary. And the honest negative, in docs/reranking.md: board vignettes are written *not* to name their diagnosis, so on "what causes croup" it prefers a question that says the word in passing over the barking-cough vignette that never says it. Some of the bi-encoder's strength is traded away. Not on the typeahead. A page of results is a choice being made and worth a third of a second; a typeahead is a word being finished, runs on every keystroke, and has nothing to judge yet. The three-state thresholds stay on cosine, argued at the constant: a reranker only ever sees a shortlist and structurally cannot answer the corpus-wide question those numbers ask, and whether an answer claims to come from the library is a promise that must not depend on a network hop. Every failure returns None and leaves the order alone — unconfigured, no proxy, connect error, bare 502, timeout, non-JSON, a duplicate or out-of-range index, a non-numeric score, a list the wrong length. Verified against the running site with a bogus model name: same results, fused order, no error to the reader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
10 KiB
Reranking: the second opinion on order
Retrieval fuses two rankers that never see the query and the document together.
A bi-encoder embedded every row long before anybody typed anything, so cosine
distance answers "is this the same topic"; ts_rank_cd answers "do these words
coincide". Neither answers "does this document answer this question", and that
is the question a learner is actually asking.
A cross-encoder answers it, at the cost of one forward pass per pair, per query, which is why it can only ever be given a shortlist the cheap rankers produced. It reorders. It never decides what exists.
What is served, and where the query goes
The proxy at LITELLM_API_BASE already serves three rerankers — /model/info
reports "mode": "rerank" for each:
| model | notes |
|---|---|
cohere-rerank-v4.0-pro |
in use — the best of the three here, see the numbers below |
cohere-rerank-v4.0-fast |
roughly 150 ms quicker, and measurably worse on this corpus |
jina-reranker-v2-base-multilingual |
open-weight, but rate-limited to the point of unusability on this proxy: a 60-query sweep got 429s on nearly every call |
Both POST /rerank and POST /v1/rerank answer, with the body Cohere defined —
{model, query, documents, top_n} in, {results: [{index, relevance_score}]}
out. Jina, BGE and LiteLLM's own rerank route all speak it, so replacing the
model behind LITELLM_RERANK_MODEL with a self-hosted open-weight one is a
configuration change and nothing else.
Where it is applied
| place | reranked | why |
|---|---|---|
AI Mode shortlist (ai_mode_service.retrieve) |
yes | The shortlist is the evidence the answer may use, and the turn is already waiting on a chat model. It is also the only place where four corpora have to be compared with each other — see below. |
Question search (/questions?q=, /search) |
yes | A page of results is a choice being made. |
Article search (/articles?q=, /search) |
yes, through its sections | An article row is a title, a summary and a topical vector; the prose is in its sections. Handing a cross-encoder the article row is handing it a stub and asking about a document it cannot see. Its best section already decides its place in the fusion, so reranking the sections reranks the articles. |
| Test builder from a description | yes | A sentence about what somebody wants to study is exactly the query shape a cross-encoder is trained on, and the top of that list becomes the test rather than a page to scroll past. |
| Test builder from an uploaded file | no | The "query" is a whole handout. Clipped to 2000 characters it judges every candidate against whichever part that happened to be — confidently, and about the wrong thing. |
| Flashcard and media search | no | Short fields (front/back, title/caption) where the lexical ranker is already doing the whole job, and both are small personal collections. Latency for nothing. |
Typeahead (/search/suggest) |
no | It runs on every keystroke and has half a word to judge. It is prefix matching on titles, which is what finishing a word wants. Adding a 400 ms round trip per character would make the fastest thing on the site the slowest. |
Latency
Measured from the backend container against cohere-rerank-v4.0-pro, median of
three, real question stems as documents:
| candidates | round trip |
|---|---|
| 10 | 250 ms |
| 30 | 380 ms |
| 50 | 430 ms |
| 100 | 580 ms |
Mostly fixed cost, then roughly 2 ms a pair. MAX_CANDIDATES = 50: the fused
pool is 60 wide and a page shows ten, so past fifty the model is being paid to
confirm that rows neither cheap ranker wanted are still not wanted.
In place, including reading the candidate text out of Postgres, over 120 live queries never asked before: median 480 ms, p90 550 ms. AI Mode's whole retrieval step goes from about 85 ms to about 500 ms — one call over roughly forty candidates across all four corpora.
Two caches sit in front of that, and both matter more than the raw figure. Ours is Redis, keyed on model, query and the document text, for a day: 3–5 ms on a hit, which is most repeat queries and every reload of a results page. The proxy keeps its own cache of identical rerank requests and answers those in about 30 ms, so a query one learner has already asked is cheap for the next even after our key expires. A full four-corpus search page measured 94 ms warm, 236 ms with our cache cleared but the proxy's still warm, and about a second on a genuinely new query, which is two corpora reranked one after the other. Running the per-corpus finders concurrently would recover half of that and has not been done.
Does it help
Two measurements, both against labels neither ranker produced.
Questions, query = a disease tag's name, relevant = the questions carrying that tag, 60 tags sampled at random from those with 3–40 questions:
| precision@3 | MRR@10 | |
|---|---|---|
| fused | 0.394 | 0.595 |
+ cohere-rerank-v4.0-pro |
0.483 | 0.641 |
+ cohere-rerank-v4.0-fast |
0.439 | 0.642 |
Top result changed on 38 of 60; precision@3 improved on 21 and fell on 9.
Article sections, query = an article's title, relevant = that article's sections, 60 articles with three or more sections:
| precision@3 | MRR@10 | |
|---|---|---|
| fused | 0.772 | 0.847 |
+ cohere-rerank-v4.0-pro |
0.833 | 0.908 |
+ cohere-rerank-v4.0-fast |
0.811 | 0.917 |
Top result changed on 41 of 60; improved on 12, fell on 4.
Queries where the old top result was wrong
- "management of bronchiolitis in an infant" — fused first three sections were influenza transmission, foreign body ingestion, and the management of a pregnancy. Reranked: three supportive-management sections, the third of them bronchiolitis's own and the first two transient tachypnoea of the newborn. Better, and not right: on this corpus the reranker will take a Management section from a neighbouring topic over a differently-titled one from the right topic.
- "when do you image a first febrile seizure" — fused returned the definition of a febrile seizure. Reranked returned "routine laboratory studies and acute neuroimaging are unnecessary…", which is the answer.
- "teenager with knee pain worse after sport" — fused led with a 6-year-old girl's case and, in the articles, with bone pain that wakes a child at night. Reranked led with a 15-year-old's sports knee pain, and with patellofemoral pain syndrome.
- "delayed passage of meconium" — fused returned functional constipation. Reranked returned "obstruction presents with failure to pass meconium within 24–48 hours of birth".
- "child limping with a fever and refusing to bear weight" — fused led with a urinary tract infection's presentation and acute gastroenteritis. Reranked led with osteomyelitis, septic arthritis, and the differential for a limping toddler.
- "why does my patient with milk in the bottle have anemia" (AI Mode) — fused led with the general anemia article; reranked led with iron deficiency anemia.
Where it does not help, and one place it is worse
The gain is uneven across corpora, and the reason is worth keeping in mind: a bank question is a vignette written so as not to name its diagnosis. Asked "what causes croup", the reranker prefers a question that says the word croup in passing (a post-influenza bacterial tracheitis case) over the vignette with the barking cough that never says it. That is defensible — someone searching "croup" is not badly served by either — but it is the bi-encoder's strength being partly traded away, and it is why the question-corpus gain (+0.09 precision@3) is smaller than the prose-corpus one relative to how wrong the old order was.
The failure to watch for is a query the library does not cover. "How is croup treated at home", against a corpus with no croup treatment section, is answered with Heat-Related Illness › Treatment, Tetanus › Treatment, Patellofemoral Pain Syndrome › Treatment — the cross-encoder matching the shape of the question when it cannot match the subject. Fusion's answer was no better, but it was wrong in a way that looked wrong. Two things contain this: AI Mode's closeness gate is measured on cosine over the whole corpus and still says "nothing here covers this" regardless of what the reranker thought (see retrieval-thresholds.md), and search results carry their own snippet. A minimum relevance score would be the third, and it is deliberately not implemented: it would need the same measured calibration the cosine thresholds have, on a scale that changes with the model, and getting it wrong empties a page.
When it is not there
Unset LITELLM_RERANK_MODEL (or the settings:rerank_model override in Redis,
which wins) and every list is in fusion order, with no error anywhere. The same
is true of a proxy that is down, slow, rate-limiting, or answering with
something that does not line up with the request: rerank() returns None and
every caller reads that as "keep what you had". The 429 storm from the Jina
model above is what that looks like in practice — a warning line per call and
results in their previous order.
POST /admin/rerank/test checks the configured model end to end, including
whether it puts an obviously relevant document above an obviously irrelevant
one. A reranker that answers 200 and ranks the decoy first is worse than one
that is switched off, and nothing else on the site would ever tell you.
Re-measuring
docker compose exec -T backend python - <<'EOF'
from app.database import SessionLocal
from app.services import search_service as ss
db = SessionLocal()
for q in ["management of bronchiolitis in an infant", "delayed passage of meconium"]:
ranked, _ = ss.hybrid_ids(db, q, "article_section", limit=200)
print(q, "\n before", ranked[:3], "\n after ", ss.rerank_ids(db, q, "article_section", ranked)[:3])
EOF
The precision figures above come from labels that already exist in the database — disease tags on questions, and an article owning its sections. Re-run that comparison rather than judging by eye after a corpus change; ten queries read by hand will agree with whichever ordering was looked at second.