031de53034
Retrieval fused a bi-encoder and BM25 by reciprocal rank. A bi-encoder embeds a document long before the question exists, so the two never meet: it is good at "same topic" and mediocre at "answers this". A cross-encoder reads the pair. The proxy already serves three — `cohere-rerank-v4.0-pro` is the default and measurably better than the fast variant. Query text goes exactly where the embeddings already go, and nothing new was signed up for. It found a defect nobody was looking for. In AI Mode each finder scored `1/(1+rank)` *within its own corpus*, so the best article, section, question and card all scored 1.0 and the shortlist was a meaningless round-robin. A cross-encoder is the first thing in this system that can compare a question with a section. Candidates per kind widened so it can select rather than merely reorder. Measured against labels neither ranker produced. Questions, 60 disease tags: precision@3 0.394 → 0.483. Sections, 60 article titles: 0.772 → 0.833. "Management of bronchiolitis" led with influenza transmission and a pregnancy question; "when do you image a first febrile seizure" returned the definition rather than the sentence saying imaging is unnecessary. And the honest negative, in docs/reranking.md: board vignettes are written *not* to name their diagnosis, so on "what causes croup" it prefers a question that says the word in passing over the barking-cough vignette that never says it. Some of the bi-encoder's strength is traded away. Not on the typeahead. A page of results is a choice being made and worth a third of a second; a typeahead is a word being finished, runs on every keystroke, and has nothing to judge yet. The three-state thresholds stay on cosine, argued at the constant: a reranker only ever sees a shortlist and structurally cannot answer the corpus-wide question those numbers ask, and whether an answer claims to come from the library is a promise that must not depend on a network hop. Every failure returns None and leaves the order alone — unconfigured, no proxy, connect error, bare 502, timeout, non-JSON, a duplicate or out-of-range index, a non-numeric score, a list the wrong length. Verified against the running site with a bogus model name: same results, fused order, no error to the reader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
111 lines
5.6 KiB
Markdown
111 lines
5.6 KiB
Markdown
# What "close enough" means
|
||
|
||
AI Mode gives one of three answers, and which one is decided by a number rather
|
||
than by asking the model to work out its own situation. This is where that
|
||
number comes from, what it was measured against, and how to re-measure it.
|
||
|
||
## Why a number and not a longer prompt
|
||
|
||
Retrieval could not say "nothing". `hybrid_ids()` fuses a lexical ranker and a
|
||
semantic one by reciprocal rank, throws the distances away, and returns the
|
||
*union* — so the shortlist was never empty. A question about photosynthesis came
|
||
back with six paediatric sources and an instruction to answer only from them.
|
||
|
||
Adding scenarios to the prompt would have asked the model to classify, in prose,
|
||
a situation the data already knows. Classification-by-prose is the part that does
|
||
not work, and it is also the part that makes prompts long. So the branch lives in
|
||
code: `ai_mode_service.answer_mode()`.
|
||
|
||
## The three answers
|
||
|
||
| closeness | mode | what the learner gets |
|
||
|---|---|---|
|
||
| `>= STRONG_MATCH` | `sourced` | answered from the library, cited |
|
||
| `>= ADJACENT_MATCH` | `adjacent` | "nothing covers this directly; the closest is…", then an answer marking which parts came from where |
|
||
| below | `open` | one line saying the library does not cover it, then an answer from general knowledge, citing nothing |
|
||
| unmeasurable (`None`) | `sourced` | see below |
|
||
|
||
**Unmeasurable is not low.** No vector database, or a downed encoder, returns
|
||
`None`. Retrieval still found its rows by other means, so they are still cited;
|
||
dropping every citation because the ruler is missing is the worse failure.
|
||
|
||
## The numbers, and the measurement behind them
|
||
|
||
In `backend/app/services/ai_mode_service.py`:
|
||
|
||
```python
|
||
STRONG_MATCH = 0.55
|
||
ADJACENT_MATCH = 0.50
|
||
```
|
||
|
||
Measured on 2026-09-12, against the corpus with article bodies embedded — eight
|
||
clearly on-topic questions and eight clearly off-topic, scored with
|
||
`search_service.top_similarity()` over the article and section corpora:
|
||
|
||
| | range | examples |
|
||
|---|---|---|
|
||
| off-topic | **0.339 – 0.499** | the French revolution 0.339 · how do I bake sourdough 0.410 · quantum entanglement 0.448 · javascript closures 0.452 · tell me a joke 0.455 · what is a mortgage 0.462 · discuss love 0.491 · photosynthesis 0.499 |
|
||
| on-topic | **0.586 – 0.740** | neonatal jaundice phototherapy 0.586 · what causes croup 0.594 · febrile seizure workup 0.638 · Kawasaki disease 0.660 · testicular torsion 0.713 · bronchiolitis in an infant 0.716 · iron deficiency in a toddler 0.733 · posterior urethral valves 0.740 |
|
||
|
||
The thresholds sit in the gap between those two bands.
|
||
|
||
Worth noticing that **"discuss love" scores 0.491, alongside "tell me a joke"**.
|
||
It feels adjacent to a paediatrics library — attachment, behaviour — and it is
|
||
not: 0.49 is where anything written in English lands against any corpus. That is
|
||
the reading to keep in mind. A number in the 0.4s is noise, not a weak signal.
|
||
|
||
## Why these are not the reranker's score
|
||
|
||
Retrieval now has a cross-encoder in it (docs/reranking.md), which is a better
|
||
judge of a query-document pair than cosine distance is by a wide margin. The
|
||
three-state decision still does not go through it, on purpose.
|
||
|
||
The question here is *"is there anything in this library about this at all"*,
|
||
and that is a question about the corpus, not about a shortlist. `top_similarity`
|
||
answers it by scanning every embedded row in two corpora through the vector
|
||
index, in about 25 ms. A reranker can only score the candidates something else
|
||
already shortlisted, so a reranked closeness could not tell "the library does
|
||
not cover this" from "retrieval had a bad day" — and on this corpus it is
|
||
exactly the uncovered query where the cross-encoder is least trustworthy: asked
|
||
how croup is treated at home, with no croup treatment section in the library, it
|
||
promotes the *Treatment* section of whatever else is lying around.
|
||
|
||
There is also a failure argument. Order is a preference, so a reranker being
|
||
down costs a worse-ordered page and nothing else. Whether the answer *claims to
|
||
come from the library* is a promise, and putting a network hop in the path of a
|
||
promise means a proxy restart changes what the assistant asserts.
|
||
|
||
So: the cross-encoder decides the order of the shortlist; cosine decides what
|
||
the answer is allowed to say about it. The numbers below are unchanged and did
|
||
not need re-measuring, because nothing that feeds them changed.
|
||
|
||
## Why these are not `SEMANTIC_FLOOR`
|
||
|
||
`search_service.SEMANTIC_FLOOR` (0.45) decides what is worth putting in a list,
|
||
where a weak hit costs a reader one glance. These decide whether an answer
|
||
*claims to come from the library*, and a wrong claim there costs a learner their
|
||
trust in every other answer. Different jobs, different numbers, and tying them
|
||
together would mean one could not be tuned without moving the other.
|
||
|
||
## Re-measuring
|
||
|
||
Re-measure when the corpus changes size or subject — a much larger library
|
||
raises the floor for everything, because there is more for any query to be
|
||
vaguely near.
|
||
|
||
```
|
||
docker compose exec -T backend python -c "
|
||
from app.database import SessionLocal
|
||
from app.services.search_service import top_similarity
|
||
db = SessionLocal()
|
||
for q in ['the French revolution', 'tell me a joke', 'discuss love',
|
||
'what causes croup', 'bronchiolitis in an infant']:
|
||
print(f'{top_similarity(db, q):.3f} {q}')
|
||
"
|
||
```
|
||
|
||
Pick your own on- and off-topic sets, run both, and put the thresholds in the
|
||
gap. If there is no gap, the embeddings are wrong before the thresholds are —
|
||
that is what happened before the article bodies were indexed, when 98% of the
|
||
corpus was embedded on its title and summary alone and every distance was
|
||
measuring the wrong thing.
|