pdf-quiz-generator/docs/retrieval-thresholds.md

What "close enough" means

AI Mode gives one of three answers, and which one is decided by a number rather than by asking the model to work out its own situation. This is where that number comes from, what it was measured against, and how to re-measure it.

Why a number and not a longer prompt

Retrieval could not say "nothing". hybrid_ids() fuses a lexical ranker and a semantic one by reciprocal rank, throws the distances away, and returns the union — so the shortlist was never empty. A question about photosynthesis came back with six paediatric sources and an instruction to answer only from them.

Adding scenarios to the prompt would have asked the model to classify, in prose, a situation the data already knows. Classification-by-prose is the part that does not work, and it is also the part that makes prompts long. So the branch lives in code: ai_mode_service.answer_mode().

The three answers

closeness	mode	what the learner gets
>= STRONG_MATCH	sourced	answered from the library, cited
>= ADJACENT_MATCH	adjacent	"nothing covers this directly; the closest is…", then an answer marking which parts came from where
below	open	one line saying the library does not cover it, then an answer from general knowledge, citing nothing
unmeasurable (None)	sourced	see below

Unmeasurable is not low. No vector database, or a downed encoder, returns None. Retrieval still found its rows by other means, so they are still cited; dropping every citation because the ruler is missing is the worse failure.

The numbers, and the measurement behind them

In backend/app/services/ai_mode_service.py:

STRONG_MATCH = 0.55
ADJACENT_MATCH = 0.50

Measured on 2026-09-12, against the corpus with article bodies embedded — eight clearly on-topic questions and eight clearly off-topic, scored with search_service.top_similarity() over the article and section corpora:

	range	examples
off-topic	0.339 – 0.499	the French revolution 0.339 · how do I bake sourdough 0.410 · quantum entanglement 0.448 · javascript closures 0.452 · tell me a joke 0.455 · what is a mortgage 0.462 · discuss love 0.491 · photosynthesis 0.499
on-topic	0.586 – 0.740	neonatal jaundice phototherapy 0.586 · what causes croup 0.594 · febrile seizure workup 0.638 · Kawasaki disease 0.660 · testicular torsion 0.713 · bronchiolitis in an infant 0.716 · iron deficiency in a toddler 0.733 · posterior urethral valves 0.740

The thresholds sit in the gap between those two bands.

Worth noticing that "discuss love" scores 0.491, alongside "tell me a joke". It feels adjacent to a paediatrics library — attachment, behaviour — and it is not: 0.49 is where anything written in English lands against any corpus. That is the reading to keep in mind. A number in the 0.4s is noise, not a weak signal.

Why these are not the reranker's score

Retrieval now has a cross-encoder in it (docs/reranking.md), which is a better judge of a query-document pair than cosine distance is by a wide margin. The three-state decision still does not go through it, on purpose.

The question here is "is there anything in this library about this at all", and that is a question about the corpus, not about a shortlist. top_similarity answers it by scanning every embedded row in two corpora through the vector index, in about 25 ms. A reranker can only score the candidates something else already shortlisted, so a reranked closeness could not tell "the library does not cover this" from "retrieval had a bad day" — and on this corpus it is exactly the uncovered query where the cross-encoder is least trustworthy: asked how croup is treated at home, with no croup treatment section in the library, it promotes the Treatment section of whatever else is lying around.

There is also a failure argument. Order is a preference, so a reranker being down costs a worse-ordered page and nothing else. Whether the answer claims to come from the library is a promise, and putting a network hop in the path of a promise means a proxy restart changes what the assistant asserts.

So: the cross-encoder decides the order of the shortlist; cosine decides what the answer is allowed to say about it. The numbers below are unchanged and did not need re-measuring, because nothing that feeds them changed.

Why these are not SEMANTIC_FLOOR

search_service.SEMANTIC_FLOOR (0.45) decides what is worth putting in a list, where a weak hit costs a reader one glance. These decide whether an answer claims to come from the library, and a wrong claim there costs a learner their trust in every other answer. Different jobs, different numbers, and tying them together would mean one could not be tuned without moving the other.

Re-measuring

Re-measure when the corpus changes size or subject — a much larger library raises the floor for everything, because there is more for any query to be vaguely near.

docker compose exec -T backend python -c "
from app.database import SessionLocal
from app.services.search_service import top_similarity
db = SessionLocal()
for q in ['the French revolution', 'tell me a joke', 'discuss love',
          'what causes croup', 'bronchiolitis in an infant']:
    print(f'{top_similarity(db, q):.3f}  {q}')
"

Pick your own on- and off-topic sets, run both, and put the thresholds in the gap. If there is no gap, the embeddings are wrong before the thresholds are — that is what happened before the article bodies were indexed, when 98% of the corpus was embedded on its title and summary alone and every distance was measuring the wrong thing.