448bfdd71c
Two halves of one TODO, settled with a measurement rather than a guess. *Shrunk readiness* was already done — `CandidateRanking.accuracy()` pulls a topic towards NEUTRAL_RECALL by PRIOR_ANSWERS, so one miss does not read as 0% — and there is now a test pinning it, because the note claiming otherwise outlived the fix by weeks. *Difficulty as a dimension the session moves along* cannot be built, and the reason is a number: all 2,924 questions have a NULL `difficulty`, and the empirical route is no better at 788 answers over 706 questions from 5 learners, about one answer each. A ladder scored against that would be scoring noise while looking as though it worked. What is built instead is honesty in the control that already exists: the Difficulty facet counts each level under the other filters and disables one that would empty the bank, so nobody picks Hard and watches the count fall to zero with no explanation. Reopen the ordering when something writes that column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
223 lines
10 KiB
Markdown
223 lines
10 KiB
Markdown
# How the study recommendations and the adaptive session work
|
||
|
||
Two questions were asked and owed an answer: *how do the study recommendations
|
||
work?* and *how would an adaptive session work?* Both are built; this is what
|
||
the code actually does, including the parts that are weaker than they look.
|
||
|
||
---
|
||
|
||
## 1. Study recommendations
|
||
|
||
`GET /study-tools/recommendations` — `backend/app/routers/study_tools.py`.
|
||
|
||
### What goes in
|
||
|
||
Only the learner's **completed, non-expired, non-course** answers:
|
||
|
||
```python
|
||
QuizAttempt.user_id == user.id,
|
||
QuizAttempt.completed_at.isnot(None),
|
||
or_(QuizAttempt.expired == 0, QuizAttempt.expired.is_(None)),
|
||
Quiz.course_id.is_(None),
|
||
*([exam_filter] if exam_filter is not None else []),
|
||
```
|
||
|
||
`exam_filter` is the learner's chosen study objective. Answers to questions
|
||
linked to a *different* exam are excluded, and so is that material from the
|
||
denominator — someone revising for a paediatrics board who happens to sit a
|
||
step-exam plan does not have it steer their recommendations. Questions linked
|
||
to no exam stay in: unlinked means unclassified, not excluded.
|
||
|
||
### Membership: the same answers, three ways
|
||
|
||
Every answer is attributed to a set of **group keys**, and the grouping decides
|
||
what a key is:
|
||
|
||
| Tab | Key | How a question reaches it |
|
||
|---|---|---|
|
||
| Articles | article id | the question's categories → the published article filed under one |
|
||
| Disciplines | top-level category id | the question's categories, rolled up to the top of the tree |
|
||
| Systems | organ-system tag id | the question's symptom keyword → the system that keyword is filed under |
|
||
|
||
A question counts towards **every** ancestor it has, not just its primary
|
||
category, and `question_category_links` adds cross-filed categories on top. So
|
||
one answer on *Kawasaki disease* raises the count for Kawasaki, for Cardiology,
|
||
and for the Kawasaki article.
|
||
|
||
Systems is the odd one: no question is tagged with an organ system directly.
|
||
It carries a symptom keyword (`question_tags.type = 'keyword'`) and 726 of
|
||
those keywords have a system as `parent_id`. That path reaches **1,502 of
|
||
2,948 questions**, which is why that tab states its own coverage instead of
|
||
implying it can see the whole bank.
|
||
|
||
### The four numbers per row
|
||
|
||
```python
|
||
accuracy = 100 * correct / answered # what you actually scored
|
||
coverage = 100 * seen / available # how much of it you have met
|
||
relevance = 100 * available / denominator # how much of the bank it is
|
||
readiness = 100 * (c + 8 * overall) / (n + 8) # accuracy, shrunk
|
||
```
|
||
|
||
**Readiness** is the only one that is not arithmetic on raw counts. It is a
|
||
plain empirical-Bayes shrinkage: a category's accuracy is pulled toward the
|
||
learner's own overall accuracy in proportion to how few answers it has. With
|
||
`READINESS_PRIOR_ANSWERS = 8`, one lucky question in a category does not read
|
||
as mastery and one unlucky one does not read as a gap — the category has to
|
||
earn its distance from your average. It stays locked until
|
||
`READINESS_UNLOCK_ANSWERS = 40` answers exist at all, because before that the
|
||
learner's own average is not a stable thing to shrink toward.
|
||
|
||
It is **not** a psychometric score and not a prediction of any exam. The
|
||
`basis` string returned with every response says so, in those words, because a
|
||
number on a dashboard acquires authority it has not earned unless something
|
||
tells the reader what it is.
|
||
|
||
**Relevance** divides by `bank_total` normally, but by `grouped_total` for
|
||
Systems — dividing by the whole bank when the grouping can only see half of it
|
||
would make every system look half as relevant as it is.
|
||
|
||
### Ranking
|
||
|
||
```python
|
||
gap = max(0, (baseline - score) / 100) # how far below your own average
|
||
unseen = 1 - coverage / 100
|
||
priority = gap * relevance + 0.25 * unseen * relevance
|
||
```
|
||
|
||
`baseline` is the learner's overall accuracy (70% before there is one). So a
|
||
topic rises when it is **weak** *and* **a big part of the bank**. Material never
|
||
touched has no gap to measure, so it enters at `gap = 0.5` and is carried by
|
||
the `unseen` term at quarter weight — enough to surface, not enough to crowd
|
||
out a topic you are demonstrably failing.
|
||
|
||
The top three rows with any answers are flagged `is_focus_area`.
|
||
|
||
---
|
||
|
||
## 2. The adaptive session
|
||
|
||
`adaptive_select()` — `backend/app/services/quiz_builder.py`.
|
||
|
||
Four steps:
|
||
|
||
1. **Candidates.** The filtered bank for the chosen categories/state/difficulty.
|
||
2. **Unanswered first**, sorted by the accuracy of their category — so a
|
||
question you have never seen, in the area you are worst at, comes first. A
|
||
category you have never answered in scores `0.5`, which puts it mid-pack.
|
||
3. **Then recycle.** If there are not enough unanswered, previously answered
|
||
questions are pulled back in, weighted `1 - 0.85` if you got it right and
|
||
`1 - 0.25` if you got it wrong — so a question you failed is roughly five
|
||
times likelier to return than one you passed. Ties break toward the oldest
|
||
incorrect answer.
|
||
4. **Damping.** Each time a category is picked, its weight is halved
|
||
(`damping[category] *= 0.5`). Without this a single weak category would fill
|
||
the whole session; with it, the session walks across your weak areas.
|
||
|
||
### Where it is weaker than it looks
|
||
|
||
- **`.limit(2000)` on the candidate query.** The bank is 2,948 questions, so
|
||
adaptive selection cannot currently see about a third of it, and which third
|
||
depends on database order. This is a real cap, not a tuning choice.
|
||
- **`accuracy()` is computed per category from a linear scan** —
|
||
`next((r[1] for r in rows if r[0] == qid), None)` inside a loop over every
|
||
answer. That is O(answers × candidates); at 2,000 candidates and a few
|
||
thousand answers it is the slowest part of building a session.
|
||
- **Difficulty is not used in the ordering**, only as a filter — and as of
|
||
2026-09-12 it cannot be. Every one of the 2,924 questions has a NULL
|
||
`difficulty`, and deriving it empirically is no better: 788 answers over 706
|
||
questions from 5 learners is about one answer each, so a measured difficulty
|
||
would be 0% or 100% per question. A ladder scored against either would be
|
||
scoring noise while appearing to work. The control now counts each level and
|
||
disables one that would empty the bank; the ordering waits for something that
|
||
actually writes that column.
|
||
- ~~**Step 2 sorts by raw category accuracy, not readiness.**~~ Fixed, and
|
||
pinned by a test: `CandidateRanking.accuracy()` is
|
||
`(correct + PRIOR_ANSWERS × NEUTRAL_RECALL) / (total + PRIOR_ANSWERS)`, so a
|
||
topic with one wrong answer sits near neutral rather than at the top of the
|
||
queue. The recommendations page sorts on `readiness` wherever it has one and
|
||
falls back to raw accuracy only below the unlock threshold, where there is no
|
||
overall accuracy stable enough to shrink towards.
|
||
|
||
The first two are defects and are worth fixing. The last two are honest
|
||
limitations of the current design and would be the substance of a better one.
|
||
|
||
---
|
||
|
||
## 3. What the adaptive session should be
|
||
|
||
Written 2026-09-11, after the exam blueprint landed. The design below is not
|
||
built; this is the argument for it, so the decision is on paper before the code
|
||
is.
|
||
|
||
### The thing that was missing
|
||
|
||
Every version of this so far has ranked questions by **how badly you are doing**
|
||
and nothing else. That is half a question. The other half is **how much it
|
||
matters**, and until today there was nothing in the database that could answer
|
||
it — so the code guessed, by treating every category as equally worth an hour.
|
||
|
||
`exam_blueprints.weight` answers it now. The ABP publishes that preventive care
|
||
is 12% of a general paediatrics paper and rheumatology is 2%. Six points of
|
||
weakness in preventive care costs six times what the same weakness costs in
|
||
rheumatology, and a session that does not know this will spend your evening in
|
||
the wrong place while looking perfectly reasonable.
|
||
|
||
### The score
|
||
|
||
For each topic, one number — the marks you would expect to gain by studying it:
|
||
|
||
```
|
||
expected_gain = relevance × headroom × confidence
|
||
```
|
||
|
||
* **relevance** — the domain's published weight, divided among the topics under
|
||
it. From the blueprint. This is the part that is a fact rather than a model.
|
||
* **headroom** — `1 − readiness`. How much of that share you are currently
|
||
losing. Readiness is the shrunk accuracy the recommendations page already
|
||
computes, *not* raw accuracy: one wrong answer out of one must not read as
|
||
"you know nothing about neonatology".
|
||
* **confidence** — how much the estimate can be trusted, `n / (n + k)`. A topic
|
||
you have answered twice cannot outrank one you have answered forty times on
|
||
the strength of a bad afternoon. This is what stops the session chasing noise.
|
||
|
||
Sort topics by expected gain; fill the session from the top, damped as now so
|
||
one topic cannot take the whole session.
|
||
|
||
### Then, within a topic: difficulty that moves
|
||
|
||
The current session filters by difficulty and then ignores it. It should walk:
|
||
start near the learner's demonstrated level for that topic, step up after two
|
||
right, step down after one wrong. The point is not to be hard, it is to sit
|
||
where the information is — a question you would get right nine times in ten
|
||
teaches nothing, and neither does one you would get right once in ten.
|
||
|
||
### And a floor on coverage
|
||
|
||
A pure gain ranking will never show you a topic you are already good at, which
|
||
is how people arrive at an exam having forgotten something they knew in March.
|
||
Reserve a share of every session — a fifth, say — for **spaced return**: topics
|
||
you were right about, longest ago first. This is the one part that should not be
|
||
optimised, because its whole purpose is to be unwelcome.
|
||
|
||
### What has to be fixed first
|
||
|
||
The two defects named in §2 are load-bearing here, not tidying:
|
||
|
||
* **`.limit(2000)`** means a third of the bank is invisible to selection. A
|
||
scheme that reasons carefully about which question matters most, over an
|
||
arbitrary two-thirds of the questions, is a scheme that reasons carefully
|
||
about the wrong set.
|
||
* **The O(answers × candidates) accuracy scan** is already the slowest part of
|
||
building a session. This design asks for per-topic readiness *and* per-topic
|
||
difficulty, which makes it worse. It needs to become one grouped query.
|
||
|
||
### What this deliberately does not do
|
||
|
||
No item-response theory, no per-question difficulty estimated from other
|
||
learners' answers. Both would be better with enough data and worse without it,
|
||
and a bank this size with a handful of learners does not have it. The weights
|
||
are published, the readiness is measured, and the arithmetic above can be
|
||
explained to a learner in two sentences — which is the point, because a session
|
||
that cannot say why it chose a question is asking to be trusted rather than
|
||
earning it.
|