danvics/pdf-quiz-generator
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions

fix: the audit may not detach a clinical figure on its own opinion

Browse source 
The rule was in my head and in a commit message; it needed to be in the task.
The first run detached sixteen good figures — a tick on a leaf against a July
fever, fungal hyphae against a scaly rash, a recessed chin in a two-week-old's
notes, an ECG on a tachypnoeic neonate — because a judgement about relevance was
allowed to act on a photograph, and the connection between a clinical figure and
its question is often indirect.

Two changes. The prompt now separates the two kinds of figure and says how each
is judged: a clinical one — photograph, radiograph, ultrasound, ECG, fundoscopy,
otoscopy, microscopy — is refused only when it is anatomically impossible, and a
table, citation, chart or nomogram is refused whenever it is about another
subject. And the model must now say which kind it is looking at, so the code can
enforce it: "no" on anything clinical becomes "unsure", which flags it for a
person instead of detaching it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
This commit is contained in:
Daniel 2026-09-12 21:11:17 +02:00
parent 658482c53a
commit 500e8430a5
1 changed files with 34 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
backend/app/tasks/quiz_tasks.py
View file

@ -395,16 +395,30 @@ FIGURE_AUDIT_PROMPT = """This figure is attached to the exam question below. Dec
to it.
It belongs if a candidate would need it, or could reasonably use it, to answer
*this* question — a radiograph of the finding, a growth chart the question asks
you to read, a table of the schedule being asked about. It does not belong if it
is about some other topic entirely, which happens when a figure was lifted from
the wrong page of a source document.
*this* question. It does not belong if it is about some other topic entirely,
which happens when a figure is lifted from the wrong page of a source document.
Be strict about "no" and generous about "unsure": detaching a figure a question
needs is worse than leaving a stray one attached for somebody to notice.
**The rule that matters, learned by getting it wrong.** Two kinds of figure sit
on these questions and they are not judged the same way.
A *clinical* figure — a photograph, radiograph, ultrasound, ECG, fundoscopy,
otoscopy, a microscopy slide, a specimen — is almost always the right one, and
the connection is often indirect: a tick on a leaf against a July fever, fungal
hyphae against a scaly rash, a recessed chin in the notes of a two-week-old.
Answer "no" for one of these ONLY when it is anatomically or clinically
impossible — a shoulder radiograph on a knee injury. Otherwise "yes", or
"unsure" if you truly cannot tell. Detaching a clinical figure a question needs
is the worst outcome here.
A *non-clinical* figure — a table, a citation, a reference list, a bar chart, a
nomogram, a page of text — belongs only when it is about the same subject as the
question. These are where mis-extraction shows up, and where "no" is usually
right.
Answer ONLY with JSON:
{"belongs": "yes" | "no" | "unsure", "shows": "one sentence describing the figure"}
{"belongs": "yes" | "no" | "unsure",
"kind": "clinical" | "table" | "other",
"shows": "one sentence describing the figure"}
QUESTION
"""
@ -486,8 +500,17 @@ def audit_question_figures(self, job_id: str = "", limit: int | None = None,
continue
belongs = str(verdict.get("belongs", "unsure")).strip().lower()
kind = str(verdict.get("kind", "")).strip().lower()
shows = str(verdict.get("shows", "")).strip()[:600]
# The prompt says it and this enforces it: a clinical figure is
# never detached on the model's say-so alone. Sixteen good ones went
# in the first run — a tick on a leaf, fungal hyphae, an ECG on a
# tachypnoeic neonate — because a judgement about relevance was
# allowed to act on a photograph. It becomes a flag for a person.
if belongs == "no" and kind == "clinical":
belongs = "unsure"
# The description is worth keeping whatever the verdict: "Figure
# from question #1206" is a filename with extra steps.
asset = db.query(MediaAsset).filter(MediaAsset.path == question.image_path).first()
@ -501,7 +524,8 @@ def audit_question_figures(self, job_id: str = "", limit: int | None = None,
if belongs == "no":
mismatches.append({"question_id": question.id, "path": question.image_path,
"shows": shows, "stem": stem[:160], "verdict": "no"})
"shows": shows, "stem": stem[:160], "kind": kind,
"verdict": "no"})
if detach:
question.image_path = None
db.query(QuestionMedia).filter(
@ -518,7 +542,8 @@ def audit_question_figures(self, job_id: str = "", limit: int | None = None,
# Named, not just counted: "2 unsure" is a number nobody can
# act on. These stay attached and go on the list for a person.
mismatches.append({"question_id": question.id, "path": question.image_path,
"shows": shows, "stem": stem[:160], "verdict": "unsure"})
"shows": shows, "stem": stem[:160], "kind": kind,
"verdict": "unsure"})
else:
kept += 1