fix: MinIO name collision, and refuse to write an article from fragments
MinIO was resolving to the wrong container Putting the backend on danvics_milvus to reach the clinical index gave it a second service called `minio`, and Docker resolved that one first. Every object read failed with InvalidAccessKeyId while the bucket simply looked empty — all 435 stem images unservable, and nothing in the logs saying why. The quiz MinIO now answers to `quiz-minio`, which nothing else on this host claims. A topic named after a shelf retrieved headings, not prose "Pediatric Pulmonology" returned ten chunks whose top hit was 29 characters — `**270** Pediatric Pulmonology`, an index line. Chapter titles rank well against a query that looks like a chapter title. The model was handed a prompt with citations and no content and said so, which was the correct response and read as a JSON failure. Two gates, both stated in the code. A chunk under 200 characters is a heading or a running header rather than something to write from. A topic whose passages total under 3,000 characters is skipped with the count in the reason, rather than asking a model to write a medical article out of fragments — it will either refuse or invent, and only one of those is visible. The 71 generated drafts are deleted at the user's request. Nothing linked to them and generation is resumable, so the cost was model calls rather than work. Question bank corrections, from the agent that ran alongside: 262 questions had OCR-mangled units repaired — `inEq/L`, `mrnol/L`, flattened `10⁹` superscripts and the rest — each with a version snapshot written first, so every edit is reversible from the existing question editor. 94 stem images that belonged to the explanation were removed; PREP's own `Item Q37A` / `Item C37B` labels turned out to be a far better signal than word cues, taking the confident split from 69/58/308 to 300/81/54. 13 uncertain images are listed for a person. 203 backend, 223 frontend green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
This commit is contained in:
Daniel
parent
025e5bb4ac
commit
af59fdb960
5 changed files with 575 additions and 5 deletions
|
|
@ -29,7 +29,10 @@ from app.services.ai_service import _call_model, get_model_for_task
|
|||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PASSAGES = 10
|
||||
PASSAGES = 14
|
||||
# Enough retrieved prose to write from. Below this the model is being asked to
|
||||
# write a medical article out of fragments, and it will either refuse or invent.
|
||||
MIN_SOURCE_CHARS = 3000
|
||||
# Long enough to be worth reading, short enough that nobody skims past the point.
|
||||
LONG_WORDS = 550
|
||||
SHELF = "Pediatrics"
|
||||
|
|
@ -134,10 +137,13 @@ def write_article(db: Session, topic: str, category_id: int | None = None,
|
|||
background run over hundreds of topics and wants to know what happened.
|
||||
"""
|
||||
passages = clinical_library.search(topic, limit=PASSAGES, folder_contains=SHELF)
|
||||
if len(passages) < 3:
|
||||
source_chars = sum(len(p["text"]) for p in passages)
|
||||
if len(passages) < 3 or source_chars < MIN_SOURCE_CHARS:
|
||||
# Too little to ground an article. Writing one anyway would produce
|
||||
# exactly the confident unverifiable prose this is designed to avoid.
|
||||
return {"topic": topic, "status": "skipped", "reason": "not enough source material",
|
||||
# exactly the confident unverifiable prose this is designed to avoid,
|
||||
# and a topic named after a shelf rather than a condition lands here.
|
||||
return {"topic": topic, "status": "skipped",
|
||||
"reason": f"only {source_chars} chars from {len(passages)} passages",
|
||||
"passages": len(passages)}
|
||||
|
||||
model_id, api_key = get_model_for_task(db, "extraction")
|
||||
|
|
|
|||
|
|
@ -30,6 +30,10 @@ logger = logging.getLogger(__name__)
|
|||
# Enough context to write a section from, without turning a prompt into a book.
|
||||
SNIPPET_CHARS = 1200
|
||||
DEFAULT_LIMIT = 12
|
||||
# Below this a chunk is a heading, an index line or a running header, not prose.
|
||||
# They rank well against a query that looks like a chapter title and then fill
|
||||
# the shortlist with nothing — a topic named after a shelf retrieves ten of them.
|
||||
MIN_PROSE_CHARS = 200
|
||||
|
||||
|
||||
TIMEOUT = 20
|
||||
|
|
@ -138,7 +142,7 @@ def search(query: str, limit: int = DEFAULT_LIMIT,
|
|||
except ValueError:
|
||||
raw = {}
|
||||
text = _tidy(raw.get("excerpt"))
|
||||
if not text:
|
||||
if len(text) < MIN_PROSE_CHARS:
|
||||
continue
|
||||
results.append({
|
||||
"text": text[:SNIPPET_CHARS],
|
||||
|
|
|
|||
273
backend/scripts/triage_question_images.py
Normal file
273
backend/scripts/triage_question_images.py
Normal file
|
|
@ -0,0 +1,273 @@
|
|||
"""Take the answer figure off the question stem.
|
||||
|
||||
PDF extraction attached one image per question to `image_path`, which the quiz
|
||||
player shows above the stem. For many questions that image is not the vignette's
|
||||
photograph at all — it is the figure from the *explanation*, the annotated smear
|
||||
or the labelled radiograph that gives the answer away. Showing it during a test
|
||||
is a spoiler.
|
||||
|
||||
Three groups, decided in this order:
|
||||
|
||||
1. the stem itself refers to a figure → the image belongs there, leave it;
|
||||
2. only the explanation refers to one → the image belongs to the explanation,
|
||||
so it comes off the stem;
|
||||
(PREP labels its figures by where they are printed — "Item Q37A" beside the
|
||||
question, "Item C37B" beside the critique — which decides most of these;)
|
||||
3. neither says anything → ask a vision model what the picture
|
||||
shows and which half of the question it illustrates.
|
||||
|
||||
Removal means `image_path = NULL`. The image is not moved to
|
||||
`explanation_image_path`: an unverified guess about where a figure belongs is
|
||||
worth less than a clean stem, and the file itself is untouched in object
|
||||
storage. Every removal snapshots the question first, through the same helper the
|
||||
edit UI uses, so any of this can be restored from the question's version history.
|
||||
|
||||
docker compose exec backend python -m scripts.triage_question_images
|
||||
docker compose exec backend python -m scripts.triage_question_images --apply
|
||||
|
||||
--no-vision only apply the deterministic rule, ask no model
|
||||
--limit N classify at most N unknowns (for a spot check)
|
||||
--model ID override the model (default: the configured extraction model)
|
||||
--report PATH where the human-review list is written
|
||||
|
||||
Anything the model is not confident about is left exactly as it is and listed in
|
||||
the report, with the question id and the image path, for a person to decide.
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import mimetypes
|
||||
import re
|
||||
import sys
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import litellm
|
||||
|
||||
from app.config import settings
|
||||
from app.database import SessionLocal
|
||||
from app.models.question import Question
|
||||
from app.routers.questions import _snapshot_question
|
||||
from app.services import storage_service
|
||||
from app.services.ai_service import _proxy_model, get_model_for_task
|
||||
|
||||
SCRIPT_EDITOR_ID = None
|
||||
|
||||
# Does this text tell the reader to look at a picture? The stem list is the
|
||||
# wider of the two on purpose: a stem that might need its figure keeps it.
|
||||
STEM_CUE = re.compile(
|
||||
r"figure|photograph|photo\b|image|picture|pictured|shown|depicted|illustrat|"
|
||||
r"radiograph|x-ray|xray|roentgenogram|ultrasound|scan\b|smear|tracing|"
|
||||
r"electrocardiogram|\becg\b|\bekg\b|\bmri\b|\bct\b|growth chart",
|
||||
re.I)
|
||||
|
||||
# Evidence that the figure is the explanation's — the same words. An
|
||||
# explanation saying "the rash shown is typical of…" is describing a figure
|
||||
# printed with the critique, not with the vignette.
|
||||
EXPLANATION_CUE = STEM_CUE
|
||||
|
||||
# The strongest signal in this bank, and one specific to PREP: figures are
|
||||
# labelled by where they are printed. "Item Q37A" is a figure beside the
|
||||
# question; "Item C37B" is a figure beside the critique. The stem citing an
|
||||
# Item Q settles it on its own, and only a critique cites an Item C.
|
||||
# "ltem" with a lowercase L is the scanner reading a capital I.
|
||||
STEM_FIGURE_LABEL = re.compile(r"\b[Il]tem\s*Q\s*\d", re.I)
|
||||
EXPLANATION_FIGURE_LABEL = re.compile(r"\b[Il]tem\s*C\s*\d", re.I)
|
||||
|
||||
# How sure the model has to be before an image is removed on its say-so. Set
|
||||
# from a spot check of the replies: every 0.95 verdict held up against the
|
||||
# picture, while one of the three 0.85s was wrong — it called an unlabelled
|
||||
# photograph of the vignette's own rash an explanation figure. Below the bar
|
||||
# the question is left untouched and goes on the report for a person to read.
|
||||
MIN_CONFIDENCE = 0.9
|
||||
|
||||
PROMPT = """You are auditing a pediatric board-exam question.
|
||||
|
||||
The image above was attached to this question during PDF extraction. It may
|
||||
belong to the QUESTION STEM (a photograph the examinee must look at to answer)
|
||||
or to the EXPLANATION (a figure printed with the answer critique, which would
|
||||
give the answer away if shown during the test).
|
||||
|
||||
QUESTION STEM:
|
||||
{stem}
|
||||
|
||||
EXPLANATION:
|
||||
{explanation}
|
||||
|
||||
Decide which one the image illustrates. Weigh these signals:
|
||||
- Does the stem describe a finding that this image shows? Then it is "stem".
|
||||
- Is the image annotated, labelled, captioned, a diagram, a table, an algorithm,
|
||||
a graph, or does it show the diagnosis being named? Then it is "explanation".
|
||||
- Is it a page header, logo, decorative rule, or blank? Then it is "neither".
|
||||
|
||||
Answer with ONLY this JSON, no markdown fence and no preamble:
|
||||
{{"belongs": "stem" | "explanation" | "neither" | "unclear",
|
||||
"confidence": <0.0-1.0>,
|
||||
"reason": "<one short sentence>"}}"""
|
||||
|
||||
|
||||
def image_bytes(key: str) -> tuple[bytes, str] | None:
|
||||
"""The image and the media type a vision model will accept, or None."""
|
||||
data = storage_service.s3_object(key) or storage_service.load(key)
|
||||
if not data:
|
||||
return None
|
||||
media_type = mimetypes.guess_type(key)[0] or "image/jpeg"
|
||||
if key.lower().endswith((".jpx", ".jp2")) or media_type in ("image/jp2", "image/jpx"):
|
||||
# JPEG 2000 is not accepted by the vision APIs; PyMuPDF is already a
|
||||
# dependency of the PDF pipeline and re-encodes it without a new one.
|
||||
try:
|
||||
import fitz
|
||||
|
||||
return fitz.Pixmap(data).tobytes("png"), "image/png"
|
||||
except Exception:
|
||||
return None
|
||||
return data, media_type
|
||||
|
||||
|
||||
def classify(stem: str, explanation: str, key: str, model: str,
|
||||
api_key: str | None) -> dict:
|
||||
"""Ask the vision model where the image belongs. Never raises."""
|
||||
loaded = image_bytes(key)
|
||||
if not loaded:
|
||||
return {"belongs": "unclear", "confidence": 0.0, "reason": "image could not be read"}
|
||||
data, media_type = loaded
|
||||
|
||||
kwargs = {
|
||||
"model": _proxy_model(model),
|
||||
"temperature": 0,
|
||||
"messages": [{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {
|
||||
"url": f"data:{media_type};base64,{base64.b64encode(data).decode()}"}},
|
||||
{"type": "text", "text": PROMPT.format(
|
||||
stem=(stem or "")[:4000] or "(empty)",
|
||||
explanation=(explanation or "")[:4000] or "(empty)")},
|
||||
]}],
|
||||
}
|
||||
if api_key or settings.LITELLM_API_KEY:
|
||||
kwargs["api_key"] = api_key or settings.LITELLM_API_KEY
|
||||
if settings.LITELLM_API_BASE:
|
||||
kwargs["api_base"] = settings.LITELLM_API_BASE
|
||||
|
||||
try:
|
||||
text = litellm.completion(**kwargs).choices[0].message.content or ""
|
||||
except Exception as error: # a failed call must not decide anything
|
||||
return {"belongs": "unclear", "confidence": 0.0, "reason": f"model error: {error}"[:160]}
|
||||
|
||||
match = re.search(r"\{.*\}", text, re.S)
|
||||
if not match:
|
||||
return {"belongs": "unclear", "confidence": 0.0, "reason": "unparseable reply"}
|
||||
try:
|
||||
answer = json.loads(match.group(0))
|
||||
except json.JSONDecodeError:
|
||||
return {"belongs": "unclear", "confidence": 0.0, "reason": "unparseable reply"}
|
||||
return {
|
||||
"belongs": str(answer.get("belongs", "unclear")).lower(),
|
||||
"confidence": float(answer.get("confidence") or 0.0),
|
||||
"reason": str(answer.get("reason", ""))[:200],
|
||||
}
|
||||
|
||||
|
||||
def deterministic_verdict(stem: str, explanation: str) -> str:
|
||||
""""keep" when the stem needs the figure, "remove" when only the explanation
|
||||
mentions one, "unknown" when neither says anything."""
|
||||
stem, explanation = stem or "", explanation or ""
|
||||
if STEM_FIGURE_LABEL.search(stem) or STEM_CUE.search(stem):
|
||||
return "keep"
|
||||
if EXPLANATION_FIGURE_LABEL.search(explanation) or EXPLANATION_CUE.search(explanation):
|
||||
return "remove"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--apply", action="store_true", help="write the changes")
|
||||
parser.add_argument("--no-vision", action="store_true",
|
||||
help="deterministic rule only; ask no model")
|
||||
parser.add_argument("--limit", type=int, default=0, help="classify at most N unknowns")
|
||||
parser.add_argument("--model", help="vision model id (default: the extraction model)")
|
||||
parser.add_argument("--workers", type=int, default=4, help="parallel vision calls")
|
||||
parser.add_argument("--report", default="image_triage_report.txt",
|
||||
help="where the human-review list is written")
|
||||
args = parser.parse_args()
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
questions = db.query(Question).filter(
|
||||
Question.image_path.isnot(None), Question.image_path != ""
|
||||
).order_by(Question.id).all()
|
||||
|
||||
keep, remove, unknown = [], [], []
|
||||
for question in questions:
|
||||
verdict = deterministic_verdict(question.question_text, question.explanation)
|
||||
{"keep": keep, "remove": remove, "unknown": unknown}[verdict].append(question)
|
||||
|
||||
print(f"stem images : {len(questions)}")
|
||||
print(f" stem names a figure : {len(keep)} (left alone)")
|
||||
print(f" only the explanation : {len(remove)} (remove)")
|
||||
print(f" neither : {len(unknown)}")
|
||||
|
||||
judged: dict[int, dict] = {}
|
||||
if unknown and not args.no_vision:
|
||||
model = args.model or get_model_for_task(db, "extraction")[0]
|
||||
api_key = None if args.model else get_model_for_task(db, "extraction")[1]
|
||||
batch = unknown[: args.limit] if args.limit else unknown
|
||||
print(f"\n asking {model} about {len(batch)} image(s)…")
|
||||
payload = [(q.id, q.question_text, q.explanation, q.image_path) for q in batch]
|
||||
with ThreadPoolExecutor(max_workers=args.workers) as pool:
|
||||
results = list(pool.map(
|
||||
lambda row: (row[0], classify(row[1], row[2], row[3], model, api_key)),
|
||||
payload))
|
||||
judged = dict(results)
|
||||
|
||||
# Only a confident "explanation" removes an image. "neither" is a page
|
||||
# header or a logo — junk, but junk that gives nothing away, so it stays
|
||||
# for a person to look at rather than being deleted on a guess.
|
||||
by_id = {q.id: q for q in unknown}
|
||||
vision_removals, uncertain = [], []
|
||||
for qid, answer in judged.items():
|
||||
if answer["belongs"] == "explanation" and answer["confidence"] >= MIN_CONFIDENCE:
|
||||
vision_removals.append((by_id[qid], answer))
|
||||
elif answer["belongs"] != "stem":
|
||||
uncertain.append((by_id[qid], answer))
|
||||
|
||||
if judged:
|
||||
counts: dict[str, int] = {}
|
||||
for answer in judged.values():
|
||||
counts[answer["belongs"]] = counts.get(answer["belongs"], 0) + 1
|
||||
print(" model verdicts : " +
|
||||
", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
|
||||
print(f" confident enough to act : {len(vision_removals)}")
|
||||
print(f" for human review : {len(uncertain)}")
|
||||
|
||||
removals = [(q, {"reason": "only the explanation mentions a figure"}) for q in remove]
|
||||
removals += vision_removals
|
||||
|
||||
for question, answer in removals:
|
||||
print(f" remove q{question.id} {question.image_path} — {answer['reason']}")
|
||||
if args.apply:
|
||||
_snapshot_question(db, question, SCRIPT_EDITOR_ID)
|
||||
question.image_path = None
|
||||
if args.apply:
|
||||
db.commit()
|
||||
|
||||
if uncertain:
|
||||
lines = ["Questions whose stem image could not be classified confidently.",
|
||||
"The image is still attached to the stem; decide by hand.",
|
||||
""]
|
||||
lines += [f"question {q.id}\t{q.image_path}\t{a['belongs']} "
|
||||
f"({a['confidence']:.2f})\t{a['reason']}" for q, a in uncertain]
|
||||
with open(args.report, "w", encoding="utf-8") as handle:
|
||||
handle.write("\n".join(lines) + "\n")
|
||||
print(f"\n {len(uncertain)} left for review -> {args.report}")
|
||||
|
||||
print("\n" + ("APPLIED" if args.apply else "DRY RUN"))
|
||||
print(f" images removed from stems : {len(removals)}")
|
||||
print(f" images kept : {len(questions) - len(removals)}")
|
||||
if not args.apply:
|
||||
print("\n Re-run with --apply to write these changes.")
|
||||
return 0
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
279
backend/tests/test_bank_cleanup.py
Normal file
279
backend/tests/test_bank_cleanup.py
Normal file
|
|
@ -0,0 +1,279 @@
|
|||
"""The two bank-repair scripts: OCR units, lab tables, and stem-image triage.
|
||||
|
||||
Run: DATABASE_URL=sqlite:///:memory: PYTHONPATH=backend python -m unittest discover -s backend/tests.
|
||||
No network and no AI: the vision classifier is never reached, only the
|
||||
deterministic rule that decides most of the images on the text alone.
|
||||
"""
|
||||
import os
|
||||
os.environ.setdefault("DATABASE_URL", "sqlite:///:memory:")
|
||||
|
||||
import sys
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.pool import StaticPool
|
||||
|
||||
from app.database import Base
|
||||
from app.models.course import Course # noqa — Quiz.course_id FK needs the table in metadata.
|
||||
from app.models.media import MediaAsset # noqa — health report walks every embeddable table.
|
||||
from app.models.question import Question, QuestionVersion
|
||||
from app.models.user import User
|
||||
from scripts import fix_lab_formatting as labs
|
||||
from scripts import triage_question_images as images
|
||||
|
||||
# Question 3333 as it came out of the scanner: two mangled units, a lost
|
||||
# superscript, and a per-decilitre spelling of a per-litre value.
|
||||
MANGLED_STEM = (
|
||||
"Laboratory studies demonstrate the following: "
|
||||
"Complete blood cell count, within normal limits; "
|
||||
"White blood cell count, 3,500/µL (3.5 x 109/L); "
|
||||
"Hemoglobin, 13.5 g/dL (135 g/L); "
|
||||
"Chloride, 105 inEq/L (105 mrnol/L); "
|
||||
"Phosphorus, 1.8 mg/dL (0.58 mmol/dL); "
|
||||
"Glucose, 115 mg/dL (6.4 mmol/dL). "
|
||||
"Of the following, you are MOST likely to recommend")
|
||||
|
||||
|
||||
class UnitRepairTests(unittest.TestCase):
|
||||
def test_the_repairs_fire_on_real_mangled_input(self):
|
||||
fixed, fired = labs.repair_units(MANGLED_STEM)
|
||||
self.assertIn("105 mEq/L (105 mmol/L)", fixed)
|
||||
self.assertIn("0.58 mmol/L", fixed)
|
||||
self.assertIn("6.4 mmol/L", fixed)
|
||||
self.assertIn("3.5 × 10⁹/L", fixed)
|
||||
self.assertEqual(len(fired), 4)
|
||||
# Everything that was already right is untouched.
|
||||
self.assertIn("13.5 g/dL (135 g/L)", fixed)
|
||||
self.assertIn("3,500/µL", fixed)
|
||||
|
||||
def test_repairs_are_idempotent(self):
|
||||
once, _ = labs.repair_units(MANGLED_STEM)
|
||||
twice, fired = labs.repair_units(once)
|
||||
self.assertEqual(once, twice)
|
||||
self.assertEqual(fired, [])
|
||||
|
||||
def test_only_real_powers_of_ten_get_a_superscript(self):
|
||||
# 10³, 10⁶, 10⁹ and 10¹² name units; "10¹" is a lost digit, not a lost
|
||||
# superscript, and inventing one would invent a number.
|
||||
for source, expected in [
|
||||
("Platelets, 86 x 103/µL", "86 × 10³/µL"),
|
||||
("Red blood cells, 4.5 x 106/µL", "4.5 × 10⁶/µL"),
|
||||
("Red blood cells, 4.5 x 1012/L", "4.5 × 10¹²/L"),
|
||||
]:
|
||||
self.assertIn(expected, labs.repair_units(source)[0], source)
|
||||
unchanged = "White blood cell count, 10.0 x 101/µL"
|
||||
self.assertEqual(labs.repair_units(unchanged)[0], unchanged)
|
||||
# A count that merely happens to read "100/µL" is not a power of ten.
|
||||
plain = "Eosinophils, 100/µL"
|
||||
self.assertEqual(labs.repair_units(plain)[0], plain)
|
||||
|
||||
def test_conservative_about_ambiguous_units(self):
|
||||
# "3,000/mL" is almost certainly "/µL" and "10A" a mangled exponent,
|
||||
# but neither correction is certain, so neither is made. The lost
|
||||
# superscript beside them is unambiguous and is still fixed.
|
||||
out, _ = labs.repair_units("White blood cells 3,000/mL (3 x 109/L)")
|
||||
self.assertEqual(out, "White blood cells 3,000/mL (3 × 10⁹/L)")
|
||||
for source in ["White blood cell count of 14,600/µL (14.6 x 10A)",
|
||||
"a 4-year-old boy", "the 2013 guideline",
|
||||
"Creatinine, 1.0 mg/dL (88.4 mcmol/L)"]:
|
||||
self.assertEqual(labs.repair_units(source)[0], source, source)
|
||||
|
||||
|
||||
class LabTableTests(unittest.TestCase):
|
||||
def test_a_full_panel_becomes_a_table(self):
|
||||
fixed, _ = labs.repair_units(MANGLED_STEM)
|
||||
out, panels = labs.tabulate_panels(fixed)
|
||||
self.assertEqual(panels, 1)
|
||||
self.assertIn("| Test | Result | SI units |", out)
|
||||
self.assertIn("| Chloride | 105 mEq/L | 105 mmol/L |", out)
|
||||
self.assertIn("| Complete blood cell count | within normal limits | |", out)
|
||||
# The prose on either side survives, and the table stands apart from it.
|
||||
self.assertTrue(out.startswith("Laboratory studies demonstrate the following:"))
|
||||
self.assertTrue(out.rstrip().endswith("you are MOST likely to recommend"))
|
||||
self.assertIn("\n\n|", out)
|
||||
|
||||
def test_tabulating_is_idempotent(self):
|
||||
once, _ = labs.tabulate_panels(labs.repair_units(MANGLED_STEM)[0])
|
||||
twice, panels = labs.tabulate_panels(once)
|
||||
self.assertEqual((twice, panels), (once, 0))
|
||||
|
||||
def test_a_differential_is_left_as_prose(self):
|
||||
# "29% segmented neutrophils, 28% bands" reads the other way round;
|
||||
# pairing name with value here would shift every result by one.
|
||||
text = ("Laboratory data shows: Differential count, 29% segmented neutrophils, "
|
||||
"28% bands, 30% lymphocytes, 11% monocytes, 2% eosinophils")
|
||||
self.assertEqual(labs.tabulate_panels(text), (text, 0))
|
||||
|
||||
def test_a_sentence_of_vital_signs_is_left_as_prose(self):
|
||||
text = ("On examination, his temperature is 37.2°C, heart rate is 96 beats/min, "
|
||||
"and respiratory rate is 18 breaths/min. The boy appears ill.")
|
||||
self.assertEqual(labs.tabulate_panels(text), (text, 0))
|
||||
|
||||
def test_a_short_run_is_left_as_prose(self):
|
||||
text = "Initial studies show: Sodium, 139 mEq/L; Potassium, 3.5 mEq/L. He is admitted."
|
||||
self.assertEqual(labs.tabulate_panels(text), (text, 0))
|
||||
|
||||
def test_a_reference_range_annotates_the_result_above_it(self):
|
||||
text = ("Laboratory tests produce the following results: "
|
||||
"Hemoglobin, 12.8 g/dL (128 g/L); "
|
||||
"White blood cell count, 6,500/µL (6.5 × 10⁹/L); "
|
||||
"Aspartate aminotransferase, 125 U/L; reference range, ≤40 U/L; "
|
||||
"Alanine aminotransferase, 35 U/L; reference range, ≤30 U/L; "
|
||||
"Total bilirubin, 2.0 mg/dL (34.2 µmol/L). Results are normal.")
|
||||
out, panels = labs.tabulate_panels(text)
|
||||
self.assertEqual(panels, 1)
|
||||
self.assertIn("| Aspartate aminotransferase | 125 U/L (reference range, ≤40 U/L) |", out)
|
||||
self.assertNotIn("| reference range |", out)
|
||||
|
||||
def test_an_si_column_only_holds_a_single_value(self):
|
||||
items = [("Sodium", "139 mEq/L", "139 mmol/L")]
|
||||
self.assertIn("| Sodium | 139 mEq/L | 139 mmol/L |", labs.render_table(items))
|
||||
# A differential in brackets opens with a digit too, but it is not SI.
|
||||
text = ("Results: White blood cells, 147/µL (4% segmented, 83% lymphocytes); "
|
||||
"Red blood cells, 10/µL; Glucose, 25 mg/dL (1.4 mmol/L); "
|
||||
"Protein, 179 mg/dL. Cultures are pending.")
|
||||
out, _ = labs.tabulate_panels(text)
|
||||
self.assertIn("| White blood cells | 147/µL (4% segmented, 83% lymphocytes) | |", out)
|
||||
|
||||
|
||||
class SqliteBankTests(unittest.TestCase):
|
||||
"""A disposable bank, with the scripts pointed at it instead of the real DB."""
|
||||
|
||||
def setUp(self):
|
||||
self.engine = create_engine("sqlite://", connect_args={"check_same_thread": False},
|
||||
poolclass=StaticPool)
|
||||
Base.metadata.create_all(self.engine)
|
||||
self.db = Session(self.engine)
|
||||
self.db.add(User(id=1, name="Mod", email="mod@example.test",
|
||||
hashed_password="unused", role="moderator"))
|
||||
self.db.commit()
|
||||
|
||||
def tearDown(self):
|
||||
self.db.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def add_question(self, **fields):
|
||||
question = Question(question_type="mcq", options=["a", "b"], correct_answer="a",
|
||||
is_shared=1, **fields)
|
||||
self.db.add(question)
|
||||
self.db.commit()
|
||||
return question
|
||||
|
||||
def run_script(self, module, *argv):
|
||||
with patch.object(module, "SessionLocal", lambda: self.db), \
|
||||
patch.object(sys, "argv", ["script", *argv]):
|
||||
return module.main()
|
||||
|
||||
|
||||
class LabScriptTests(SqliteBankTests):
|
||||
def test_an_edit_snapshots_the_stem_before_changing_it(self):
|
||||
question = self.add_question(id=1, question_text=MANGLED_STEM,
|
||||
explanation="Refeeding syndrome.")
|
||||
self.assertEqual(self.run_script(labs, "--apply"), 0)
|
||||
|
||||
versions = self.db.query(QuestionVersion).filter_by(question_id=1).all()
|
||||
self.assertEqual(len(versions), 1)
|
||||
# The snapshot holds the text as it was, so the edit can be undone.
|
||||
self.assertEqual(versions[0].snapshot["question_text"], MANGLED_STEM)
|
||||
stem = self.db.get(Question, 1).question_text
|
||||
self.assertIn("105 mEq/L (105 mmol/L)", stem)
|
||||
self.assertNotIn("inEq", stem)
|
||||
|
||||
def test_a_dry_run_writes_nothing(self):
|
||||
self.add_question(id=1, question_text=MANGLED_STEM)
|
||||
self.assertEqual(self.run_script(labs), 0)
|
||||
self.assertEqual(self.db.query(QuestionVersion).count(), 0)
|
||||
self.assertEqual(self.db.get(Question, 1).question_text, MANGLED_STEM)
|
||||
|
||||
def test_a_clean_stem_is_neither_touched_nor_versioned(self):
|
||||
clean = "A 5-year-old girl has a temperature of 38.5°C and a heart rate of 120 beats/min."
|
||||
self.add_question(id=1, question_text=clean)
|
||||
self.assertEqual(self.run_script(labs, "--apply"), 0)
|
||||
self.assertEqual(self.db.query(QuestionVersion).count(), 0)
|
||||
self.assertEqual(self.db.get(Question, 1).question_text, clean)
|
||||
|
||||
def test_tables_are_opt_in(self):
|
||||
self.add_question(id=1, question_text=MANGLED_STEM)
|
||||
self.run_script(labs, "--apply")
|
||||
self.assertNotIn("| Test |", self.db.get(Question, 1).question_text)
|
||||
self.run_script(labs, "--apply", "--tables")
|
||||
self.assertIn("| Test | Result | SI units |", self.db.get(Question, 1).question_text)
|
||||
|
||||
|
||||
class ImageTriageRuleTests(unittest.TestCase):
|
||||
def test_the_rule_fires_only_when_the_explanation_alone_names_a_figure(self):
|
||||
verdict = images.deterministic_verdict
|
||||
self.assertEqual(verdict("The rash shown below is on his trunk.",
|
||||
"The photograph shows erythema migrans."), "keep")
|
||||
self.assertEqual(verdict("He has a rash on his trunk.",
|
||||
"The photograph shows erythema migrans."), "remove")
|
||||
self.assertEqual(verdict("He has a rash on his trunk.",
|
||||
"Erythema migrans is the classic finding."), "unknown")
|
||||
# A stem that names a figure keeps its image even when the critique
|
||||
# names one too — both halves have a figure, and only one is attached.
|
||||
self.assertEqual(verdict("The lesion pictured is tender.",
|
||||
"The figure shows the same lesion."), "keep")
|
||||
|
||||
def test_prep_figure_labels_decide_where_the_figure_was_printed(self):
|
||||
verdict = images.deterministic_verdict
|
||||
# "Item Q22" is printed beside the question, "Item C28" beside the critique.
|
||||
self.assertEqual(verdict("The ankle is in a varus position (Item Q22).",
|
||||
"Clubfoot is treated with casting."), "keep")
|
||||
self.assertEqual(verdict("A mother presents with premature rupture of membranes.",
|
||||
"Survival rises with antenatal steroids (Item C28)."), "remove")
|
||||
# The scanner reads a capital I as a lowercase L.
|
||||
self.assertEqual(verdict("The rash is widespread (ltem Q37A).", ""), "keep")
|
||||
|
||||
def test_empty_text_is_never_a_removal(self):
|
||||
self.assertEqual(images.deterministic_verdict("", ""), "unknown")
|
||||
self.assertEqual(images.deterministic_verdict(None, None), "unknown")
|
||||
|
||||
|
||||
class ImageTriageScriptTests(SqliteBankTests):
|
||||
def test_only_the_explanation_only_case_loses_its_image(self):
|
||||
spoiler = self.add_question(
|
||||
id=1, question_text="He has a rash on his trunk.",
|
||||
explanation="The photograph shows erythema migrans.",
|
||||
image_path="images/doc_5/page_26_img_0.jpeg")
|
||||
needed = self.add_question(
|
||||
id=2, question_text="The rash shown below is on his trunk.",
|
||||
explanation="Erythema migrans.", image_path="images/doc_5/page_27_img_0.jpeg")
|
||||
unknown = self.add_question(
|
||||
id=3, question_text="He has a rash.", explanation="Erythema migrans.",
|
||||
image_path="images/doc_5/page_28_img_0.jpeg")
|
||||
|
||||
spoiler_id, needed_id, unknown_id = spoiler.id, needed.id, unknown.id
|
||||
self.assertEqual(self.run_script(images, "--apply", "--no-vision"), 0)
|
||||
paths = {q.id: q.image_path for q in self.db.query(Question).all()}
|
||||
self.assertIsNone(paths[spoiler_id])
|
||||
self.assertEqual(paths[needed_id], "images/doc_5/page_27_img_0.jpeg")
|
||||
# Without the model there is nothing to say about the unknown, so it stays.
|
||||
self.assertEqual(paths[unknown_id], "images/doc_5/page_28_img_0.jpeg")
|
||||
# The removal is undoable: the path is in the snapshot.
|
||||
version = self.db.query(QuestionVersion).filter_by(question_id=1).one()
|
||||
self.assertEqual(version.snapshot["image_path"], "images/doc_5/page_26_img_0.jpeg")
|
||||
|
||||
def test_a_dry_run_removes_nothing(self):
|
||||
question = self.add_question(
|
||||
id=1, question_text="He has a rash.",
|
||||
explanation="The photograph shows erythema migrans.",
|
||||
image_path="images/doc_5/page_26_img_0.jpeg")
|
||||
question_id = question.id
|
||||
self.assertEqual(self.run_script(images, "--no-vision"), 0)
|
||||
self.assertEqual(self.db.get(Question, question_id).image_path,
|
||||
"images/doc_5/page_26_img_0.jpeg")
|
||||
self.assertEqual(self.db.query(QuestionVersion).count(), 0)
|
||||
|
||||
def test_rerunning_changes_nothing_further(self):
|
||||
self.add_question(id=1, question_text="He has a rash.",
|
||||
explanation="The photograph shows erythema migrans.",
|
||||
image_path="images/doc_5/page_26_img_0.jpeg")
|
||||
self.run_script(images, "--apply", "--no-vision")
|
||||
self.run_script(images, "--apply", "--no-vision")
|
||||
self.assertEqual(self.db.query(QuestionVersion).count(), 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -153,6 +153,14 @@ services:
|
|||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?set MINIO_ROOT_PASSWORD}
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
networks:
|
||||
default:
|
||||
# An unambiguous name. The backend also sits on danvics_milvus, which
|
||||
# has a MinIO of its own called `minio`, and Docker resolved that one
|
||||
# first — every object read failed with InvalidAccessKeyId while the
|
||||
# bucket looked simply empty.
|
||||
aliases:
|
||||
- quiz-minio
|
||||
# No host ports: the backend reaches MinIO over the compose network, and
|
||||
# 9000/9001 are already taken on this host. Publish deliberately if the
|
||||
# console is ever needed from outside.
|
||||
|
|
|
|||
Loading…
Reference in a new issue