From ab059673e51d0c694dfdd7b7b18cb87c4fc3c112 Mon Sep 17 00:00:00 2001 From: Daniel Date: Sat, 12 Sep 2026 18:16:59 +0200 Subject: [PATCH] content: every article has a high-yield view, and three ties that were not ties MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Missing views.** The note said 10 articles without a clinical view and 13 without a short one; the real numbers were 8 and 11, and the eight originals turned out to carry no `variant` key at all, so everything in them was being read as the long view. 33 sections written across 11 articles, every sentence traceable to that article's own body, and the legacy sections stamped with the variant they were always being treated as. No doses appear anywhere, because not one of those eight states one. Every article now has a high-yield view and `mdm_pass check` reads 330 of 331 conforming with no placeholder sections. **Pediatric Dentistry** was neither an educator's job nor a delete. Its body is real — scope, disparities, records and consent, public health, biomaterials — and only its clinical view was the disclaimer. The three placeholder sections are gone and the article stays: the reader hides a view an article does not have, and a discipline overview about consent and public health has no bedside path to write. Inventing one was the thing not to do. **The 15 tied conditions were stale**, ticked with the evidence rather than worked: the taxonomy rebuild gave every one of them a single home, and the duplicate-name query now returns nothing. **The 31 tied questions were 3**, and reading all three showed none was a real tie. The best of them: a question about acute flaccid paralysis after minor trauma was filed between "Nursemaid's Elbow" and "Radial Head Subluxation" — two names for one condition, neither anything to do with the question. It is under Spinal Cord Injury now, cross-filed to Acute Flaccid Paralysis, and the two elbow links are removed rather than cross-filed: a wrong link surfaces a question under a topic it does not belong to, which is the opposite of what cross-filing is for. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN --- backend/scripts/settle_tied_primaries.py | 143 ++++++ backend/scripts/write_missing_views.py | 568 +++++++++++++++++++++++ docs/TODO.md | 50 +- docs/tied-conditions.md | 8 + 4 files changed, 757 insertions(+), 12 deletions(-) create mode 100644 backend/scripts/settle_tied_primaries.py create mode 100644 backend/scripts/write_missing_views.py diff --git a/backend/scripts/settle_tied_primaries.py b/backend/scripts/settle_tied_primaries.py new file mode 100644 index 0000000..c58e733 --- /dev/null +++ b/backend/scripts/settle_tied_primaries.py @@ -0,0 +1,143 @@ +"""Give the three questions whose topics tied a primary, by reading them. + +`promote_primary_categories` promotes a question from its system to a condition +when one candidate is the most specific. Where two candidates sit at the same +depth and carry the same number of questions there is nothing in the data to +separate them, so it leaves them alone and reports the count. That count was 31 +before today's taxonomy rebuild and is 3 after it. + +Three is small enough to read. Doing so shows that none of them is the even +contest the counts imply: + +* Q1201 — a nonsurvivable head injury where the apnea test cannot be performed + and the next step is an ancillary cerebral blood flow scan. Brain Death is + what the question teaches; Intracranial Hemorrhage is how the child got there. +* Q853 — painless, large-volume rectal bleeding in a well 8-year-old whose + answer is heterotopic gastric mucosa. Gastrointestinal Bleeding is the + presentation; Juvenile Polyp is one of the wrong options, and stays as a + cross-link because the differential the explanation walks through includes it. +* Q826 — acute flaccid paralysis of both arms after minor trauma, answered by + MRI of the spine. Neither Nursemaid's Elbow nor Radial Head Subluxation has + anything to do with it; they are two names for one condition and the + classifier reached both from "wasn't moving his arms" in a toddler. Those two + links are removed rather than cross-filed, because a wrong link makes the + question surface under a topic it does not belong to. + +Whatever a question moves off stays reachable: the system it was filed under +becomes a `question_category_links` row, and so does the second topic wherever +that topic is genuinely about the question. + +Idempotent, and a dry run by default: + + docker compose exec backend python -m scripts.settle_tied_primaries + docker compose exec backend python -m scripts.settle_tied_primaries --apply +""" +import sys + +from sqlalchemy import text as sa_text + +from app.database import SessionLocal + +#: One decision per question. `system` is the primary it must still be filed +#: under for the decision to apply, `primary` the topic it moves to, `cross` the +#: topics kept as links, and `drop` the links that are simply wrong. Names are +#: carried beside every id so a renumbered taxonomy fails the guard instead of +#: silently filing a question under whatever now holds that id. +DECISIONS = [ + { + "question": 1201, + "system": (14876, "Critical Care"), + "primary": (15297, "Brain Death"), + "cross": [(15438, "Intracranial Hemorrhage")], + "drop": [], + }, + { + "question": 853, + "system": (14881, "Gastroenterology & Nutrition"), + "primary": (15526, "Gastrointestinal Bleeding"), + "cross": [(15891, "Juvenile Polyp")], + "drop": [], + }, + { + "question": 826, + "system": (14879, "Emergency Medicine"), + "primary": (15606, "Spinal Cord Injury"), + "cross": [(15460, "Acute Flaccid Paralysis")], + "drop": [(15697, "Nursemaid's Elbow"), (15699, "Radial Head Subluxation")], + }, +] + + +def category_name(db, category_id: int) -> str | None: + return db.execute(sa_text("SELECT name FROM question_categories WHERE id = :c"), + {"c": category_id}).scalar() + + +def main(apply: bool) -> int: + db = SessionLocal() + try: + planned, skipped = [], [] + for decision in DECISIONS: + qid = decision["question"] + current = db.execute(sa_text( + "SELECT question_category_id FROM questions WHERE id = :q"), {"q": qid}).scalar() + if current is None: + skipped.append((qid, "no such question, or nothing filed")) + continue + if current == decision["primary"][0]: + skipped.append((qid, "already settled")) + continue + if current != decision["system"][0]: + skipped.append((qid, f"filed under {category_name(db, current)}, " + f"not {decision['system'][1]}")) + continue + wrong_name = next( + (f"{cid} is now {category_name(db, cid)!r}, expected {name!r}" + for cid, name in [decision["primary"], decision["system"], + *decision["cross"], *decision["drop"]] + if category_name(db, cid) != name), None) + if wrong_name: + skipped.append((qid, f"taxonomy has moved: {wrong_name}")) + continue + planned.append(decision) + + for decision in planned: + qid = decision["question"] + print(f" Q{qid}: {decision['system'][1]} -> {decision['primary'][1]}") + print(f" cross-filed : {', '.join(n for _c, n in decision['cross'])}, " + f"{decision['system'][1]}") + if decision["drop"]: + print(f" links removed: {', '.join(n for _c, n in decision['drop'])}") + for qid, why in skipped: + print(f" Q{qid} skipped — {why}") + + print(f"\n{len(planned)} questions to settle, {len(skipped)} left alone") + if not apply: + print("\ndry run. Pass --apply to write.") + return 0 + + for decision in planned: + qid = decision["question"] + db.execute(sa_text("UPDATE questions SET question_category_id = :c WHERE id = :q"), + {"c": decision["primary"][0], "q": qid}) + for category_id, _name in [decision["system"], *decision["cross"]]: + db.execute(sa_text(""" + INSERT INTO question_category_links (question_id, category_id) + SELECT :q, :c WHERE NOT EXISTS ( + SELECT 1 FROM question_category_links + WHERE question_id = :q AND category_id = :c) + """), {"q": qid, "c": category_id}) + for category_id, _name in decision["drop"]: + db.execute(sa_text(""" + DELETE FROM question_category_links + WHERE question_id = :q AND category_id = :c + """), {"q": qid, "c": category_id}) + db.commit() + print(f"\nsettled. {len(planned)} questions updated.") + return 0 + finally: + db.close() + + +if __name__ == "__main__": + sys.exit(main("--apply" in sys.argv)) diff --git a/backend/scripts/write_missing_views.py b/backend/scripts/write_missing_views.py new file mode 100644 index 0000000..2fbbf56 --- /dev/null +++ b/backend/scripts/write_missing_views.py @@ -0,0 +1,568 @@ +"""Write the short and clinical views the oldest articles never had. + +Eleven articles are missing a view. Eight of them — Bronchiolitis, Acute otitis +media, Kawasaki disease, Type 1 diabetes, Iron deficiency anemia, Urinary tract +infection, Asthma in children and Developmental milestones — were hand-written +before the three-view system existed, so their sections carry no `variant` at +all and `normalise_sections` reads them as the long view. Three more are +discipline overviews (Pediatric Orthopedics, Pediatric Metabolic Disorders, +Pediatric Infectious Disease) that were given a long and a clinical view but +never a short one. + +Nothing is broken by this — the reader hides a view an article does not have — +but the eight legacy articles are the ones a learner is most likely to open +first, and they are the only articles in the corpus with a single view. + +Every sentence below comes from the body of the article it is added to. No +prevalence, dose or threshold appears here that the article does not already +state; where the source gives no number, the new view gives none either. The +short view is bullets to revise from; the clinical view is the medical +decision-making order `mdm_pass` enforces — Clinical paths, Diagnosis, +Management, and Prognosis and outcome only where the source says something real +about how it ends. + +Pediatric Dentistry is handled here too, in the opposite direction. Its clinical +view is three placeholders that say the source contained no medicine, which was +the honest thing to write and is the wrong thing to keep: a section that +announces the absence of an answer is worse than a view the reader is never +offered. Its long and short views are real content about the discipline, so the +article stays and the placeholder view goes. + +Because it edits `articles.sections`, the section index and the article vectors +go stale. Follow a run with: + + docker compose exec backend python -m scripts.reindex_article_search --apply + +Idempotent, and a dry run by default: + + docker compose exec backend python -m scripts.write_missing_views + docker compose exec backend python -m scripts.write_missing_views --apply +""" +import re +import sys +import uuid + +from sqlalchemy.orm.attributes import flag_modified + +from app.database import SessionLocal +from app.models.article import Article + +#: Long, then short, then clinical — the order every generated article uses. +VARIANT_ORDER = {"long": 0, "short": 1, "clinical": 2} + +#: How much of a section's existing text has to match before it is touched. Long +#: enough to catch an edit, short enough that a trailing tweak does not veto a run. +FINGERPRINT = 60 + + +def slugify(title: str) -> str: + return re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-") + + +def unique_slugs(sections: list[dict]) -> list[dict]: + """Slugs are unique within an article, and three views repeat their titles.""" + seen: set[str] = set() + for section in sections: + base = section.get("slug") or slugify(section.get("title") or "") or "section" + candidate, n = base, 2 + while candidate in seen: + candidate, n = f"{base}-{n}", n + 1 + section["slug"] = candidate + seen.add(candidate) + return sections + + +BRONCHIOLITIS_SHORT = """\ +- Bronchiolitis is the most common lower respiratory tract infection in infants, usually caused by respiratory syncytial virus +- One to three days of coryza precede the cough, tachypnea, wheeze and crackles +- Peak severity is typically day 3-5 of the illness +- Apnea can be the presenting sign in young or preterm infants +- The diagnosis is clinical; routine chest radiographs and viral testing are not needed in a typical case +- Work of breathing, hydration and feeding, and oxygen saturation are what the assessment turns on +- Care is supportive: suctioning, hydration, and oxygen when saturations are persistently low +- Bronchodilators, corticosteroids and antibiotics are not routinely indicated""" + +OTITIS_SHORT = """\ +- Acute otitis media is a middle-ear effusion with acute signs of inflammation +- The diagnosis requires a bulging tympanic membrane, or new otorrhea not due to [[316|otitis externa]], together with acute onset of symptoms +- A cloudy, immobile drum supports the presence of an effusion +- Analgesia is for every child +- Antibiotics are indicated for young infants, for severe symptoms, and for bilateral disease in the very young +- Otherwise observation with follow-up is an option +- Complications are uncommon but important: mastoiditis, tympanic membrane perforation, [[248|hearing loss]] from persistent effusion, and rarely intracranial extension""" + +KAWASAKI_SHORT = """\ +- Kawasaki disease is an acute, self-limited vasculitis of young children +- Fever lasting at least 5 days is the entry criterion +- The accompanying features are bilateral non-exudative conjunctivitis, oral mucosal changes, polymorphous rash, extremity changes, and cervical lymphadenopathy +- Inflammatory markers are raised +- Echocardiography assesses the coronary arteries at diagnosis and on follow-up +- Coronary artery aneurysm is the key concern +- Intravenous immunoglobulin with aspirin, given as early as possible, reduces the risk of coronary aneurysms""" + +DIABETES_SHORT = """\ +- [[270|Type 1 diabetes]] results from autoimmune destruction of pancreatic beta cells and often presents acutely in children +- Polyuria, polydipsia and weight loss develop over days to weeks +- [[384|Nocturnal enuresis]] in a previously dry child is a useful clue +- [[241|Diabetic ketoacidosis]] is hyperglycemia with [[387|metabolic acidosis]] and ketosis +- Kussmaul breathing, vomiting and abdominal pain are common in ketoacidosis +- Cerebral edema is the most feared complication +- Circulation is restored first, fluid and electrolyte deficits are corrected gradually, and an insulin infusion follows the fluids per local protocol""" + +IRON_SHORT = """\ +- Iron deficiency is the most common cause of anemia in young children +- It is usually nutritional in toddlers and related to menstrual loss in adolescents +- The risk factors are excessive cow's milk intake in toddlers, prematurity, low dietary iron, and chronic blood loss +- The picture is a microcytic, hypochromic anemia with a raised red cell distribution width +- Ferritin and transferrin saturation are both low +- Oral iron is given with dietary counseling +- The response is rechecked after several weeks +- A source of blood loss is sought when the history does not fit""" + +UTI_SHORT = """\ +- Urinary tract infection should be considered in any young child with unexplained fever +- Infants present with fever, irritability, vomiting or poor feeding +- Older children describe dysuria, frequency and flank pain +- The diagnosis requires both pyuria and a positive urine culture +- The specimen has to be appropriately collected; bag specimens are unreliable for culture +- Antibiotics are guided by local resistance patterns and adjusted once sensitivities return +- Imaging decisions follow age and recurrence per local guidance""" + +ASTHMA_SHORT = """\ +- Childhood asthma presents with recurrent wheeze, cough and breathlessness that respond to bronchodilators +- The diagnosis is a pattern of recurrent, variable symptoms with documented reversible airflow obstruction +- Spirometry is feasible usually from about age 5 +- An acute exacerbation is treated with inhaled short-acting beta agonists, systemic corticosteroids and oxygen as needed +- The response is reassessed and care is escalated for poor responders +- Chronic control is step therapy based on symptom control +- Inhaler technique and adherence are reviewed at every visit before stepping up""" + +MILESTONES_SHORT = """\ +- Developmental surveillance at every health visit, with standardized screening at set ages, is the basis of early identification +- Gross motor, fine motor, language and social/adaptive development are tracked separately +- Delay in one domain does not always imply global delay +- Loss of previously acquired skills at any age is always a red flag and warrants prompt evaluation +- A standardized tool confirms the diagnosis +- Hearing and vision are checked +- Referral to early intervention services is made while the evaluation proceeds""" + +ORTHOPEDICS_SHORT = """\ +- Bone and joint problems are common presentations in pediatric practice, varying widely in origin, severity and natural history +- Some are apparent from birth; others emerge as the child grows +- The primary care pediatrician's role is to recognize which problems warrant reassurance alone, which need monitoring, and which need referral +- The field covers structural deformities, rotational abnormalities of the lower limbs, angular deformities, and acquired or congenital disorders +- Evaluation begins with history and physical examination, establishing age of onset, whether the condition is progressive, and any associated symptoms or functional limitations +- Many torsional and angular variations of the lower extremities resolve spontaneously without treatment +- Reference values for normal rotational and angular measurements distinguish pathological deformity from normal variation +- Referral to a pediatric orthopedic surgeon is indicated for progressive deformity, significant functional impairment, evaluation beyond the scope of primary care, or where surgery may be necessary""" + +METABOLIC_SHORT = """\ +- Pediatric metabolic disorders span inherited diseases — galactosemia, tyrosinemia, hereditary fructose intolerance, urea cycle defects, fatty acid oxidation disorders — and acquired conditions such as [[333|metabolic syndrome]] +- Presentation may be acute in the neonatal period or insidious over months to years +- Metabolic disorders account for 28-36% of pediatric acute liver failure in infants and young children +- Wilson disease becomes clinically apparent in older children +- Consanguinity, recurrent pregnancy loss, stillbirths or early childhood deaths in the family history raise suspicion, as do [[354|chronic diarrhea]], vomiting, failure to thrive and developmental delay +- Metabolic syndrome requires at least three of six findings: body mass index above the 97th percentile, triglycerides above 110 mg/dL, HDL cholesterol below 40 mg/dL, systolic or diastolic blood pressure above the 90th percentile, fasting glucose above 110 mg/dL or an oral glucose tolerance test result above 140 mg/dL, and waist circumference above the 90th percentile +- Some frameworks make central obesity mandatory, and the International Diabetes Federation uses ethnic-specific waist circumference thresholds +- Neonatal hypoglycemia affects infants with intrauterine growth restriction and infants of diabetic mothers; the duration of hypoglycemia and the delay before treatment influence the risk of seizures +- [[368|Hypocalcemia]] occurs in low birthweight infants, infants of diabetic mothers, asphyxiated infants, DiGeorge syndrome and maternal hyperparathyroidism, often with hypomagnesemia +- [[170|Hyponatremia]] follows improper fluid management or syndrome of inappropriate antidiuretic hormone secretion; hypernatremia follows dehydration from inadequate intake in breastfed infants or incorrect formula dilution +- Pyridoxine dependency presents with seizures resistant to standard anticonvulsants, sometimes with intrauterine convulsions""" + +INFECTIOUS_SHORT = """\ +- Pediatric infectious diseases cover bacterial, viral and other pathogens in children, from common self-limited illnesses to severe, life-threatening infections +- The field spans acute infection, chronic infection, and immunocompromised host considerations specific to children +- Respiratory tract diseases are a significant category of transmission in group settings such as child care and schools +- Some infections show seasonal patterns and others occur year-round +- Immunocompromised children and those with underlying conditions such as congenital heart disease face increased risk of severe infection +- Clinical manifestations vary with the causative organism and the affected body system +- Diagnosis integrates clinical history, physical examination and appropriate testing, with the incubation period guiding clinical suspicion +- Prevention rests on routine childhood immunization, with individualized assessment for high-risk populations +- Treatment is pathogen-specific: antimicrobial therapy, supportive care, and monitoring for complications +- Antimicrobial selection considers age-appropriate dosing, route of administration and local resistance patterns""" + + +#: article_id -> the sections it must currently have, as +#: (variant-or-None, title, opening of the content). A row that does not match +#: has been edited since this was written and is skipped rather than overwritten. +EXPECTED: dict[int, list[tuple[str | None, str, str]]] = { + 3: [ + (None, 'Presentation', 'Preceded by 1–3 days of coryza, then cough, tachypnea, wheez'), + (None, 'Evaluation', 'The diagnosis is clinical. Routine chest radiographs and vir'), + (None, 'Management', 'Supportive care: suctioning, hydration and oxygen when satur'), + ], + 4: [ + (None, 'Diagnosis', 'Requires a bulging tympanic membrane, or new otorrhea not du'), + (None, 'Management', 'Analgesia for every child. Antibiotics are indicated for you'), + (None, 'Complications', 'Uncommon but important: mastoiditis, tympanic membrane perfo'), + ], + 5: [ + (None, 'Clinical features', 'Fever lasting at least 5 days plus features such as bilatera'), + (None, 'Evaluation', 'Inflammatory markers are raised. Echocardiography assesses t'), + (None, 'Management', 'Intravenous immunoglobulin with aspirin, given as early as p'), + ], + 6: [ + (None, 'Presentation', 'Polyuria, polydipsia and weight loss over days to weeks. [[3'), + (None, 'Diabetic ketoacidosis', 'Hyperglycemia with [[387|metabolic acidosis]] and ketosis. K'), + (None, 'Initial management', 'Circulation is restored, fluid and electrolyte deficits are '), + ], + 7: [ + (None, 'Risk factors', "Excessive cow's milk intake in toddlers, prematurity, low di"), + (None, 'Laboratory findings', 'Microcytic, hypochromic anemia with a raised red cell distri'), + (None, 'Management', 'Oral iron is given with dietary counselling; the response is'), + ], + 8: [ + (None, 'Presentation', 'Infants present with fever, irritability, vomiting or poor f'), + (None, 'Diagnosis', 'Requires both pyuria and a positive urine culture from an ap'), + (None, 'Management', 'Antibiotics guided by local resistance patterns, adjusted on'), + ], + 9: [ + (None, 'Diagnosis', 'A pattern of recurrent, variable symptoms with documented re'), + (None, 'Acute exacerbation', 'Inhaled short-acting beta agonists, systemic corticosteroids'), + (None, 'Chronic control', 'Step therapy based on symptom control, with inhaler techniqu'), + ], + 10: [ + (None, 'Domains', 'Gross motor, fine motor, language and social/adaptive develo'), + (None, 'Red flags', 'Loss of previously acquired skills at any age is always a re'), + (None, 'Next steps', 'Diagnosis is confirmed with a standardised tool, hearing and'), + ], + 87: [ + ('long', 'Introduction', 'Bone and joint problems are common presentations in pediatri'), + ('long', 'Scope of Pediatric Orthopedics', 'Pediatric orthopedic conditions encompass a broad spectrum o'), + ('long', 'Clinical Approach', 'Evaluation of a child with an orthopedic problem begins with'), + ('long', 'Lower Extremity Problems', 'Lower extremity abnormalities are among the most frequently '), + ('long', 'Management Principles', 'Management of pediatric orthopedic problems ranges from obse'), + ('long', 'Specialist Referral', 'Pediatricians should recognize when referral to a pediatric '), + ('clinical', 'Clinical paths', '**Initial Evaluation**\n\nWhen a child presents with an orthop'), + ('clinical', 'Diagnosis', '**Physical examination**: Any deformities are measured and d'), + ('clinical', 'Management', '**Decision-Making**\n\n- **Reassurance and observation**: Many'), + ], + 91: [ + ('long', 'Definition', 'Pediatric metabolic disorders represent a broad category of '), + ('long', 'Epidemiology and Clinical Presentation', 'Metabolic disorders account for a substantial proportion of '), + ('long', 'Metabolic Syndrome in Children and Adolescents', 'Metabolic syndrome represents a clustering of cardiovascular'), + ('long', 'Neonatal Metabolic Disturbances', 'Several metabolic derangements occur commonly in the neonata'), + ('long', 'Diagnostic Approach', 'Diagnosis of inherited metabolic diseases requires a systema'), + ('clinical', 'Clinical paths', '**[[368|Hypocalcemia]]**\n- Infants are screened for [[368|hy'), + ('clinical', 'Diagnosis', '**Hypoglycemia**\n- Blood glucose is assessed urgently in at-'), + ('clinical', 'Management', '**Hypoglycemia**\n- Early treatment initiation is critical to'), + ('clinical', 'Prognosis and outcome', '**Hypoglycemia**\n- Duration of [[321|hypoglycemia]] and time'), + ], + 95: [ + ('long', 'Definition and Scope', 'Pediatric infectious diseases represent infections caused by'), + ('long', 'Epidemiology and Transmission', 'Infectious diseases in children spread through multiple rout'), + ('long', 'Clinical Presentation and Diagnosis', 'Clinical manifestations of pediatric infectious diseases var'), + ('long', 'Prevention and Immunization', 'Prevention of pediatric infectious diseases relies heavily o'), + ('long', 'Treatment Principles', 'Management of pediatric infectious diseases is pathogen-spec'), + ('clinical', 'Clinical paths', '**Initial Evaluation**\n\n- A detailed history including expos'), + ('clinical', 'Diagnosis', '**Diagnostic Approach**\n\n- Pathogen-specific diagnostic test'), + ('clinical', 'Management', '**Treatment Considerations**\n\n- Antimicrobial therapy is sel'), + ], +} + + +#: article_id -> (variant, title, content) to append. Order within a variant is +#: the order the reader gets. +ADDITIONS: dict[int, list[tuple[str, str, str]]] = { + 3: [ + ("short", "In short", BRONCHIOLITIS_SHORT), + ("clinical", "Clinical paths", + "1. An infant presents after 1-3 days of coryza with cough, tachypnea, wheeze and crackles\n" + "2. The day of illness is established, because peak severity is typically day 3-5\n\n" + "**Branch point**\n\n" + "In a young or preterm infant, apnea can itself be the presenting sign."), + ("clinical", "Diagnosis", + "The diagnosis is clinical. Routine chest radiographs and viral testing are not needed " + "for typical cases.\n\n" + "Three things are assessed:\n" + "- Work of breathing\n" + "- Hydration and feeding\n" + "- Oxygen saturation"), + ("clinical", "Management", + "Care is supportive:\n\n" + "1. Suctioning is performed\n" + "2. Hydration is maintained\n" + "3. Oxygen is given when saturations are persistently low\n\n" + "Bronchodilators, corticosteroids and antibiotics are not routinely indicated."), + ], + 4: [ + ("short", "In short", OTITIS_SHORT), + ("clinical", "Clinical paths", + "1. A child presents with acute onset of symptoms referable to the ear\n" + "2. The tympanic membrane is examined for bulging, for new otorrhea, and for the " + "cloudy, immobile appearance of an effusion\n\n" + "**Branch point**\n\n" + "Once the diagnosis is made the choice is between antibiotics and observation with " + "follow-up, decided on the child's age, the severity of symptoms, and whether disease " + "is bilateral."), + ("clinical", "Diagnosis", + "The diagnosis requires a bulging tympanic membrane, or new otorrhea not due to " + "[[316|otitis externa]], together with acute onset of symptoms. A cloudy, immobile " + "drum supports the presence of an effusion."), + ("clinical", "Management", + "1. Analgesia is given to every child\n\n" + "2. Antibiotics are given where indicated:\n" + " - Young infants\n" + " - Severe symptoms\n" + " - Bilateral disease in the very young\n\n" + "3. Where none of those applies, observation with follow-up is an option"), + ("clinical", "Prognosis and outcome", + "Complications are uncommon but important: mastoiditis, tympanic membrane perforation, " + "[[248|hearing loss]] from persistent effusion, and rarely intracranial extension."), + ], + 5: [ + ("short", "In short", KAWASAKI_SHORT), + ("clinical", "Clinical paths", + "1. A young child presents with fever lasting at least 5 days\n" + "2. The accompanying features are looked for:\n" + " - Bilateral non-exudative conjunctivitis\n" + " - Oral mucosal changes\n" + " - Polymorphous rash\n" + " - Extremity changes\n" + " - Cervical lymphadenopathy"), + ("clinical", "Diagnosis", + "Inflammatory markers are raised.\n\n" + "Echocardiography assesses the coronary arteries at diagnosis and again on follow-up, " + "because coronary artery aneurysm is the key concern."), + ("clinical", "Management", + "Intravenous immunoglobulin with aspirin is given as early as possible, which reduces " + "the risk of coronary aneurysms."), + ], + 6: [ + ("short", "In short", DIABETES_SHORT), + ("clinical", "Clinical paths", + "1. A child presents with polyuria, polydipsia and weight loss over days to weeks; " + "[[384|nocturnal enuresis]] in a previously dry child is a useful clue\n\n" + "**Branch point**\n\n" + "Where there is hyperglycemia with [[387|metabolic acidosis]] and ketosis, the " + "presentation is [[241|diabetic ketoacidosis]]. Kussmaul breathing, vomiting and " + "abdominal pain are common on this path."), + ("clinical", "Diagnosis", + "Diabetic ketoacidosis is hyperglycemia with metabolic acidosis and ketosis."), + ("clinical", "Management", + "1. Circulation is restored\n" + "2. Fluid and electrolyte deficits are corrected gradually\n" + "3. An insulin infusion is started after fluids, per local protocol\n\n" + "Cerebral edema is the most feared complication."), + ], + 7: [ + ("short", "In short", IRON_SHORT), + ("clinical", "Clinical paths", + "1. The history establishes which route to iron deficiency is in play:\n" + " - Toddlers: excessive cow's milk intake, low dietary iron, prematurity\n" + " - Adolescents: menstrual loss\n" + " - Any age: chronic blood loss\n\n" + "**Branch point**\n\n" + "Where the history does not fit, a source of blood loss is sought."), + ("clinical", "Diagnosis", + "A microcytic, hypochromic anemia with a raised red cell distribution width, low " + "ferritin and low transferrin saturation."), + ("clinical", "Management", + "1. Oral iron is given\n" + "2. Dietary counseling accompanies it\n" + "3. The response is rechecked after several weeks"), + ], + 8: [ + ("short", "In short", UTI_SHORT), + ("clinical", "Clinical paths", + "1. Urinary tract infection is considered in any young child with unexplained fever\n" + "2. The presentation depends on age:\n" + " - Infants: fever, irritability, vomiting or poor feeding\n" + " - Older children: dysuria, frequency and flank pain"), + ("clinical", "Diagnosis", + "1. Urine is obtained by an appropriate collection method; bag specimens are " + "unreliable for culture\n" + "2. Both pyuria and a positive urine culture are required"), + ("clinical", "Management", + "1. Antibiotics are started according to local resistance patterns\n" + "2. The choice is adjusted once sensitivities return\n" + "3. Imaging decisions follow the child's age and whether infection has recurred, " + "per local guidance"), + ], + 9: [ + ("short", "In short", ASTHMA_SHORT), + ("clinical", "Clinical paths", + "**Branch point**\n\n" + "Two presentations, managed differently:\n\n" + "- The acute exacerbation, needing treatment and reassessment\n" + "- The routine review, where chronic control is adjusted"), + ("clinical", "Diagnosis", + "A pattern of recurrent, variable symptoms with documented reversible airflow " + "obstruction where spirometry is feasible, usually from about age 5."), + ("clinical", "Management", + "**Acute exacerbation**\n\n" + "1. Inhaled short-acting beta agonists, systemic corticosteroids and oxygen are " + "given as needed\n" + "2. The response is reassessed\n" + "3. Care is escalated for poor responders\n\n" + "**Chronic control**\n\n" + "Step therapy is based on symptom control. Inhaler technique and adherence are " + "reviewed at every visit before stepping up."), + ], + 10: [ + ("short", "In short", MILESTONES_SHORT), + ("clinical", "Clinical paths", + "1. Developmental surveillance is carried out at every health visit, with " + "standardized screening at set ages\n" + "2. A concern is placed by domain — gross motor, fine motor, language, " + "social/adaptive — because delay in one domain does not always imply global delay\n\n" + "**Branch point**\n\n" + "Loss of previously acquired skills at any age is always a red flag and warrants " + "prompt evaluation."), + ("clinical", "Diagnosis", + "1. The diagnosis is confirmed with a standardized tool\n" + "2. Hearing and vision are checked"), + ("clinical", "Management", + "Referral to early intervention services is made while the evaluation proceeds."), + ], + 87: [("short", "In short", ORTHOPEDICS_SHORT)], + 91: [("short", "In short", METABOLIC_SHORT)], + 95: [("short", "In short", INFECTIOUS_SHORT)], +} + +#: (article_id, section title, exact current content, replacement). Two British +#: spellings in articles this script is already rewriting, in a corpus and for a +#: board that are American. Both articles' summaries already say counseling and +#: standardized; only the section text was left behind. +RESPELL: list[tuple[int, str, str, str]] = [ + (7, "Management", + "Oral iron is given with dietary counselling; the response is rechecked after several " + "weeks. A source of blood loss is sought when the history does not fit.", + "Oral iron is given with dietary counseling; the response is rechecked after several " + "weeks. A source of blood loss is sought when the history does not fit."), + (10, "Next steps", + "Diagnosis is confirmed with a standardised tool, hearing and vision are checked, and " + "referral to early intervention services is made while the evaluation proceeds.", + "Diagnosis is confirmed with a standardized tool, hearing and vision are checked, and " + "referral to early intervention services is made while the evaluation proceeds."), +] + +#: The placeholder clinical view to remove, as (title, exact content). Matched in +#: full rather than by prefix: this is a deletion, and a deletion earns the +#: stricter guard. +DENTISTRY_ID = 176 +DENTISTRY_PLACEHOLDERS: list[tuple[str, str]] = [ + ("Clinical paths", "No presentation details are available."), + ("Diagnosis", "No diagnostic protocols are available."), + ("Management", + "The passages provided do not contain specific clinical management protocols, drug " + "dosages, routes of administration, or step-by-step bedside procedures. To provide " + "accurate clinical guidance with specific doses and management sequences, additional " + "reference material containing treatment protocols and pharmacological data would be " + "required."), +] + + +def plan_article(article: Article) -> tuple[list[dict] | None, str]: + """The article's new sections, or None and the reason for leaving it alone.""" + sections = [dict(s) for s in (article.sections or [])] + additions = ADDITIONS[article.id] + wanted = {variant for variant, _t, _c in additions} + present = {s.get("variant") for s in sections} + if wanted <= present: + return None, "already written" + + expected = EXPECTED.get(article.id) + if expected is not None: + actual = [(s.get("variant"), s.get("title"), (s.get("content") or "")[:FINGERPRINT]) + for s in sections] + if actual != expected: + return None, "sections have changed since this script was written" + for section in sections: + # The legacy eight predate variants; long is what they have always + # rendered as. The overviews already carry theirs and keep them. + if section.get("variant") not in VARIANT_ORDER: + section["variant"] = "long" + section.setdefault("parent_id", None) + + for article_id, title, old, new in RESPELL: + if article_id != article.id: + continue + for section in sections: + if section.get("title") == title and section.get("content") == old: + section["content"] = new + + for variant, title, content in additions: + sections.append({ + "id": uuid.uuid4().hex, + "slug": slugify(title)[:60], + "title": title, + "content": content, + "parent_id": None, + "variant": variant, + }) + sections.sort(key=lambda s: VARIANT_ORDER.get(s.get("variant"), 0)) + return unique_slugs(sections), "" + + +def plan_dentistry(article: Article) -> tuple[list[dict] | None, str]: + """Pediatric Dentistry with its placeholder clinical view removed.""" + sections = [dict(s) for s in (article.sections or [])] + clinical = [s for s in sections if s.get("variant") == "clinical"] + if not clinical: + return None, "placeholder view already removed" + actual = [(s.get("title"), (s.get("content") or "").strip()) for s in clinical] + if actual != DENTISTRY_PLACEHOLDERS: + return None, "clinical view is no longer the placeholder text" + return [s for s in sections if s.get("variant") != "clinical"], "" + + +def main(apply: bool) -> int: + db = SessionLocal() + try: + writes: list[tuple[Article, list[dict], str]] = [] + skipped: list[tuple[int, str, str]] = [] + + for article_id in sorted(ADDITIONS): + article = db.get(Article, article_id) + if article is None: + skipped.append((article_id, "?", "no such article")) + continue + sections, why = plan_article(article) + if sections is None: + skipped.append((article_id, article.title, why)) + else: + added = ", ".join(sorted({v for v, _t, _c in ADDITIONS[article_id]})) + writes.append((article, sections, f"+{added}")) + + dentistry = db.get(Article, DENTISTRY_ID) + if dentistry is None: + skipped.append((DENTISTRY_ID, "?", "no such article")) + else: + sections, why = plan_dentistry(dentistry) + if sections is None: + skipped.append((DENTISTRY_ID, dentistry.title, why)) + else: + writes.append((dentistry, sections, "-clinical (placeholder)")) + + for article, sections, what in writes: + before = {s.get("variant") or "long" for s in (article.sections or [])} + after = {s.get("variant") for s in sections} + print(f" #{article.id} {article.title} {what}") + print(f" views {sorted(before)} -> {sorted(after)}") + for section in sections: + if section.get("variant") in ("short", "clinical") or article.id == DENTISTRY_ID: + head = (section.get("content") or "").split("\n")[0][:88] + print(f" {section['variant']:8} {section['title']:22} {head}") + print() + for article_id, title, why in skipped: + print(f" #{article_id} {title} — skipped, {why}") + + print(f"\n{len(writes)} articles to rewrite, {len(skipped)} left alone") + if not apply: + print("\ndry run. Pass --apply to write.") + return 0 + + for article, sections, _what in writes: + article.sections = sections + flag_modified(article, "sections") + db.commit() + print(f"\nwritten. {len(writes)} articles updated.") + print("Now re-run: python -m scripts.reindex_article_search --apply") + return 0 + finally: + db.close() + + +if __name__ == "__main__": + sys.exit(main("--apply" in sys.argv)) diff --git a/docs/TODO.md b/docs/TODO.md index ec90265..b6ccca1 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -55,13 +55,25 @@ Captured so nothing is lost while the article writing runs. 323 got one). It was 413 free-form sections under 242 different titles. `scripts/mdm_pass.py` does the moving and refuses any article that lost a cross-reference, changed a fact, or came back outside 0.8-1.3x. -- [ ] **Pediatric Dentistry has no clinical content** — its source was a - disclaimer, so Clinical paths and Diagnosis say so rather than inventing - medicine. `mdm_pass check` lists it. Needs an educator or a delete. -- [ ] **10 articles have no clinical view at all** and 13 no high-yield view — the - original hand-written ones (Bronchiolitis, Kawasaki disease, Acute otitis - media …) predate the three-view system. The reader hides a view an - article does not have, so nothing is broken; they are simply thinner. +- [x] **Pediatric Dentistry's placeholder clinical view removed** — done + 2026-09-12. Neither an educator nor a delete: the article is a discipline + overview whose long and high-yield views are real content about the + discipline, and only its clinical view was three sections announcing that + the source held no medicine. The article stays and the placeholder view + goes, so the reader is offered two views instead of three that include a + disclaimer. `mdm_pass check` reports nothing. +- [x] **The 11 articles missing a view now have one** — done 2026-09-12. The + note said 10 with no clinical view and 13 with no high-yield; the real + numbers on the day were 8 and 11. The eight hand-written originals + (Bronchiolitis, Acute otitis media, Kawasaki disease, Type 1 diabetes, + Iron deficiency anemia, Urinary tract infection, Asthma in children, + Developmental milestones) had no `variant` at all and read as the long + view; the three discipline overviews had long and clinical but no + high-yield. `scripts/write_missing_views.py` wrote 33 new sections from + each article's own body — no number, dose or threshold appears that the + article does not already state — and stamped the legacy sections `long`. + Clinical views are in MDM order, so `mdm_pass check` reads 330 of 331 with + a clinical view, all conforming. Every article now has a high-yield view. ### Reading and study - [x] **Recommendations by Articles / Disciplines / Systems** — done @@ -433,10 +445,13 @@ Two screenshots, one flow. safety property as the earlier pass — each moved question keeps a `question_category_links` row to the system it came from, so no filter narrows. 786 categories to 662; 0 questions unfiled, 0 orphan links. -- [ ] **15 genuine ties need a human call.** An even split has no - evidence-based winner, so nothing was merged. Listed in - `docs/tied-conditions.md`; pick a home for each and the merge is one - command. +- [x] **The 15 genuine ties are gone** — closed 2026-09-12 as stale, not + decided. The taxonomy rebuild gave every condition one home, so there is + no longer a duplicate to merge: `merge_duplicate_conditions` reports 0 + duplicated names and 0 needing a human call, and each of the 15 names in + `docs/tied-conditions.md` now resolves to exactly one row (Chlamydial + Infection became Chlamydia Infection under Adolescent Medicine). The doc + is kept as the record of a decision that no longer needs making. - [x] **Primary categories pushed down** — done 2026-09-10. 1,244 questions promoted from system to condition; 2,730 → 1,486 filed at system level. The system stays as a cross-link so no filter narrows. @@ -450,7 +465,18 @@ Two screenshots, one flow. instead, and `question_tag_links` holds zero rows of type `system`. Every one of the 569 topics carries a system and all 2,922 questions reach one through their topic, so there is no longer a per-question pick to review. -- [ ] **31 questions tie between equally specific conditions** — left alone. +- [x] **The tied conditions are settled** — done 2026-09-12. The rebuild took + the tie count from 31 to 3, which is few enough to read. None was the even + contest the counts implied: Q1201 teaches brain death determination rather + than the intracranial hemorrhage that caused it, Q853 is gastrointestinal + bleeding with juvenile polyp as a distractor, and Q826 was tagged + Nursemaid's Elbow and Radial Head Subluxation — two names for one + condition, and neither of them what a question about acute flaccid + paralysis and MRI of the spine is about. `scripts/settle_tied_primaries.py` + files each under one topic, cross-links the second and the old system, and + drops the two wrong elbow links. `promote_primary_categories` now reports + 0 ambiguous. Nursemaid's Elbow and Radial Head Subluxation are still two + topic rows for one idea and want merging. - [x] **Real organ systems, and symptoms grouped under them** — done 2026-09-10. The 27 "systems" were disciplines; that tree is now the Topics facet, and Systems is a new flat axis of 16 organ systems. `question_tags` gained diff --git a/docs/tied-conditions.md b/docs/tied-conditions.md index 7c997ff..ede2ede 100644 --- a/docs/tied-conditions.md +++ b/docs/tied-conditions.md @@ -1,5 +1,13 @@ # The 15 conditions with no evidence-based home +> **Settled 2026-09-12 — nothing here needs deciding any more.** The taxonomy +> rebuild gave every condition a single home, so none of these fifteen is +> duplicated across systems now and `merge_duplicate_conditions` reports zero +> names needing a human call. Chlamydial Infection came back as Chlamydia +> Infection under Adolescent Medicine; the other fourteen kept their names. The +> table below is left as the record of the decision that was never needed. + + Every other duplicated condition name had a clear plurality — one system held most of its questions — and 99 were merged into it, keeping a cross-link to the systems folded away so no filter narrows.