From 595b61951557297f2d3c1a54640b5762ac932972 Mon Sep 17 00:00:00 2001 From: Daniel Date: Fri, 11 Sep 2026 05:10:24 +0200 Subject: [PATCH] feat: cross-links follow a rule instead of linking every mention MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 3,560 links became 2,191, and the distribution is the point: Seizures was linked 119 times and is now 48, across the 50 articles that mention it — about one per view, which is what a first mention means. 1. First mention per view, not per section. Short, Long and Clinical are read separately so each earns one; nine sections did not earn nine. Metabolic Acidosis: 39 links to 19. 2. Lists are jump lists, prose is not. A differential or causes list keeps a link on every distinct condition — that is the one place a reader wants ten in a row. 3. The 23 titles mentioned in more than 5% of articles link from lists only. "Seizures may occur" is not a topic anyone breaks off reading to visit; it is the vocabulary of paediatrics. 4. A finished marker is now stashed, so a shorter title cannot re-cut one already made. 6. A link whose label is not the target's title was written by hand and is never touched, which is what makes --apply re-runnable: it strips only its own work and reapplies the rule. Correcting myself: I first measured hub terms on text that was already linked, where the word pattern cannot see a mention wrapped in a marker. That undercounted by four times and made the corpus look far less repetitive than it is. The threshold is measured on clean prose. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN --- backend/scripts/link_articles.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/backend/scripts/link_articles.py b/backend/scripts/link_articles.py index 07bf8b3..352a3c6 100644 --- a/backend/scripts/link_articles.py +++ b/backend/scripts/link_articles.py @@ -19,6 +19,9 @@ stop clicking. The rule now: 3. **Hub terms link from lists only.** A title mentioned across more than HUB_SHARE of all articles is too general to be worth a jump from prose. It still links as a list item, where it is something you might pick. + Measure this on clean prose: counting mentions in text that is already + linked hides most of them behind markers the word pattern will not match, + which made the corpus look four times less repetitive than it is. 4. **Specific beats general.** Longest title first, and a marker once made is protected, so "Otitis media with effusion" cannot be re-cut into "Otitis media". @@ -48,7 +51,10 @@ from app.models.article import Article MIN_TITLE = 4 #: A title mentioned in more than this share of articles links from lists only. -HUB_SHARE = 0.25 +#: At 5% of 333 articles that is 23 terms — Seizures (mentioned in 50), Sepsis +#: (46), Pneumonia (34), Respiratory Distress (34) and the like: the vocabulary +#: of paediatrics rather than a topic anyone would break off reading to visit. +HUB_SHARE = 0.05 #: Terms that are never worth a jump, however specific the match looks. NEVER_LINK = {"history", "examination", "management", "treatment", "prognosis"} @@ -60,8 +66,6 @@ TABLE = re.compile(r"^\s*\|") FENCE = re.compile(r"^\s*(```|~~~)") LIST_ITEM = re.compile(r"^\s*([-*+]|\d+[.)])\s") -LEAD = "__lead__" # summary and the whole-article introduction - def word_pattern(title: str) -> re.Pattern: """Whole-word, case-insensitive, and never biting into an existing marker."""