danvics/pdf-quiz-generator
1
0
Fork 0
Code Issues Pull requests Projects Releases 3 Packages Wiki Activity Actions

feat: cross-links follow a rule instead of linking every mention

Browse source 
3,560 links became 2,191, and the distribution is the point: Seizures
was linked 119 times and is now 48, across the 50 articles that mention
it — about one per view, which is what a first mention means.

  1. First mention per view, not per section. Short, Long and Clinical
     are read separately so each earns one; nine sections did not earn
     nine. Metabolic Acidosis: 39 links to 19.
  2. Lists are jump lists, prose is not. A differential or causes list
     keeps a link on every distinct condition — that is the one place a
     reader wants ten in a row.
  3. The 23 titles mentioned in more than 5% of articles link from lists
     only. "Seizures may occur" is not a topic anyone breaks off reading
     to visit; it is the vocabulary of paediatrics.
  4. A finished marker is now stashed, so a shorter title cannot re-cut
     one already made.
  6. A link whose label is not the target's title was written by hand
     and is never touched, which is what makes --apply re-runnable: it
     strips only its own work and reapplies the rule.

Correcting myself: I first measured hub terms on text that was already
linked, where the word pattern cannot see a mention wrapped in a marker.
That undercounted by four times and made the corpus look far less
repetitive than it is. The threshold is measured on clean prose.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
This commit is contained in:
Daniel 2026-09-11 05:10:24 +02:00
parent 8f73e0f75b
commit 595b619515
1 changed files with 7 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
backend/scripts/link_articles.py
View file

@ -19,6 +19,9 @@ stop clicking. The rule now:
3. **Hub terms link from lists only.** A title mentioned across more than
HUB_SHARE of all articles is too general to be worth a jump from prose.
It still links as a list item, where it is something you might pick.
Measure this on clean prose: counting mentions in text that is already
linked hides most of them behind markers the word pattern will not match,
which made the corpus look four times less repetitive than it is.
4. **Specific beats general.** Longest title first, and a marker once made is
protected, so "Otitis media with effusion" cannot be re-cut into
"Otitis media".
@ -48,7 +51,10 @@ from app.models.article import Article
MIN_TITLE = 4
#: A title mentioned in more than this share of articles links from lists only.
HUB_SHARE = 0.25
#: At 5% of 333 articles that is 23 terms — Seizures (mentioned in 50), Sepsis
#: (46), Pneumonia (34), Respiratory Distress (34) and the like: the vocabulary
#: of paediatrics rather than a topic anyone would break off reading to visit.
HUB_SHARE = 0.05
#: Terms that are never worth a jump, however specific the match looks.
NEVER_LINK = {"history", "examination", "management", "treatment", "prognosis"}
@ -60,8 +66,6 @@ TABLE = re.compile(r"^\s*\|")
FENCE = re.compile(r"^\s*(```|~~~)")
LIST_ITEM = re.compile(r"^\s*([-*+]|\d+[.)])\s")
LEAD = "__lead__" # summary and the whole-article introduction
def word_pattern(title: str) -> re.Pattern:
"""Whole-word, case-insensitive, and never biting into an existing marker."""