7cd2fa5cf4
Measured first, by the ped-ai session, fifteen runs of five prompts with the gateway cache bypassed. Retrieval was already deterministic: identical shortlist and identical scores every time, and the citation checker stripped none of the 45 markers written — invented citations are not the problem here. Generation was the whole variance. At temperature 0.3 the same sources and the same prompt gave answers differing by 15-70% of their text; one differential swung between a 35-word uncited paraphrase and a 180-word cited list. So temperature 0 and a seed. Temperature 0 alone was not enough — three runs still differed — and temperature 0 with a fixed seed came back byte-identical. The seed is derived from the question, normalised for case and spacing, so two people asking the same thing get the same answer and a different question is not pinned to the same sample. An empty reply is asked once more before it becomes a 502. One in fifteen came back empty from a healthy model in 4.9 seconds — not a refusal, not an error, just nothing. A short query that finds almost nothing is retried against the nearest article title. "kawasaki criteria" finds fourteen sources; "kawasaki critera" found none — the lexical ranker cannot match a token that is in no index, and the embedding of a misspelling is not near the embedding of the word. Trigrams do not care: that typo scores 0.36 against "Kawasaki disease" with the next article at 0.11, and the gap is what makes it safe to act on. pg_trgm is created at startup beside vector, with a migration for the record. And an answer drawn from the library must cite it. Not a hallucination guard — nothing was stripped in fifteen runs — but one answer used the sources and cited none of them, which leaves the learner an assertion and nowhere to check it. Also, article drafts are weighted towards mechanism, in the wording the ped-ai rewriter is using, so the two lanes read alike: why the body does what it does, with features and management explained through it rather than listed. Figure lines and cross-references survive a refine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
626 lines
28 KiB
Python
626 lines
28 KiB
Python
"""Retrieval and citation handling for AI Mode.
|
||
|
||
The safety property, stated plainly: **retrieval decides what may be cited, and
|
||
the server enforces it.** The model is handed a numbered shortlist and told to
|
||
cite from it by id. Afterwards every citation it wrote is checked against that
|
||
shortlist and anything else is deleted. A citation the model invented cannot
|
||
survive, because surviving is not a thing the model gets to decide.
|
||
|
||
This is the same discipline as the article page not printing answers: a property
|
||
the system holds, not one the model is trusted to respect.
|
||
|
||
Two kinds of link exist and are treated differently. *Curated* links are rows an
|
||
educator created — a question tied to an article section, a card to a question.
|
||
They are assertions, so they are trusted, and a retrieved row that carries one to
|
||
another retrieved row is boosted: two things an educator already tied together
|
||
answering the same query is evidence, not coincidence. *Retrieved* links are
|
||
ranked guesses computed per query and stored nowhere.
|
||
|
||
The shortlist is chosen twice. Each corpus offers twice as many candidates as it
|
||
can have places for, and a cross-encoder reads all of them against the query and
|
||
decides which survive and in what order. Unlike everywhere else on the site,
|
||
where reranking is a strict permutation of a page of results, here it selects —
|
||
because here the shortlist is not a page somebody skims past but the entire
|
||
evidence an answer may be built from, and a source that does not make it is one
|
||
the model cannot lean on. If no reranker answers, each corpus keeps the six it
|
||
elected, in the order it elected them, exactly as before there was one.
|
||
"""
|
||
import logging
|
||
import re
|
||
|
||
from sqlalchemy import text as sa_text
|
||
from sqlalchemy.orm import Session
|
||
|
||
from app.models.article import Article, ArticleSectionIndex, QuestionArticleLink
|
||
from app.models.flashcard import Flashcard, FlashcardDeck
|
||
from app.models.question import Question
|
||
from app.models.user import User
|
||
from app.services import rerank_service
|
||
from app.services.quiz_builder import bank_query, exam_scope_predicate
|
||
from app.services.search_service import hybrid_ids, top_similarity
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
# How many of each kind retrieval offers the model. Small on purpose: a
|
||
# shortlist the model can hold is worth more than a corpus it skims.
|
||
PER_KIND = 6
|
||
# How many of each kind the cross-encoder gets to choose from. Handing it only
|
||
# the six a corpus already elected would leave the membership of the shortlist
|
||
# entirely to rankers that never read the query against the document; twelve
|
||
# lets a rank-nine section reach the answer, and keeps the union of four corpora
|
||
# inside a single rerank call.
|
||
CANDIDATES_PER_KIND = PER_KIND * 2
|
||
MAX_SOURCES = 14
|
||
# An excerpt long enough to answer from, short enough that fourteen of them fit.
|
||
EXCERPT_CHARS = 700
|
||
# A curated tie between two retrieved rows is evidence an educator left behind.
|
||
CURATED_BOOST = 0.5
|
||
|
||
CITATION_RE = re.compile(r"\[\[(article|section|question|card):([A-Za-z0-9#_-]+)\]\]")
|
||
|
||
|
||
def _clean(value: str | None, limit: int = EXCERPT_CHARS) -> str:
|
||
text = re.sub(r"!\[[^\]]*\]\([^)]*\)", " ", value or "")
|
||
text = re.sub(r"\[([^\]]*)\]\([^)]*\)", r"\1", text)
|
||
text = re.sub(r"<[^>]+>", " ", text)
|
||
text = re.sub(r"\s+", " ", text).strip()
|
||
return text[:limit] + ("…" if len(text) > limit else "")
|
||
|
||
|
||
def _articles(db: Session, user: User, query: str) -> list[dict]:
|
||
ranked, _ = hybrid_ids(db, query, "article", limit=PER_KIND * 3)
|
||
if not ranked:
|
||
return []
|
||
rows = db.query(Article).filter(Article.id.in_(ranked)).all()
|
||
if not user.is_moderator:
|
||
rows = [a for a in rows if a.status == "published" or a.user_id == user.id]
|
||
order = {rid: i for i, rid in enumerate(ranked)}
|
||
rows.sort(key=lambda a: order.get(a.id, len(order)))
|
||
return [{
|
||
"kind": "article", "ref": str(a.id), "id": a.id,
|
||
"title": a.title,
|
||
"text": _clean(a.summary or a.content),
|
||
"score": 1.0 / (1 + order.get(a.id, 0)),
|
||
} for a in rows[:CANDIDATES_PER_KIND]]
|
||
|
||
|
||
def _sections(db: Session, user: User, query: str) -> list[dict]:
|
||
ranked, _ = hybrid_ids(db, query, "article_section", limit=PER_KIND * 3)
|
||
if not ranked:
|
||
return []
|
||
rows = db.query(ArticleSectionIndex).filter(ArticleSectionIndex.id.in_(ranked)).all()
|
||
if not rows:
|
||
return []
|
||
articles = {a.id: a for a in db.query(Article).filter(
|
||
Article.id.in_({r.article_id for r in rows})).all()}
|
||
order = {rid: i for i, rid in enumerate(ranked)}
|
||
rows.sort(key=lambda r: order.get(r.id, len(order)))
|
||
out = []
|
||
for row in rows[:CANDIDATES_PER_KIND]:
|
||
article = articles.get(row.article_id)
|
||
if not article:
|
||
continue
|
||
if article.status != "published" and not user.is_moderator and article.user_id != user.id:
|
||
continue
|
||
out.append({
|
||
"kind": "section", "ref": f"{article.id}#{row.section_id}",
|
||
"id": article.id, "section_id": row.section_id,
|
||
"title": f"{article.title} › {row.title or 'section'}",
|
||
"text": _clean(row.content),
|
||
"score": 1.0 / (1 + order.get(row.id, 0)),
|
||
})
|
||
return out
|
||
|
||
|
||
def _questions(db: Session, user: User, query: str) -> list[dict]:
|
||
ranked, _ = hybrid_ids(db, query, "question", limit=PER_KIND * 3)
|
||
if not ranked:
|
||
return []
|
||
q = bank_query(db, user).filter(Question.id.in_(ranked))
|
||
scope = exam_scope_predicate(db, user)
|
||
if scope is not None:
|
||
q = q.filter(scope)
|
||
order = {rid: i for i, rid in enumerate(ranked)}
|
||
rows = sorted(q.all(), key=lambda r: order.get(r.id, len(order)))
|
||
return [{
|
||
"kind": "question", "ref": str(row.id), "id": row.id,
|
||
# The opening of the stem, not the row's number. "Question #2320" tells
|
||
# a reader nothing about whether it is worth following, and the list of
|
||
# sources under an answer is exactly where that has to be decidable.
|
||
"title": _clean(row.question_text, 90) or f"Question #{row.id}",
|
||
# The stem only. An answer belongs to the quiz runner, and a chat that
|
||
# printed it would hand away the practice it is meant to prepare for.
|
||
"text": _clean(row.question_text, 320),
|
||
"score": 1.0 / (1 + order.get(row.id, 0)),
|
||
} for row in rows[:CANDIDATES_PER_KIND]]
|
||
|
||
|
||
def _cards(db: Session, user: User, query: str) -> list[dict]:
|
||
ranked, _ = hybrid_ids(db, query, "flashcard", limit=PER_KIND * 3)
|
||
if not ranked:
|
||
return []
|
||
own = [d.id for d in db.query(FlashcardDeck.id).filter(
|
||
FlashcardDeck.user_id == user.id, FlashcardDeck.deleted_at.is_(None)).all()]
|
||
if not own:
|
||
return []
|
||
rows = db.query(Flashcard).filter(
|
||
Flashcard.id.in_(ranked), Flashcard.deck_id.in_(own)).all()
|
||
order = {rid: i for i, rid in enumerate(ranked)}
|
||
rows.sort(key=lambda r: order.get(r.id, len(order)))
|
||
return [{
|
||
"kind": "card", "ref": str(row.id), "id": row.id,
|
||
"title": _clean(row.front, 90),
|
||
"text": _clean(row.back, 260),
|
||
"score": 1.0 / (1 + order.get(row.id, 0)),
|
||
} for row in rows[:CANDIDATES_PER_KIND]]
|
||
|
||
|
||
def _apply_curated_boost(db: Session, sources: list[dict]) -> None:
|
||
"""Lift a source that an educator tied to another source in this shortlist.
|
||
|
||
Two rows an educator already linked, both surfacing for the same question, is
|
||
a stronger signal than either ranking alone — that is what a curated link is
|
||
for. Nothing is stored; the boost lives only in this ordering.
|
||
"""
|
||
article_ids = {s["id"] for s in sources if s["kind"] in ("article", "section")}
|
||
question_ids = {s["id"] for s in sources if s["kind"] == "question"}
|
||
if not article_ids or not question_ids:
|
||
return
|
||
pairs = db.query(QuestionArticleLink).filter(
|
||
QuestionArticleLink.article_id.in_(article_ids),
|
||
QuestionArticleLink.question_id.in_(question_ids)).all()
|
||
if not pairs:
|
||
return
|
||
linked_articles = {p.article_id for p in pairs}
|
||
linked_questions = {p.question_id for p in pairs}
|
||
for source in sources:
|
||
if source["kind"] in ("article", "section") and source["id"] in linked_articles:
|
||
source["score"] += CURATED_BOOST
|
||
source["curated"] = True
|
||
elif source["kind"] == "question" and source["id"] in linked_questions:
|
||
source["score"] += CURATED_BOOST
|
||
source["curated"] = True
|
||
|
||
|
||
def _cross_encode(query: str, sources: list[dict]) -> None:
|
||
"""Rescore the whole shortlist on one scale, in place, if a reranker answers.
|
||
|
||
The scores the finders assign are `1/(1+rank)` *within their own corpus*, so
|
||
the best article, the best section, the best question and the best card all
|
||
score 1.0 and the order between them is whatever the sort happened to do.
|
||
A cross-encoder is the first thing in this pipeline that can compare a
|
||
section against a question, because it reads both against the same query.
|
||
|
||
The new score is the position, not the raw relevance: 0.75 from one reranker
|
||
and 0.78 from another mean nothing to `CURATED_BOOST`, whereas `1/(1+rank)`
|
||
keeps a curated pair worth the same few places it has always been worth.
|
||
|
||
Silence leaves the scores alone. That is the whole degradation story here:
|
||
without a reranker the shortlist is exactly what it was before there was one.
|
||
"""
|
||
if len(sources) < 2:
|
||
return
|
||
scores = rerank_service.rerank(query, [f"{s['title']}. {s['text']}" for s in sources])
|
||
if scores is None:
|
||
return
|
||
ranking = sorted(range(len(sources)), key=lambda i: (-scores[i], i))
|
||
for position, index in enumerate(ranking):
|
||
sources[index]["score"] = 1.0 / (1 + position)
|
||
|
||
|
||
#: A query of this many words or fewer is a term, not a sentence. A term is
|
||
#: where a typo does the most damage: there is nothing else in it for either
|
||
#: ranker to catch hold of.
|
||
SHORT_QUERY_WORDS = 3
|
||
#: Below this the nearest title is not a correction, it is a coincidence.
|
||
#: "kawasaki critera" matches "Kawasaki disease" at 0.36, and the next article
|
||
#: down scores 0.11 — the gap is the signal.
|
||
NEAREST_TITLE = 0.3
|
||
|
||
|
||
def nearest_topic(db: Session, query: str) -> str | None:
|
||
"""The article this was probably meant to say.
|
||
|
||
A one-letter slip empties the library: "kawasaki criteria" finds fourteen
|
||
sources, "kawasaki critera" finds none — both rankers miss it, lexical
|
||
because the token is not in any index and semantic because the embedding
|
||
of a misspelling is not near the embedding of the word. Trigrams do not
|
||
care how it is spelled.
|
||
"""
|
||
try:
|
||
row = db.execute(sa_text(
|
||
"SELECT title, similarity(title, :q) AS sim FROM articles "
|
||
"WHERE deleted_at IS NULL AND status = 'published' "
|
||
"ORDER BY sim DESC LIMIT 1"
|
||
), {"q": query}).first()
|
||
except Exception:
|
||
# No pg_trgm, no correction — and no failure either.
|
||
logger.warning("AI Mode: trigram lookup unavailable", exc_info=True)
|
||
return None
|
||
if row and row.sim and float(row.sim) >= NEAREST_TITLE:
|
||
return row.title
|
||
return None
|
||
|
||
|
||
def retrieve(db: Session, user: User, query: str) -> list[dict]:
|
||
"""The only things the model will be allowed to cite for this message."""
|
||
query = (query or "").strip()
|
||
if len(query) < 2:
|
||
return []
|
||
found = _gather(db, user, query)
|
||
# A short query that found almost nothing is usually a misspelled term
|
||
# rather than a subject the library does not hold. Ask again with the
|
||
# closest title, and keep whichever attempt did better.
|
||
if len(found) < 3 and len(query.split()) <= SHORT_QUERY_WORDS:
|
||
topic = nearest_topic(db, query)
|
||
if topic and topic.lower() != query.lower():
|
||
wider = _gather(db, user, topic)
|
||
if len(wider) > len(found):
|
||
logger.info("AI Mode: %r found %d, retried as %r and found %d",
|
||
query, len(found), topic, len(wider))
|
||
found = wider
|
||
return found
|
||
|
||
|
||
def _gather(db: Session, user: User, query: str) -> list[dict]:
|
||
sources: list[dict] = []
|
||
for finder in (_sections, _articles, _questions, _cards):
|
||
try:
|
||
sources.extend(finder(db, user, query))
|
||
except Exception:
|
||
# One corpus failing narrows the answer; it does not end it.
|
||
logger.warning("AI Mode retrieval failed for %s", finder.__name__, exc_info=True)
|
||
_cross_encode(query, sources)
|
||
_apply_curated_boost(db, sources)
|
||
sources.sort(key=lambda s: -s["score"])
|
||
|
||
# Still no more than `PER_KIND` of anything. A shortlist the cross-encoder
|
||
# filled with fourteen sections of one article would score well and read
|
||
# like a single source quoted fourteen times, and it would leave the chat
|
||
# with no question to send the learner to practise.
|
||
kept: list[dict] = []
|
||
seen: dict[str, int] = {}
|
||
for source in sources:
|
||
if seen.get(source["kind"], 0) >= PER_KIND:
|
||
continue
|
||
seen[source["kind"]] = seen.get(source["kind"], 0) + 1
|
||
kept.append(source)
|
||
if len(kept) >= MAX_SOURCES:
|
||
break
|
||
return kept
|
||
|
||
|
||
def sources_block(sources: list[dict]) -> str:
|
||
"""The shortlist, as the model sees it."""
|
||
lines = []
|
||
for source in sources:
|
||
marker = f"[[{source['kind']}:{source['ref']}]]"
|
||
curated = " (an educator linked this to another source here)" if source.get("curated") else ""
|
||
lines.append(f"{marker} {source['title']}{curated}\n{source['text']}")
|
||
return "\n\n".join(lines)
|
||
|
||
|
||
# How close the nearest thing in the library has to be before the answer is
|
||
# treated as coming from it. Measured against this corpus with the bodies
|
||
# embedded, eight clearly on-topic questions and eight clearly off-topic ones:
|
||
#
|
||
# off-topic 0.339 – 0.499 (the French revolution … photosynthesis)
|
||
# on-topic 0.586 – 0.740 (what causes croup … posterior urethral valves)
|
||
#
|
||
# The gap between them is where these sit. They are not the retrieval floor and
|
||
# must not be: `SEMANTIC_FLOOR` decides what is worth showing in a list, where a
|
||
# weak hit costs a reader one glance. Here it decides whether an answer claims
|
||
# to come from the library, and a wrong claim costs them their trust in every
|
||
# other answer.
|
||
#
|
||
# Worth re-measuring when the corpus changes size or subject; the method and
|
||
# the full measurement are in docs/retrieval-thresholds.md. Anything else is
|
||
# tuning by feel against numbers nobody wrote down.
|
||
#
|
||
# Deliberately not the cross-encoder's score, though it is the better judge of a
|
||
# pair. The question here is "is there anything in this library about this at
|
||
# all", and `top_similarity` answers it by scanning every embedded row in two
|
||
# corpora through the vector index. A reranker can only score candidates that
|
||
# were shortlisted first, so a reranked closeness cannot tell "the library does
|
||
# not cover this" from "retrieval had a bad day", and it would put a network hop
|
||
# in the path of a decision about what the answer *claims* — the one place where
|
||
# a service being down must not change the output. Order is a preference;
|
||
# whether an answer says it came from the library is a promise.
|
||
STRONG_MATCH = 0.55
|
||
ADJACENT_MATCH = 0.50
|
||
|
||
ROLE = "You are a study assistant for a pediatrics learning platform.\n"
|
||
|
||
CITE = (
|
||
"Cite with the exact marker shown, for example [[article:7]] or "
|
||
"[[section:7#abc123]], placed at the end of the sentence it supports. Never "
|
||
"write a URL and never cite a marker that is not listed here.\n\n"
|
||
# Not a hallucination guard — over fifteen measured runs the checker
|
||
# stripped none of the 45 markers written, so invented citations are not
|
||
# the problem. The problem is the opposite: one differential answered from
|
||
# the sources and cited nothing at all, which leaves the learner with an
|
||
# assertion and nowhere to check it.
|
||
"You have been given sources, so at least one sentence must carry a "
|
||
"citation. An answer drawn from this library and citing none of it is not "
|
||
"an answer the learner can check.\n\n"
|
||
"Never reveal the answer to a practice question. You may say what a question "
|
||
"is about so the learner can go and attempt it.\n\n"
|
||
# Asked for five questions and able to see one, it explained at length that
|
||
# it had "only actually looked at one cervicitis item so far" and offered to
|
||
# go and gather the rest. None of that is the learner's problem, and none of
|
||
# it is negotiable: the session is built by the button under the answer,
|
||
# from whatever the answer cited, capped at twenty.
|
||
"Never describe your own retrieval: not how many sources you were given, "
|
||
"not that you have not looked at more, not what you could go and fetch, "
|
||
"and never how many questions there are.\n\n"
|
||
"A learner may ask for a number of questions — five, twenty, fifty. Ignore "
|
||
"the number. Do not agree to it, do not apologise for it, do not explain "
|
||
"what you have instead, and never offer to find more. Answer what they "
|
||
"asked about and say, in one short sentence, that they can practise this "
|
||
"below. If they ask again, say the same thing again. Do not write practice "
|
||
"questions of your own.\n\n"
|
||
"Be brief: a few sentences or a short list.\n\n"
|
||
)
|
||
|
||
|
||
def closeness(db: Session, query: str) -> float | None:
|
||
"""How close the nearest thing in the library is, or None if unmeasurable."""
|
||
try:
|
||
return top_similarity(db, query)
|
||
except Exception:
|
||
logger.warning("AI Mode could not measure closeness", exc_info=True)
|
||
return None
|
||
|
||
|
||
#: Words a message can be made entirely of and still not be a question about
|
||
#: anything. Kept as a closed list rather than a length rule: "croup dose?" is
|
||
#: two words and is very much a query, while "ok thanks" is two words and is
|
||
#: not.
|
||
SMALL_TALK_WORDS = {
|
||
"hi", "hello", "hey", "yo", "hiya", "greetings", "howdy", "morning",
|
||
"afternoon", "evening", "good", "day", "night", "sup", "hallo",
|
||
"thanks", "thank", "thx", "ty", "cheers", "appreciated", "much", "you",
|
||
"ok", "okay", "okey", "k", "sure", "cool", "nice", "great", "perfect",
|
||
"yes", "yeah", "yep", "no", "nope", "nah", "please", "sorry", "welcome",
|
||
"bye", "goodbye", "later", "see", "ya", "ciao",
|
||
"lol", "haha", "hmm", "hm", "oh", "ah", "wow", "test", "testing",
|
||
}
|
||
|
||
#: Questions about the assistant itself. The library has nothing to say about
|
||
#: these by construction, and searching it for them produces exactly the
|
||
#: nonsense this guard exists to stop: a greeting answered with four citations.
|
||
ABOUT_ASSISTANT_RE = re.compile(
|
||
r"^\s*(who\s+(are|r)\s+(you|u)|what\s+(are|r)\s+(you|u)|"
|
||
r"what\s+(can|do)\s+(you|u)\s+(do|help)|how\s+(do|does)\s+(this|it)\s+work|"
|
||
r"what\s+is\s+this|help)\b",
|
||
re.IGNORECASE,
|
||
)
|
||
|
||
|
||
def is_small_talk(text: str) -> bool:
|
||
"""Whether this message is not a query at all.
|
||
|
||
A greeting is not a low-scoring question, and the difference matters:
|
||
reciprocal-rank fusion always returns an order, so "hello" comes back with
|
||
six paediatric sources ranked confidently against nothing. The similarity
|
||
gate below catches most of that, but it is a threshold, and "ok" and "good
|
||
morning" happen to land the wrong side of it — a greeting embedded into the
|
||
same space as a corpus of clinical prose scores wherever it scores. So this
|
||
is decided before any measuring is done, on the text itself.
|
||
|
||
Deliberately narrow. Anything with a word in it that is not pleasantry
|
||
falls through to retrieval, because the cost of a missed greeting is a
|
||
slightly odd reply and the cost of a swallowed question is an unanswered
|
||
one.
|
||
"""
|
||
stripped = (text or "").strip()
|
||
if not stripped:
|
||
return True
|
||
if ABOUT_ASSISTANT_RE.match(stripped):
|
||
return True
|
||
words = re.findall(r"[a-z]+", stripped.lower())
|
||
if not words or len(words) > 5:
|
||
return False
|
||
return all(word in SMALL_TALK_WORDS for word in words)
|
||
|
||
|
||
CHAT_PROMPT = (
|
||
ROLE +
|
||
"The learner has not asked a question yet — this turn is a greeting, a "
|
||
"thank-you, or a question about you rather than about medicine.\n\n"
|
||
"Reply in one or two short sentences. Say what you can do: answer from the "
|
||
"articles, questions and cards in their library, and point them at "
|
||
"questions to practise. Do not list topics, do not cite anything, and do "
|
||
"not invent what their library contains — you have not looked."
|
||
)
|
||
|
||
|
||
def answer_mode(similarity: float | None, sources: list[dict],
|
||
question: str | None = None) -> str:
|
||
"""Which answer this turn gets: chat, sourced, adjacent, or open.
|
||
|
||
Decided by a number rather than by asking the model to work out which
|
||
situation it is in. Classification written as prose in a prompt is the part
|
||
that does not work, and it is also the part that makes the prompt long.
|
||
|
||
An unmeasurable closeness — no vector database, encoder down — is not a low
|
||
one. Retrieval still found these rows by other means, and discarding them
|
||
because the ruler is missing would silently drop every citation on a
|
||
deployment where semantic search happens to be unavailable.
|
||
"""
|
||
if question is not None and is_small_talk(question):
|
||
return "chat"
|
||
if not sources:
|
||
return "open"
|
||
if similarity is None or similarity >= STRONG_MATCH:
|
||
return "sourced"
|
||
return "adjacent" if similarity >= ADJACENT_MATCH else "open"
|
||
|
||
|
||
def build_prompt(sources: list[dict], mode: str = "sourced") -> str:
|
||
if mode == "chat":
|
||
# No sources, and — unlike "open" — no announcement that the library
|
||
# does not cover it either. Nobody who says hello is waiting to be told
|
||
# what their library lacks.
|
||
return CHAT_PROMPT
|
||
if mode == "open" or not sources:
|
||
# Nothing in the library is close, so the honest answer is to say that
|
||
# and then help anyway. Refusing outright was the old behaviour and it
|
||
# reads as a broken assistant rather than a careful one; answering as
|
||
# though the shortlist supported it would be worse still.
|
||
return (
|
||
ROLE +
|
||
"Nothing in this learner's library covers their question.\n\n"
|
||
"Open with one short sentence saying so. Then answer from general "
|
||
"knowledge, briefly and plainly. Do not cite anything: there is "
|
||
"nothing here to cite, and a marker you invent points nowhere."
|
||
)
|
||
if mode == "adjacent":
|
||
# Something related, nothing direct. Naming the gap is the point: a
|
||
# learner told "here is what is closest" can judge the answer, where one
|
||
# handed an adjacent source as though it were the answer cannot.
|
||
return (
|
||
ROLE +
|
||
"Nothing in this learner's library covers their question directly. "
|
||
"The sources below are the closest things in it.\n\n"
|
||
"Open with one short sentence saying that, naming what the closest "
|
||
"material is about. Then answer using those sources where they help, "
|
||
"citing them, and from general knowledge where they do not — saying "
|
||
"which is which.\n\n"
|
||
+ CITE +
|
||
f"SOURCES\n\n{sources_block(sources)}"
|
||
)
|
||
return (
|
||
ROLE + "\n"
|
||
"Answer only from the sources below. They are the learner's own library — "
|
||
"if they do not contain the answer, say so rather than filling the gap from "
|
||
"your own knowledge, which the learner cannot check against anything.\n\n"
|
||
+ CITE +
|
||
f"SOURCES\n\n{sources_block(sources)}"
|
||
)
|
||
|
||
|
||
def enforce_citations(reply: str, sources: list[dict]) -> tuple[str, list[dict]]:
|
||
"""Drop every citation that was not retrieved, and report the ones that stand.
|
||
|
||
This is the step that makes a hallucinated citation impossible rather than
|
||
unlikely. The model can write whatever marker it likes; only markers in the
|
||
shortlist survive contact with this function.
|
||
"""
|
||
allowed = {f"{s['kind']}:{s['ref']}": s for s in sources}
|
||
used: dict[str, dict] = {}
|
||
|
||
def replace(match: re.Match) -> str:
|
||
key = f"{match.group(1)}:{match.group(2)}"
|
||
source = allowed.get(key)
|
||
if source is None:
|
||
return "" # Invented, or pointing at something this learner may not see.
|
||
used[key] = source
|
||
return match.group(0)
|
||
|
||
cleaned = CITATION_RE.sub(replace, reply)
|
||
# Deleting a marker can leave a double space or a space before a full stop.
|
||
cleaned = re.sub(r"[ \t]{2,}", " ", cleaned)
|
||
cleaned = re.sub(r"\s+([.,;:!?])", r"\1", cleaned).strip()
|
||
|
||
citations = [{
|
||
"marker": f"[[{s['kind']}:{s['ref']}]]",
|
||
"kind": s["kind"], "id": s["id"],
|
||
"section_id": s.get("section_id"),
|
||
"title": s["title"],
|
||
"curated": bool(s.get("curated")),
|
||
} for s in used.values()]
|
||
return cleaned, citations
|
||
|
||
|
||
#: A session assembled from a conversation. Enough to be worth sitting, few
|
||
#: enough that it follows from what was just discussed rather than becoming a
|
||
#: general exam on the topic.
|
||
PRACTICE_MAX = 20
|
||
|
||
|
||
def practice_ids(db: Session, user: User, citations, question: str) -> list[int]:
|
||
"""Which questions a chat turn should send a learner to practise.
|
||
|
||
Three sources, in the order they deserve trust. A question the answer
|
||
actually cited is the closest thing to "this is what we were talking
|
||
about". An article it cited stands for a topic, so questions filed under
|
||
that article's category follow. Retrieval on the learner's own words
|
||
catches the rest.
|
||
|
||
Everything is put through the bank's own visibility rules on the way out —
|
||
a chat is not a way to reach questions a learner could not otherwise see.
|
||
"""
|
||
from app.models.question_category import QuestionCategoryLink
|
||
|
||
cited_questions = [int(c["id"]) for c in (citations or []) if c.get("kind") == "question"]
|
||
article_ids = [int(c["id"]) for c in (citations or []) if c.get("kind") == "article"]
|
||
|
||
from_articles: list[int] = []
|
||
if article_ids:
|
||
categories = [cid for (cid,) in db.query(Article.category_id).filter(
|
||
Article.id.in_(article_ids), Article.category_id.isnot(None)).all()]
|
||
if categories:
|
||
direct = db.query(Question.id).filter(
|
||
Question.question_category_id.in_(categories)).limit(PRACTICE_MAX * 3).all()
|
||
linked = db.query(QuestionCategoryLink.question_id).filter(
|
||
QuestionCategoryLink.category_id.in_(categories)).limit(PRACTICE_MAX * 3).all()
|
||
from_articles = [qid for (qid,) in [*direct, *linked]]
|
||
|
||
retrieved: list[int] = []
|
||
try:
|
||
retrieved, _ = hybrid_ids(db, question or "", "question", limit=PRACTICE_MAX * 2)
|
||
except Exception:
|
||
logger.warning("Practice retrieval failed", exc_info=True)
|
||
|
||
ordered: list[int] = []
|
||
for group in (cited_questions, from_articles, retrieved):
|
||
for qid in group:
|
||
if qid not in ordered:
|
||
ordered.append(qid)
|
||
if not ordered:
|
||
return []
|
||
|
||
allowed = bank_query(db, user).filter(Question.id.in_(ordered))
|
||
scope = exam_scope_predicate(db, user)
|
||
if scope is not None:
|
||
allowed = allowed.filter(scope)
|
||
visible = {row.id for row in allowed.all()}
|
||
return [qid for qid in ordered if qid in visible][:PRACTICE_MAX]
|
||
|
||
|
||
#: Openers a learner types before the real question. Dropped from the thread
|
||
#: name, never from the question itself.
|
||
_FILLER = re.compile(
|
||
r"^(?:hi|hey|hello|ok|okay|so|please|pls|can you|could you|tell me|"
|
||
r"i want to know|i'd like to know|explain to me)\b[\s,:-]*", re.I)
|
||
_TITLE_MAX = 60
|
||
|
||
|
||
def thread_title(question: str) -> str:
|
||
"""A name for a conversation, taken from its first question.
|
||
|
||
It used to be the raw question truncated at eighty characters, which is how
|
||
a sidebar ends up reading "how do i treat cervicitis in a teenager and wh…"
|
||
— the learner's typing, warts and all. This trims the throat-clearing,
|
||
starts with a capital, and cuts at a word rather than mid-syllable.
|
||
"""
|
||
original = " ".join((question or "").split())
|
||
text = original
|
||
# "ok so bronchiolitis" is two openers, not one.
|
||
for _ in range(3):
|
||
stripped = _FILLER.sub("", text).strip()
|
||
if stripped == text:
|
||
break
|
||
text = stripped
|
||
text = text or original
|
||
if not text:
|
||
return "New chat"
|
||
text = re.sub(r"\bi\b", "I", text)
|
||
if len(text) > _TITLE_MAX:
|
||
cut = text[:_TITLE_MAX].rsplit(" ", 1)[0] or text[:_TITLE_MAX]
|
||
text = cut.rstrip(" ,;:-") + "…"
|
||
else:
|
||
# A full stop adds nothing to a label; a question mark says what it is.
|
||
text = text.rstrip(" .,;:")
|
||
return text[0].upper() + text[1:]
|