A source PDF is chunked, vectorised and then read by a model a chunk at
a time, so a 500 MB upload is not a big file — it is an hour of work
nobody asked for and a bucket that grows for ever. The cap is 20 MB, in
the config default, in backend/.env which was overriding it at
524288000, and in nginx, which was letting 500M through to be refused
by the application afterwards. A backup of .env is beside it.
And a document is deleted where the documents are listed. The endpoint
has always existed and removes the file, the vector collection and the
row — but reaching it meant opening the document first, which is a page
you go to in order to extract from it, not somewhere you visit to tidy
up. Delete, then Delete it or Keep, in the workbench list.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
Everybody arriving through SSO landed as a learner and was promoted by
hand. That is fine for three people and wrong for three hundred: the
list of who may edit the bank then lives in two places and only one of
them is maintained. OIDC_ROLE_CLAIM names the claim that carries somebody's
groups ("groups" for most providers, "roles" for Entra app roles), and
OIDC_ADMIN_GROUPS / OIDC_MODERATOR_GROUPS say which values mean what.
All three shapes a provider might send are read — a list, a
space-separated string, a comma-separated one — and matched case-blind.
Applied on every sign-in, not only at creation, because a group somebody
can be added to and never removed from is not a list anybody can rely
on. The one demotion it refuses is the last administrator: a mistyped
group name should not lock everybody out of the settings page,
including the person who could fix the group name. It is logged when it
happens.
Blank OIDC_ROLE_CLAIM is the old behaviour exactly: role "user", set
once, never touched.
And an address the provider will not vouch for is not an identity. This
matches on email, so an explicit email_verified:false would hand an
existing account to whoever typed its address at a careless provider.
Refused now — only on an explicit false, since a provider that omits the
claim is not asserting anything either way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
**Two sources an AI draft can draw on**, both off until an administrator turns
them on, both appended to the prompt as extra material rather than woven into
it — so a draft with nothing to draw on is byte-for-byte the draft that has
been working well.
- *The clinical library.* The indexed shelf the clinical assistant already
searches, over MCP on the internal network. Ported from ped-ai: sessions are
reused, a dead one is reopened once, and a library that cannot be reached
never fails the article — it just means the educator is writing without it,
and the progress line says so.
- *PubMed.* NCBI's E-utilities, no key required. Ported whole, including the
two lessons that cost somebody an afternoon over there: PubMed ANDs every
term, so "bronchiolitis management in infants" can find nothing where
"bronchiolitis management" finds six — hence the query ladder — and three
esearch calls in a row will trip the rate limit, hence the spacing. The
reference list is written from the records rather than by the model, so every
line is a paper that exists with a PMID somebody can look up.
Measured on the live stack: 24 excerpts, 6 papers, 6 references, 6 in-text
citations, in one draft.
**The card system, which turned out to be half-built:**
- There was no way to make a deck by hand, and no way to edit a card at all —
you could browse, view and delete. Both are there now, the editor taking
front, back and a picture.
- Filing, writing, sharing and deleting are all educator work now, behind one
named gate rather than four scattered checks. A learner studies.
- A deck generated from an article inherits that article's category instead of
landing in Uncategorized for somebody to file by hand.
- A link inside a card previewed instead of going. A card is a box a few lines
tall, often inside a flipping panel, and a hover card anchored in one is
clipped by it — so the link read as broken because clicking it did nothing.
Where there is no room to preview, the honest behaviour is to take you there.
**An AI draft belonged to no editorial queue.** Nothing set `generated_by`, so
a drafted article was neither "generated, unread" nor anything else: the tile
counted it and there was nowhere to click. Drafts are stamped with the model
that wrote them, and there is now a plain Drafts queue that cannot be fallen
through.
**The sign-in code email** is laid out rather than written: the code is the
biggest thing on the screen, then which account it signs into, then a way back
to the page, then permission to ignore the whole thing.
Also: a back link out of a deck, in the same words as the rest of the app.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
**The API.** Every route now lives under `/api/v1`, with `/api/...` rewritten
onto it — one route, two spellings, so they cannot drift and the OpenAPI
document describes each endpoint once. Errors carry an `error` object with a
stable code, one human sentence and, for a validation failure, the fields that
were wrong; `detail` is untouched so nothing that reads it breaks. The whole
surface — 320 routes, their parameters and their status codes — is checked in
as `backend/tests/api-contract.json`, and a test fails on any difference,
naming the routes that moved. `docs/api.md` is the contract in prose.
**Refresh tokens**, so an app can stay signed in without keeping a password.
Rows rather than signatures: listable, withdrawable, stored as hashes, rotated
on every use. A spent token coming back ends the whole session, because a theft
and a replay look identical from the server and the safe reading is the unsafe
one. A browser is not given one — it has nowhere to put it and a person to ask.
**An end-to-end stack**: `docker-compose.test.yml` with its own Postgres and
Redis, `e2e/seed.py` for the smallest world the tests name, and Playwright with
five projects — desktop, iPhone, Pixel, iPad and a browserless API project.
Devices because every bug reported this week was a phone bug found by a person
looking at a screenshot; a desktop-only suite would have passed through all of
them. Forty tests, five clean runs.
It found four things in its first hour:
- **A fresh deploy could not start.** `create_all()` ran before
`CREATE EXTENSION vector`, so any database that had never had pgvector
installed died on the first table with a vector column. Invisible here
because this one has had the extension for a year.
- **A figure in a published article was a 404 for everyone but an admin.**
Media in the library is nobody's to read by default, and nothing made an
exception for a drawing an article actually shows — so every illustration
added this week was an empty box for every real user.
- **Every rate limit was one bucket for the whole site.** The backend saw
nginx's address for every request, so ten bad passwords from anybody locked
out everybody, and no log line could say who. nginx now takes the real
address from the proxy and overwrites the header on the way in; uvicorn runs
with --proxy-headers.
- **The reading page's breakpoints disagreed** — 1150px in the component,
820px in the stylesheet. Between them the menu button claimed the contents
drawer and then toggled a class on a rail that was still in the layout: the
contents did not open and the site menu did not either. The button was dead
on every tablet.
And two smaller ones: the login limiter counted successful sign-ins, so eleven
people behind one hospital NAT locked each other out — it is cleared by a
correct password now; and `/uploads/{path}` served GET and HEAD from one route
with one operation id, which makes every OpenAPI client generator refuse the
document.
The first admin's password is generated and printed once at first start when
`DEFAULT_ADMIN_PASSWORD` is blank, rather than the account not existing:
`docker compose logs backend | grep -A3 "FIRST ADMIN"`.
CI (`.forgejo/workflows/tests.yml`) runs the backend suite, the contract, the
frontend suite and the build on every push to dev, main or master, and the
end-to-end stack on those branches and on pull requests into them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
Retrieval fused a bi-encoder and BM25 by reciprocal rank. A bi-encoder embeds a
document long before the question exists, so the two never meet: it is good at
"same topic" and mediocre at "answers this". A cross-encoder reads the pair.
The proxy already serves three — `cohere-rerank-v4.0-pro` is the default and
measurably better than the fast variant. Query text goes exactly where the
embeddings already go, and nothing new was signed up for.
It found a defect nobody was looking for. In AI Mode each finder scored
`1/(1+rank)` *within its own corpus*, so the best article, section, question and
card all scored 1.0 and the shortlist was a meaningless round-robin. A
cross-encoder is the first thing in this system that can compare a question
with a section. Candidates per kind widened so it can select rather than merely
reorder.
Measured against labels neither ranker produced. Questions, 60 disease tags:
precision@3 0.394 → 0.483. Sections, 60 article titles: 0.772 → 0.833.
"Management of bronchiolitis" led with influenza transmission and a pregnancy
question; "when do you image a first febrile seizure" returned the definition
rather than the sentence saying imaging is unnecessary.
And the honest negative, in docs/reranking.md: board vignettes are written
*not* to name their diagnosis, so on "what causes croup" it prefers a question
that says the word in passing over the barking-cough vignette that never says
it. Some of the bi-encoder's strength is traded away.
Not on the typeahead. A page of results is a choice being made and worth a
third of a second; a typeahead is a word being finished, runs on every
keystroke, and has nothing to judge yet.
The three-state thresholds stay on cosine, argued at the constant: a reranker
only ever sees a shortlist and structurally cannot answer the corpus-wide
question those numbers ask, and whether an answer claims to come from the
library is a promise that must not depend on a network hop.
Every failure returns None and leaves the order alone — unconfigured, no proxy,
connect error, bare 502, timeout, non-JSON, a duplicate or out-of-range index,
a non-numeric score, a list the wrong length. Verified against the running site
with a bogus model name: same results, fused order, no error to the reader.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
Twenty-one stem figures are JPEG 2000. Chrome dropped it in 2015, Firefox and
Edge never had it, and the slim base image ships no MIME table — so
`guess_type` returned nothing, the fallback was `application/octet-stream`, and
`nosniff` finished the job. Those figures rendered nowhere but Safari.
The bytes were never the problem: Pillow decodes JP2 here perfectly well. Only
the delivery had to change, so it changes the way everything else already does
— through the thumbnail machinery, as a cached WebP derivative, stored beside
the original. A format no browser draws now asks for conversion whatever size
it was requested at, decided by the file's own magic rather than by the query
string. The 41 KB original comes back as an 83 KB full-size WebP or a 5 KB
thumbnail, and the stored file is untouched.
`.jp2`, `.jpx`, `.jpf` and `.webp` are registered at import, because a
container with no `/etc/mime.types` is a container that mislabels every one of
them. `.webp` had no figures behind it yet and would have failed the same way.
Two calls could hang for ten minutes. The SDK reads for that long by default
and this client retries nothing, so a stalled connection is a stalled request —
three of them in extraction, which does its own retrying. Both now pass an
explicit two-minute timeout.
Also removed: `EMBEDDING_PROVIDER`, which looks like a switch between a local
encoder and a remote one and is read nowhere, with a comment claiming
embeddings run locally when they have always gone over the network to the
proxy; and a `.replace("openai/", "")` that existed only to undo a prefix
nothing adds any more. The JPEG 2000 comment named the wrong mechanism — the
filename is no guide because there is no MIME table, not because it lies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
Proof-of-work rather than a puzzle, and — the reason for it — nothing
about the person signing up is described to a third party in order to let
them in. Turnstile and then hCaptcha were both here; both told Cloudflare
who was at the door.
The `cap` service runs on the compose network with its own Redis
database, kept apart from the app's so a flush of one cannot clear the
other's challenges. The widget talks to /cap/ on this origin, proxied by
the frontend's nginx, so the browser reaches nobody else either. Caddy
passes the whole host through to that container, so it needed no change.
Two things that had to be found rather than read:
Cap's key API is undocumented. The routes are `/auth/login` and
`/server/keys`, and the Bearer value is base64 JSON of `{token, hash}` —
not the session token itself, which is why the obvious call returns
"Malformed session token". The site key and secret were created that way
rather than by hand in a dashboard.
And an nginx proxy_pass whose target is a variable passes the URI through
untouched: the trailing slash that strips a location prefix on a literal
target does nothing. Cap was being asked for /cap/<key>/challenge and
answering NOT_FOUND until the prefix was stripped by an explicit rewrite.
Verified end to end against the running service: a challenge is issued
through the public path, and a token that was never issued is refused
rather than waved through.
Also here: the register modal's Name and Email were bare labels that
neither wrapped their input nor named it, so a screen reader met two
boxes with no names and clicking the word did nothing.
And the knowledge profile paginates ten to a page and expands each row to
its two bars beside the next step. "Correct using hints" is missing from
that bar because /study-tools/recommendations does not carry it per
topic — inferring it from the lifetime figure would be a different set of
answers, so the bar is honestly two-tone until the backend offers it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
The incomplete-block warning is the one from the screenshot: a red
heading that says the block is incomplete, the count of unanswered items,
the sentence about resuming not matching exam day, and End Block against
Remain in Block. My version asked the question in my own words and led
with the wrong button.
Pausing says "Exam Paused" and offers Return to exam. Nothing else — the
warning about real exams is somebody else's disclaimer, not ours.
Exit session asks "Are you sure you want to end this session?" before it
goes, rather than going.
Time's Up says what it is and the button says Close, which is the only
thing left to do: it is already handed in and marked, and Close lands on
the session's analysis.
One name for one action: the bottom button read Skip on an unanswered
question and Next on an answered one, while the arrow an inch above it
said Next for both.
And the rail shows stems again once the block is handed in. Numbers while
it is being sat — reading ahead is not something the exam being rehearsed
allows — but there is nothing left to protect afterwards, so the review
reads like study mode.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
Standardises cross-references the way we agreed, and puts a CMS around articles
so hundreds of generated drafts are reviewable rather than merely present.
Links, made rename-proof
`[[7|Febrile seizures]]` resolves by id and displays the text — the id is the
part that must not change, the text is what keeps prose readable while you write
it. `[[old-slug]]` still resolves and is rewritten to the id form on save, not in
a migration: an article nobody has touched is not broken, and rewriting prose no
one asked to change is how an editor stops trusting the editor. Every slug an
article has ever had is kept, so a rename redirects instead of 404ing, and a save
reports markers pointing at nothing — at the moment the person who wrote the link
is still looking at it.
Three views of one topic
The full article to study from, the key points to revise from, the clinical view
to act from, with doses. They are views of one article rather than three
articles, so the numbers cannot drift apart and a question linked to the topic
still means one thing. Each section carries its variant; articles written before
this are the long view, unchanged.
CMS
draft → in review → published, with an author able to submit and only a
moderator able to publish. Every save snapshots what was there, restorable, and
restoring is itself snapshotted or the way back from a mistaken restore is gone.
The editorial queue is work rather than inventory: waiting for review, generated
and unread, published without sources, published with nothing to practise,
barely written. An empty bucket is drawn as good news, not as an alert.
Articles from the clinical library
The library index is 1.8M chunks of reference texts embedded with bge-m3 — the
same model PedsHub already uses, so our query vectors are directly comparable and
nothing had to be re-indexed. Retrieval supplies the facts and the provenance;
the model supplies the prose. References are built from the metadata of the
passages actually retrieved, never from the model, so a reference cannot be
invented — the same property that makes an AI Mode citation trustworthy. A topic
with fewer than three grounding passages is skipped rather than written from
memory. Everything lands as a draft.
Two things worth naming. The generated text is original writing grounded in those
books, not extracts from them: their facts are usable, their sentences are their
publishers'. And there are two Milvus servers on this host — the collection with
the data is the one reached as `milvus`, not the similarly named one on the other
stack, which I wired up first and which silently refused.
Also fixed along the way: `litellm==1.28.13` has been withdrawn from PyPI, so
requirements.txt could no longer be resolved from scratch and the image only
built because of a cached layer. Later additions go in their own layer until the
pins are refreshed.
182 backend, 223 frontend green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
Storage
Media now goes through `storage_service`, which has two backends: the container
volume, and S3/MinIO. A volume can only be mounted by one host, has no presigned
URLs and no lifecycle rules, none of which suits ~860 MB of media. Reads fall
back to the volume when an object is missing, so the existing uploads keep
working and files can migrate gradually rather than in one risky pass.
A row stores the object key, never a URL: a URL embeds the backend, so a row
holding `http://minio:9000/...` breaks the moment the backend changes.
MinIO publishes no host ports — the backend reaches it over the compose network,
and 9000/9001 are already taken on this host by other stacks.
Image libraries (migration d2e3f4a5b6c7)
An image belongs to a library, and a person is granted a library the way they are
granted a category, so access can be given to some images without giving away all
of them. Tags reuse the shared `question_tags` vocabulary rather than inventing a
media-only one. Uploads are type- and size-checked, stored through the service,
and embedded so an image can be found by what it shows.
Classification finished
The 316 questions the chooser had declined are now filed with `--force`, which
takes the nearest candidate from the same shortlist the chooser saw. 306 were
forced, 10 the chooser accepted on this pass. No question sits on a bare system
any more:
system only 2,730 -> 0
condition/subsystem 214 -> 1,782
full depth 4 -> 1,166
A forced match is a weaker signal than a chosen one, so expect more errors among
those 306 — but the original system stays as a cross-link, so nothing is lost and
they can be corrected by hand.
Tests: 8 new backend covering library scoping, edit confinement, shared-vocabulary
tags, storage indirection on upload, and type/size limits. 131 backend green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WgRcMaScVEL7TBLpnAoSV9
Search
- Retrieval was hybrid in name only: the keyword filter was applied to the SQL
query, so results were the *intersection* of the two rankers. A question that
matched the meaning but not the literal string could never be returned. It is
now a union, fused with Reciprocal Rank Fusion (a text rank and a cosine
distance are not on comparable scales, so RRF uses only their orderings).
- Added a generated `search_vector` tsvector + GIN index, so the lexical half is
ranked full text rather than ILIKE substring matching.
- Chose Postgres + pgvector over OpenSearch/Elasticsearch: a search cluster
would add a second datastore to keep in sync and a JVM on this host, to
replace an index Postgres maintains inside the same transaction.
- Removed the keyword-only mode. It looks precise but silently drops the
question that asks the same thing in different words.
Embeddings — measured on 500 real questions, using each question's own
explanation as a paraphrase query (known answer, no hand labelling):
bge-small (local CPU, 384d) R@1 0.840 R@5 0.953 186ms/query
bge-m3 (LiteLLM proxy, 1024d) R@1 0.847 R@5 0.973 93ms/query
BGE-M3 wins on both quality and latency and needs no extra credential, since
llm.danvics.com already serves `openrouter-bge-m3`.
Three gaps this exposed, all fixed:
- Nothing recorded which model produced a stored vector, so changing models
silently mixed incomparable spaces. `embedding_model` / `embedded_at` now
stamp every vector, `GET /admin/embedding/health` reports current vs stale vs
missing, and regeneration defaults to stale-only.
- The generator read the model from env while the stamp read a Redis override,
so a vector could be labelled with a model that did not produce it. Both now
resolve through one function, with a regression test.
- Embedding at creation is best effort, and a failure left a question invisible
to semantic search forever. `retry_missing_embeddings` runs every 15 minutes
via Celery beat and backfills missing or stale rows.
- Query embeddings are cached in Redis per model, so typing is not a network
round-trip per keystroke.
`dimensions` is only sent to OpenAI's embedding-3 family; BGE-M3 rejects it.
Tests: 8 new backend tests (union not intersection, fusion ordering, per-ranker
failure degradation, provenance stamping, stale/missing accounting, generator
and stamp agreement). Full suites green: 95 backend, 127 frontend, build clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014yhHB8Pc7oQqyqn2Vo9DXA
LMS / Course System:
- Course → Module → Lesson hierarchy (any user can create)
- Lesson types: text, video (Vimeo/YouTube/local), document, quiz, live_session
- Video provider auto-detection from URL
- BBB API integration (create/join meetings) with env config
- Course enrollment with per-lesson progress tracking
- AI content generation/refinement for text lessons
- Draft/published/archived status workflow
- Subscription gate placeholder (requires_subscription flag)
- Thumbnail upload support
- Module/lesson reorder with up/down controls
User Quiz Creation:
- Any user can create quizzes from question bank (was moderator-only)
- User quizzes: is_published=0, is_shared=0 (private by default)
- Fixed section_id fallback: None instead of hardcoded 1
Manual Question Creation:
- POST /questions/create endpoint for manual MCQ entry
- CreateQuestionModal on QuestionBankPage with options, radio for correct answer
- Auto-generates embedding on creation
Frontend:
- CoursesPage: Browse/My Courses/Created tabs with search and pagination
- CourseDetailPage: Student view with module accordion, lesson viewer, progress
- CourseEditorPage: Full course builder with AI generate, question bank browser
- Courses link in Navbar
- Create Question button on Question Bank (available to all users)
Backend:
- 5 new tables: courses, course_modules, course_lessons, course_enrollments, course_lesson_progress
- Course model + schemas + router (22 endpoints)
- BBB_SERVER_URL + BBB_SECRET config
- Updated CLAUDE.md with LMS documentation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- FastAPI backend with JWT auth, roles (admin/moderator/user)
- PDF upload (up to 500MB) with streaming, PyMuPDF text extraction
- ChromaDB vectorization per page with metadata
- LiteLLM AI question extraction from PDF (not generation)
- Image extraction from PDF pages, graceful fallback
- Quiz modes: timed (countdown timer) + learning (answers shown inline)
- Page-by-page question navigation with dot navigator
- TTS endpoint using LiteLLM (Google Vertex / OpenAI voices)
- Admin dashboard: AI model management per task, user role management
- Moderator role: upload PDFs, create sections, generate quizzes
- Spaced repetition reminders via SMTP email (SM-2 intervals)
- APScheduler daily reminder jobs
- Celery + Redis for background PDF processing
- React frontend with all pages
- Docker Compose deployment (nginx + backend + celery + redis)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
