"Questions filed there later are not added" was the honest description of what
the previous commit built, and it was the wrong thing to build. "The Cardiology
article covers the Cardiology questions" is a standing statement about the
material, not a snapshot of who happened to be filed where on the afternoon
somebody pressed a button — and a copy stops being true the first time a
question is added, silently, with nothing on any screen to say so.
So the claim is now stored, and it is what writes the links:
* `question_article_links` is still the **only** table anything reads. No count,
no QBank button, no mirror panel on a question, no AI Mode boost learns a
second question to ask.
* `article_topic_claims` records *why* some of those rows exist, and is the one
place that makes them — when the claim is staked, when a question is filed
into the category (single, bulk, or on create), and on a half-hourly sweep
that catches whatever bypassed both.
A link made this way is an ordinary row and can still be deleted by hand; a
sweep puts it back, which is the honest consequence of a standing claim.
Dropping the claim is how you stop it, and the panel now lists what an article
follows with two ways out — stop following and keep the links, or stop and
remove them.
Migration k1b2c3d4e5f6.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqXevQJhxFrM7jJg82cgZN
Search
- Retrieval was hybrid in name only: the keyword filter was applied to the SQL
query, so results were the *intersection* of the two rankers. A question that
matched the meaning but not the literal string could never be returned. It is
now a union, fused with Reciprocal Rank Fusion (a text rank and a cosine
distance are not on comparable scales, so RRF uses only their orderings).
- Added a generated `search_vector` tsvector + GIN index, so the lexical half is
ranked full text rather than ILIKE substring matching.
- Chose Postgres + pgvector over OpenSearch/Elasticsearch: a search cluster
would add a second datastore to keep in sync and a JVM on this host, to
replace an index Postgres maintains inside the same transaction.
- Removed the keyword-only mode. It looks precise but silently drops the
question that asks the same thing in different words.
Embeddings — measured on 500 real questions, using each question's own
explanation as a paraphrase query (known answer, no hand labelling):
bge-small (local CPU, 384d) R@1 0.840 R@5 0.953 186ms/query
bge-m3 (LiteLLM proxy, 1024d) R@1 0.847 R@5 0.973 93ms/query
BGE-M3 wins on both quality and latency and needs no extra credential, since
llm.danvics.com already serves `openrouter-bge-m3`.
Three gaps this exposed, all fixed:
- Nothing recorded which model produced a stored vector, so changing models
silently mixed incomparable spaces. `embedding_model` / `embedded_at` now
stamp every vector, `GET /admin/embedding/health` reports current vs stale vs
missing, and regeneration defaults to stale-only.
- The generator read the model from env while the stamp read a Redis override,
so a vector could be labelled with a model that did not produce it. Both now
resolve through one function, with a regression test.
- Embedding at creation is best effort, and a failure left a question invisible
to semantic search forever. `retry_missing_embeddings` runs every 15 minutes
via Celery beat and backfills missing or stale rows.
- Query embeddings are cached in Redis per model, so typing is not a network
round-trip per keystroke.
`dimensions` is only sent to OpenAI's embedding-3 family; BGE-M3 rejects it.
Tests: 8 new backend tests (union not intersection, fusion ordering, per-ranker
failure degradation, provenance stamping, stale/missing accounting, generator
and stamp agreement). Full suites green: 95 backend, 127 frontend, build clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014yhHB8Pc7oQqyqn2Vo9DXA
Extraction is now fully async via Celery — UI shows a live progress panel,
job continues even if page is closed. Large documents are processed in
50-page chunks to extract all questions (not just first ~50 pages).
Backend:
- app/tasks/quiz_tasks.py: new Celery task 'extract_quiz'
- Writes step-by-step progress to Redis (extraction:steps:{job_id})
- Splits large page ranges into 50-page chunks, processes each separately
- Reports per-chunk results and running total
- Falls back to synchronous if Celery/Redis unavailable
- POST /quizzes/ now returns {job_id, status:"pending"} immediately
- GET /quizzes/job/{job_id} polls progress: steps[], status, quiz_id on completion
- Celery task list updated to include quiz_tasks
Frontend (DocumentDetailPage):
- ExtractionProgress modal component: monospace step log, auto-scrolls, spinner
- Polls job status every 2 seconds via /quizzes/job/{job_id}
- "Open Quiz →" button appears when done
- "✕ closes — job continues in background" shown while running
- beforeunload warning when job is active (preventing accidental close)
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- FastAPI backend with JWT auth, roles (admin/moderator/user)
- PDF upload (up to 500MB) with streaming, PyMuPDF text extraction
- ChromaDB vectorization per page with metadata
- LiteLLM AI question extraction from PDF (not generation)
- Image extraction from PDF pages, graceful fallback
- Quiz modes: timed (countdown timer) + learning (answers shown inline)
- Page-by-page question navigation with dot navigator
- TTS endpoint using LiteLLM (Google Vertex / OpenAI voices)
- Admin dashboard: AI model management per task, user role management
- Moderator role: upload PDFs, create sections, generate quizzes
- Spaced repetition reminders via SMTP email (SM-2 intervals)
- APScheduler daily reminder jobs
- Celery + Redis for background PDF processing
- React frontend with all pages
- Docker Compose deployment (nginx + backend + celery + redis)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
