pdf-quiz-generator/CLAUDE.md

# PedsHub — AI Synopsis for Codebase Work

## What this is
PedsHub is a pediatric medical learning platform with an integrated LMS. Admins upload PREP exam PDFs, AI extracts MCQ questions (or generates flashcards), and users study them with an AI tutor. Any user can create courses, quizzes, and questions. Everything runs in Docker.

## Critical rules
- **Never restart services while a Celery task is running** — check `docker compose logs celery --tail=5` first
- **Backend and Celery share the same code but build separate images** — after changing backend code, you must `docker compose build --no-cache backend celery` then `docker compose up -d backend celery --force-recreate`
- **Frontend is a Vite build inside Docker** — source changes require `docker compose build frontend` then `docker compose up -d frontend`. Vite minifies function names, so `grep FunctionName` on the built JS won't work.
- **Never import inside a function body if the same name exists at module level** — Python treats it as a local variable for the entire function scope, causing `UnboundLocalError` before the import line executes. This was a real bug with `sa_text`.
- **`docker compose restart` does NOT pick up code changes** — it reuses the old image. Always `build` then `up -d --force-recreate`.
- **Pydantic schemas must match DB nullability** — if a column allows NULL, the schema field must be `type | None`. A mismatch causes 500 on serialization.
- **The Question model uses `source_quiz_id` as the Python attribute but `quiz_id` as the DB column** — use `Question.source_quiz_id` in SQLAlchemy filters, never `Question.quiz_id`.

## Stack
- **Backend**: FastAPI + SQLAlchemy + PostgreSQL 16 (pgvector) + Redis + Celery
- **Frontend**: React 18 + Vite + React Router 6 + plain CSS + Milkdown (markdown WYSIWYG) + Nginx
- **AI**: LiteLLM proxy routes to Claude/GPT/Gemini/Bedrock. `_proxy_model()` in ai_service.py adds `openai/` prefix for the proxy.
- **Vectors**: ChromaDB for document page chunks (RAG), pgvector for question embeddings (semantic search)
- **Config**: Backend reads `.env` via pydantic-settings. Frontend uses runtime `window.__APP_CONFIG__` injected by `docker-entrypoint.sh` (not Vite build-time env).

## Architecture
```
Browser → Nginx (frontend) → FastAPI (4 uvicorn workers)
                                ├── PostgreSQL (users, quizzes, questions, flashcards, attempts + pgvector embeddings)
                                ├── ChromaDB (document page chunks for extraction context)
                                ├── Redis (Celery broker, rate limits, settings, job progress, session locks)
                                └── Celery (2 fork workers: PDF processing, quiz extraction, flashcard generation, classification, embedding regeneration)
                                └── db-backup (daily PostgreSQL dumps, 14 daily / 4 weekly / 6 monthly retention, ./backups/)

## Course/LMS system
Any user can create courses (not just moderators). Course structure: Course → Modules → Lessons.
- **Lesson types**: text (markdown), video (Vimeo/YouTube/local), document, quiz (from question bank), live_session (BBB/Zoom/Meet)
- **Enrollment**: users enroll in published courses, progress tracked per-lesson
- **Video**: auto-detect provider from URL (vimeo.com → vimeo, youtube.com → youtube)
- **BBB**: full API integration (create/join/end). Config: `BBB_SERVER_URL` + `BBB_SECRET` in .env
- **AI**: generate/refine lesson text via `POST /courses/{id}/lessons/{id}/ai-generate`
- **Status**: draft (creator-only) → published (visible to all) → archived
- **Subscription gate**: `requires_subscription` flag on courses (402 on enroll if set — Stripe integration placeholder)
- **Course quizzes**: fully decoupled from main quiz system. `POST /courses/{id}/quiz` copies questions and creates an independent quiz with `course_id` set. Hidden from main quizzes page, search, dashboard stats, and attempt history. Creator sets mode (timed/study), time limit, max attempts, questions per attempt (random pool), and `allow_review` (whether students can review answers). Results page is course-aware — shows "Back to Course" instead of retake/all quizzes/delete. Users see attempt history + review links on the course page only.
- **User roles**: `admin`, `moderator`, `user`. Only moderators and admins can create courses. Any user can create quizzes from the question bank.
- **Enrollee analytics**: `GET /courses/{id}/enrollees` returns progress + quiz scores. `GET /courses/{id}/enrollees/export` exports CSV.
- **Question ownership**: questions have `user_id` and `is_shared`. Users see shared + own questions. `PATCH /questions/{id}/share` toggles visibility.
- **Rich editor**: Milkdown (ProseMirror-based markdown WYSIWYG) for lesson content. Supports GFM tables, code blocks, LaTeX math (`$formula$`). No JSX parsing issues with `<` or `{`.
```

## Key directories
```
backend/app/
  main.py           — App startup, DDL migrations (setup_pgvector), router mounting, singleton lock
  config.py         — All settings from .env
  models/           — SQLAlchemy ORM (user, quiz, question, flashcard, attempt, section, pdf_document, ...)
  schemas/          — Pydantic request/response models
  routers/          — API endpoints (auth, quizzes, questions, flashcards, attempts, admin, teach, tts, tags, ...)
  services/
    ai_service.py           — LLM calls, _proxy_model(), get_model_for_task() fallback chain
    extraction_modes.py     — 6 quiz extraction modes + flashcard generation prompt
    search_service.py       — Hybrid retrieval (BM25 + pgvector, RRF), rerank_ids()
    rerank_service.py       — Cross-encoder rerank via the proxy's /v1/rerank
    vector_service.py       — ChromaDB: store/query page chunks, LiteLLMEmbeddingFunction
    embedding_service.py    — pgvector: embed questions for semantic search
    pdf_service.py          — PyMuPDF: text extraction, image extraction with MD5 hash skip list
  tasks/
    quiz_tasks.py           — Celery: extract_quiz, classify_questions, regenerate_embeddings, generate_flashcard_deck
    pdf_tasks.py            — Celery: process_pdf (text extraction + vectorization)

frontend/src/
  App.jsx                   — Routes (public, authenticated, moderator-only)
  context/AuthContext.jsx   — Login/logout/register, JWT token management
  pages/
    DocumentDetailPage.jsx  — Section management, "Extract Quiz" / "Create Flashcards" buttons, job progress
    QuestionBankPage.jsx    — Browse questions, multi-category + tag filtering, TagBrowser component
    FlashcardsPage.jsx      — Browse decks + card browser with search
    FlashcardStudyPage.jsx  — Flip cards, got-it/review, keyboard nav, progress
    QuizPage.jsx            — Take quiz (exam/study mode), timer, progress save to Redis
    CoursesPage.jsx         — Browse/enroll courses, my courses, create courses
    CourseDetailPage.jsx    — Student view: modules, lessons, video player, progress
    CourseEditorPage.jsx    — Course creator: modules, lessons, AI content, question bank browser
    AdminPage.jsx           — Model config, user management, settings
  components/
    Navbar.jsx              — Auth-aware nav with jobs badge
    TeachChat.jsx           — AI tutor drawer (lazy loaded, markdown/GFM tables)
    RichEditor.jsx          — Milkdown markdown WYSIWYG editor with toolbar, GFM tables, math, history
```

## Database tables (key ones)
| Table | Purpose | Key FKs |
|-------|---------|---------|
| users | Accounts with role (admin/moderator/user) | — |
| pdf_documents | Uploaded PDFs | user_id → users |
| sections | Page ranges within a document | document_id → pdf_documents |
| quizzes | Quiz metadata (course_id set = course-only, allow_review controls student access) | section_id → sections (nullable), user_id → users, course_id → courses (nullable) |
| questions | MCQ questions with pgvector embedding | source_quiz_id → quizzes (nullable), user_id → users (nullable) |
| quiz_question_links | Quiz ↔ Question many-to-many | quiz_id, question_id |
| flashcard_decks | Flashcard deck metadata | section_id → sections, user_id → users |
| flashcards | Individual cards (front/back) | deck_id → flashcard_decks |
| question_tags | Tag definitions (subject/disease/keyword) | — |
| question_tag_links | Question ↔ Tag | question_id, tag_id |
| flashcard_tag_links | Flashcard ↔ Tag | flashcard_id, tag_id |
| quiz_attempts | User quiz sessions with score | quiz_id, user_id |
| courses | LMS courses (draft/published/archived) | user_id → users |
| course_modules | Sections within a course | course_id → courses |
| course_lessons | Items within a module (text/video/quiz/live) | module_id → course_modules, quiz_id → quizzes |
| course_enrollments | User enrollment + progress tracking | course_id, user_id (unique) |
| course_lesson_progress | Per-lesson completion status | enrollment_id, lesson_id (unique) |

## Common patterns
- **Tag filtering SQL**: `WHERE tag_id = ANY(:tag_ids) GROUP BY ... HAVING COUNT(DISTINCT tag_id) = :cnt` — AND logic across tags
- **Multi-category filtering**: `category_ids` param (comma-separated), uses `.in_()` — OR logic within categories
- **Job progress**: Celery tasks push steps to Redis lists (`extraction:steps:{job_id}`), frontend polls `GET /quizzes/job/{job_id}`
- **Model selection**: Admin configures models per task (extraction, teach, article, tts, stt, keyword, flashcard, tool). `get_model_for_task(db, task)` returns (model_id, api_key) with fallback to `settings.LITELLM_MODEL`; `get_configured_model(db, task)` returns None instead of falling back, for work that must not run on a model nobody chose.
- **Vision fallback**: `vision_service.image_context(db, images, model_id=...)` returns the message parts to splice into a user message. If the job's model is vision-capable it gets the images; if not, the `tool` model describes them and the primary reads the description. Capability comes from the proxy's `/model/info` `supports_vision` (cached in-process), and where that is absent from a one-off 8px probe (cached in Redis) — never from a hard-coded list of model names. No tool model configured raises `VisionUnavailable`.
- **Reranking**: `search_service.rerank_ids(db, q, kind, ids)` reorders the first 50 of a fused ranking with a cross-encoder (`LITELLM_RERANK_MODEL`, default `cohere-rerank-v4.0-pro`, Redis override `settings:rerank_model`). It is a *permutation* — an unset, unreachable or malformed reranker returns the ids untouched, never fewer. Applied to question and article search, the test builder's description path, and the AI Mode shortlist; deliberately not to the typeahead, flashcard/media search, or the AI Mode closeness thresholds. Cached in Redis for a day, keyed on model + query + document text. See docs/reranking.md.
- **Uploads**: `file_intake.read()` then `kind_of()` then `text_from()` — size, then type sniffed from the leading bytes, then text. PDF, DOCX and images only (2 MB); an extension is a claim, never the decision. Images are read by the `tool` model, and say so when none is configured.
- **Small talk in AI Mode**: `ai_mode_service.is_small_talk()` runs *before* retrieval. A greeting scores 0.46–0.51 against a clinical corpus, which is either side of the adjacency threshold, so the closeness gate cannot be the guard. Mode "chat" means no shortlist, no citations, and no thread name.
- **`==key points==`**: a remark plugin (`utils/keyPoints.js`) → `<mark class="key-point">`. Never rewrite the markdown string before parsing: highlights and the read-aloud cursor are offsets into the raw text.
- **useEffect dependencies**: Use `.join(',')` on arrays to create a stable string key (e.g., `tagIdsKey`, `catIdsKey`)
- **Admin page data refresh**: `loadData(false)` — the `false` param skips the loading spinner on re-fetch after actions

## Database migrations (Alembic)

**What's a migration?** A migration is a small, ordered change to the database schema — adding a column, renaming a table, changing a type. Each change lives in a Python file under `backend/alembic/versions/`. Alembic tracks which ones have been applied in an `alembic_version` table inside Postgres, so it knows what's new next time you run it.

**Why it exists here:** until now, schema was created via `Base.metadata.create_all()` in `main.py:478`, which only creates *missing tables* — it never alters existing ones. Every column change required manual `ALTER TABLE`. Alembic makes schema changes versioned, reversible, and reproducible across environments.

**Current setup**
- `alembic.ini` contains no hardcoded URL; `alembic/env.py` injects `DATABASE_URL` from the container's env.
- Live DB is stamped at revision `9bac7bf02e38` (the latest in `alembic/versions/`).
- `Base.metadata.create_all()` remains in place as a fallback for fresh deploys — **don't remove it** without first generating a complete baseline migration from the live schema.

**Developer workflow**
```bash
# where am I?
docker compose exec backend alembic current
docker compose exec backend alembic heads

# create a new migration (auto-diff model vs live DB)
docker compose exec backend alembic revision --autogenerate -m "add some column"
# ↑ review the generated file under backend/alembic/versions/ before applying

# apply pending migrations
docker compose exec backend alembic upgrade head

# roll back the last one
docker compose exec backend alembic downgrade -1
```

**When to write one** — any schema change: new column, dropped column, renamed field, new table, altered index, new FK. Model edit → migration → apply → commit both together.

**Gotchas**
- Migrations run as a normal transaction. A failed migration rolls back cleanly.
- `--autogenerate` doesn't catch: server_default changes, CHECK constraints, enum value additions, data migrations. Hand-edit the file when needed.
- After applying a new migration in dev, rebuild the backend image (`docker compose build backend celery`) so it ships with the migration file baked in.
- The `alembic_version` table should only ever have one row. If you see multiple, you have branched heads — run `alembic merge` to reconcile.

## What NOT to do
- Don't add `from sqlalchemy import text as X` inside functions — import at module top only
- Don't use `Question.quiz_id` — it's `Question.source_quiz_id`
- Don't set `Content-Type: multipart/form-data` manually on axios FormData uploads — axios handles it
- Don't use `[someValue === null]` as a useEffect dependency — it evaluates to a constant boolean
- Don't `docker compose restart` expecting code changes to apply — must rebuild
- Don't use `window.confirm()` — user hates browser popups, use inline confirmation or the Dialog component
- Don't use MDXEditor — it's an MDX parser that chokes on `<` and `{` in medical content. Milkdown (CommonMark) is used instead.
- Don't create quizzes via `POST /questions/from-bank` for courses — use `POST /courses/{id}/quiz` which copies questions and hides the quiz from the main page.
- Don't show course quiz data on the main quizzes page, dashboard stats, or attempt history — course quizzes are fully decoupled. Filter with `Quiz.course_id.is_(None)`.
- Don't show "from pool of N" to users on course quiz display — just show the number of questions per attempt.
- Don't allow users to delete course quiz attempts — the backend returns 403.
- Don't put documents listing on the dashboard — it's in Settings page under Nextcloud.