pdf-quiz-generator/docs/deployment.md

PedsHub Deployment and Operations Guide

Docker Compose Setup

The application runs as 5 services defined in docker-compose.yml:

Service	Image	Purpose	Port
postgres	pgvector/pgvector:pg16	PostgreSQL with pgvector extension	Internal only
redis	redis:7-alpine	Caching, quiz progress, job tracking, Celery broker	Internal only
backend	Built from ./backend	FastAPI app (uvicorn, 4 workers)	Internal only
celery	Built from ./backend (same image)	Celery worker (concurrency=2)	None
frontend	Built from ./frontend	Nginx serving React SPA	127.0.0.1:8081:80

Volumes

Volume	Used by	Purpose
postgres_data	postgres	Database persistence
redis_data	redis	Redis persistence
uploads_data	backend, celery	Uploaded PDF files and extracted images
chroma_data	backend, celery	ChromaDB vector store

Health Checks

• postgres: pg_isready -U pedquiz every 5s, 10 retries.
• Backend and celery depend on postgres being healthy (condition: service_healthy) and redis being started.

Startup Order

1. Redis and Postgres start first.
2. Backend and Celery wait for Postgres to be healthy.
3. Frontend waits for backend to start.

---

Environment Variables

Backend (backend/.env)

Variable	Default	Description
DATABASE_URL	sqlite:///./quiz.db	PostgreSQL connection string. Production: postgresql://user:pass@postgres:5432/pedquiz
SECRET_KEY	change-me-...	JWT signing key. Generate with openssl rand -hex 32
ALGORITHM	HS256	JWT algorithm
ACCESS_TOKEN_EXPIRE_MINUTES	1440	Token lifetime (default 24 hours)
REDIS_URL	redis://localhost:6379/0	Redis connection URL
LITELLM_MODEL	gpt-4o-mini	Default LLM model for extraction
LITELLM_API_KEY	(empty)	API key for LiteLLM proxy or provider
LITELLM_API_BASE	(empty)	Custom API base URL (for LiteLLM proxy)
LITELLM_EMBEDDING_MODEL	(empty)	Model for semantic search embeddings
OPENAI_API_KEY	(empty)	OpenAI API key (if using OpenAI models directly)
ELEVENLABS_API_KEY	(empty)	ElevenLabs API key for TTS voices
GOOGLE_TTS_API_KEY	(empty)	Google Cloud TTS API key
AWS_ACCESS_KEY_ID	(empty)	AWS credentials for Bedrock / Polly
AWS_SECRET_ACCESS_KEY	(empty)	AWS secret key
AWS_REGION	us-east-1	AWS region
AWS_BEDROCK_REGION	us-east-1	AWS Bedrock region
EMBEDDING_DIMENSIONS	1024	Vector embedding dimension size
CHROMA_PERSIST_DIR	/app/chroma_data	ChromaDB storage directory
MAIL_USERNAME	(empty)	SMTP username
MAIL_PASSWORD	(empty)	SMTP password
MAIL_FROM	(empty)	From address for emails
MAIL_PORT	587	SMTP port
MAIL_SERVER	smtp.gmail.com	SMTP server hostname
MAIL_STARTTLS	true	Use STARTTLS
MAIL_SSL_TLS	false	Use SSL/TLS
UPLOAD_DIR	/app/uploads	Upload storage directory
MAX_UPLOAD_SIZE	524288000	Max upload size in bytes (500MB)
APP_URL	https://quiz.danvics.com	Public URL (used in emails, links)
TURNSTILE_SECRET_KEY	(empty)	Cloudflare Turnstile secret key. Leave blank to disable captcha
ADMIN_EMAIL	(empty)	Email address for contact form submissions

Frontend (frontend/.env)

Variable	Description
TURNSTILE_SITE_KEY	Cloudflare Turnstile site key (public). Injected at runtime via docker-entrypoint.sh into window.__APP_CONFIG__

---

HTTPS Setup with Caddy

The frontend binds to 127.0.0.1:8081 (not exposed externally). Use a reverse proxy for HTTPS.

Example Caddyfile:

quiz.example.com {
    reverse_proxy localhost:8081
}

Caddy handles automatic HTTPS certificate provisioning via Let's Encrypt. No additional TLS configuration is needed.

For setups where the backend needs to be accessed directly (rare):

quiz.example.com {
    reverse_proxy localhost:8081

    # Optional: direct backend access for debugging
    handle_path /api-direct/* {
        reverse_proxy localhost:8000
    }
}

Note: The frontend's Nginx already proxies /api to the backend container internally, so only the frontend port needs to be exposed.

---

Rebuilding

When to Rebuild vs Restart

Change	Action
Backend Python code	Rebuild backend: docker compose build backend
Frontend source code	Rebuild frontend: docker compose build frontend
Backend .env changes	Restart only: docker compose restart backend celery
Frontend .env changes	Restart only: docker compose restart frontend
requirements.txt changes	Rebuild backend: docker compose build backend
package.json changes	Rebuild frontend: docker compose build frontend
docker-compose.yml changes	docker compose up -d (recreates changed services)

Celery shares the backend image

The celery service builds from ./backend — the same Dockerfile as backend. Rebuilding backend requires rebuilding celery as well:

docker compose build backend
docker compose up -d backend celery

Or build both explicitly:

docker compose build backend celery
docker compose up -d

Stale code / cache issues

If code changes are not reflected after a rebuild, use --no-cache:

docker compose build --no-cache backend
docker compose build --no-cache frontend

This forces Docker to re-run all build steps, including pip install and npm ci.

---

Database

PostgreSQL with pgvector

The database uses pgvector/pgvector:pg16 which includes the vector extension. On startup, the backend runs:

CREATE EXTENSION IF NOT EXISTS vector;
ALTER TABLE questions ADD COLUMN IF NOT EXISTS embedding vector(1024);
CREATE INDEX IF NOT EXISTS questions_embedding_hnsw
    ON questions USING hnsw (embedding vector_cosine_ops);

Connection Pool

SQLAlchemy engine is configured with:

engine = create_engine(settings.DATABASE_URL, pool_pre_ping=True, pool_recycle=300)

• pool_pre_ping=True — Tests connections before use. Prevents "connection closed" errors after database restarts.
• pool_recycle=300 — Recycles connections every 5 minutes. Prevents stale connections in long-running processes (Celery workers).

Backups

# Dump the database
docker compose exec postgres pg_dump -U pedquiz pedquiz > backup_$(date +%Y%m%d).sql

# Restore from backup
docker compose exec -T postgres psql -U pedquiz pedquiz < backup_20240101.sql

For automated backups, add a cron job on the host:

0 2 * * * cd /home/danvics/docker/quiz && docker compose exec -T postgres pg_dump -U pedquiz pedquiz | gzip > /backups/pedquiz_$(date +\%Y\%m\%d).sql.gz

---

Monitoring

Checking Logs

# Backend logs (FastAPI + uvicorn)
docker compose logs -f backend

# Celery worker logs (extraction jobs, embeddings)
docker compose logs -f celery

# All services
docker compose logs -f

# Last 100 lines
docker compose logs --tail=100 backend

Common Errors and Fixes

Error	Cause	Fix
connection already closed	Stale DB connection in Celery	Already handled by pool_pre_ping and pool_recycle. If persistent, restart celery
lock timeout on startup	Previous killed process holding DDL locks	Backend auto-terminates stale connections and retries 3 times with 5s delay
429 / rate limit from LLM API	Too many extraction requests	Reduce celery concurrency or add rate limiting config
embedding failed	Embedding API rate limit or model issue	Check LITELLM_EMBEDDING_MODEL config, verify API key
CORS error in browser	Frontend not going through Nginx proxy	Ensure requests go through the frontend's Nginx (which proxies /api)
502 Bad Gateway	Backend not ready yet	Wait for backend to start, check docker compose logs backend

---

Troubleshooting

Lock Timeout on Startup

The backend runs DDL migrations (ALTER TABLE, CREATE INDEX) at startup. If a previous instance was killed while holding a lock, the new instance will hang. The startup code handles this automatically:

1. Terminates stale idle in transaction connections older than 30 seconds.
2. Sets lock_timeout = '15s' before running DDL.
3. Retries up to 3 times with 5-second delays if a lock timeout occurs.

If this still fails, manually kill stale connections:

docker compose exec postgres psql -U pedquiz -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle in transaction' AND pid <> pg_backend_pid();"

Singleton Lock for Startup Tasks

The backend uses a Redis-based singleton lock (startup:singleton_lock, 5-minute TTL) to ensure that only one of the 4 uvicorn workers runs the scheduler and backfill tasks. If Redis is unavailable, it assumes single-worker mode and runs everything.

Stale DB Connections in Celery Tasks

Celery workers are long-running processes. Without pool_recycle=300, connections can go stale (especially after postgres restarts). The pool_pre_ping=True setting validates connections before use. If database errors appear in celery logs after a postgres restart, restart the celery service:

docker compose restart celery

Embedding Batch Size / Rate Limits

Embedding generation happens during quiz extraction and on startup (backfill). If the embedding API has rate limits, failures may appear in the celery logs. The embedding service handles individual failures gracefully (logs and continues). To re-embed all questions:

docker compose exec backend python manage.py reembed

Docker Build Cache Issues

If Python dependencies or JS packages have been updated but the build is reusing cached layers:

# Force fresh install of all dependencies
docker compose build --no-cache backend frontend
docker compose up -d

---

Scaling

Worker Counts

• Backend (uvicorn): Runs with --workers 4. For higher throughput, increase in docker-compose.yml. Each worker runs the startup lifespan (idempotent DDL + singleton lock for scheduler).
• Celery: Runs with --concurrency=2. This controls how many extraction jobs run in parallel. Increase for faster throughput, but watch LLM API rate limits.

Redis Memory

Redis stores:

• Quiz progress (per-user, per-quiz)
• Extraction job status and steps
• Celery broker messages
• Singleton lock
• User job lists

For most deployments, default Redis memory is sufficient. Monitor with:

docker compose exec redis redis-cli info memory

ChromaDB Storage

ChromaDB stores question embeddings for semantic search. Storage grows with the number of questions. The chroma_data volume persists this data. Monitor disk usage:

docker system df -v | grep chroma

---

Updating

Standard Update Flow

cd /home/danvics/docker/quiz
git pull
docker compose build backend frontend
docker compose up -d

Migration Safety

Database migrations run automatically on startup via Base.metadata.create_all() and explicit DDL in setup_pgvector(). These are idempotent:

• CREATE TABLE IF NOT EXISTS
• ADD COLUMN IF NOT EXISTS
• CREATE INDEX IF NOT EXISTS

DDL statements use SET lock_timeout = '15s' and retry up to 3 times to handle concurrent lock contention from Celery workers or other uvicorn processes.

Zero-Downtime Updates

For zero-downtime updates (if needed):

# Build new images first
docker compose build backend frontend

# Restart one service at a time
docker compose up -d --no-deps backend
docker compose up -d --no-deps celery
docker compose up -d --no-deps frontend

---

CLI Tools

The manage.py script provides management commands. Run inside the backend container:

docker compose exec backend python manage.py <command>

Commands

reset-password <email> <new-password>

Reset a user's password. Password must be at least 8 characters.

docker compose exec backend python manage.py reset-password user@example.com newpassword123

list-users

List all users with ID, email, role, verified status, and name.

docker compose exec backend python manage.py list-users

Output:

  ID  Email                                     Role          Verified  Name
------------------------------------------------------------------------------------------
   1  admin@example.com                         admin         yes       Admin
   2  user@example.com                          user          yes       John

reembed

Regenerate all question embeddings. Useful after changing the embedding model. Clears existing embeddings and re-generates them one by one.

docker compose exec backend python manage.py reembed

extract <section_id> [options]

Run quiz extraction from a section. Can run inline (blocking) or in background (via Celery).

# Inline extraction (blocking, shows live output)
docker compose exec backend python manage.py extract 5

# With options
docker compose exec backend python manage.py extract 5 --title "Chapter 3 Quiz" --mode learning --user admin@example.com

# Background (via Celery)
docker compose exec backend python manage.py extract 5 --bg

Options:

• --title — Quiz title (default: auto-generated from section name)
• --mode — timed or learning (default: timed)
• --user — User email to assign the quiz to (default: first admin user)
• --bg — Run in background via Celery

list-sections [doc_id]

List all documents and their sections. Optionally filter by document ID.

# All documents
docker compose exec backend python manage.py list-sections

# Specific document
docker compose exec backend python manage.py list-sections 3

Output:

Doc 3: PREP_2024.pdf (ready, 450 pages)
  Section 5: 'Chapter 1'  pages 1-50
  Section 6: 'Chapter 2'  pages 51-120

jobs [--user email]

Show recent extraction jobs from Redis. Optionally filter by user email.

# All users' jobs
docker compose exec backend python manage.py jobs

# Specific user
docker compose exec backend python manage.py jobs --user admin@example.com

Output:

admin@example.com:
  [completed ] Quiz: Chapter 1                           steps= 12 quiz_id=7
              Saved 25 questions to quiz
  [running   ] Quiz: Chapter 2                           steps=  5
              Extracting questions from pages 51-80...