# PedsHub Deployment and Operations Guide

## Docker Compose Setup

The application runs as 5 services defined in `docker-compose.yml`:

| Service | Image | Purpose | Port |
|---------|-------|---------|------|
| `postgres` | `pgvector/pgvector:pg16` | PostgreSQL with pgvector extension | Internal only |
| `redis` | `redis:7-alpine` | Caching, quiz progress, job tracking, Celery broker | Internal only |
| `backend` | Built from `./backend` | FastAPI app (uvicorn, 4 workers) | Internal only |
| `celery` | Built from `./backend` (same image) | Celery worker (concurrency=2) | None |
| `frontend` | Built from `./frontend` | Nginx serving React SPA | `127.0.0.1:8081:80` |

### Volumes

| Volume | Used by | Purpose |
|--------|---------|---------|
| `postgres_data` | postgres | Database persistence |
| `redis_data` | redis | Redis persistence |
| `uploads_data` | backend, celery | Uploaded PDF files and extracted images |
| `chroma_data` | backend, celery | ChromaDB vector store |

### Health Checks

- **postgres**: `pg_isready -U pedquiz` every 5s, 10 retries.
- Backend and celery depend on postgres being healthy (`condition: service_healthy`) and redis being started.

### Startup Order

1. Redis and Postgres start first.
2. Backend and Celery wait for Postgres to be healthy.
3. Frontend waits for backend to start.

---

## Environment Variables

### Backend (`backend/.env`)

| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `sqlite:///./quiz.db` | PostgreSQL connection string. Production: `postgresql://user:pass@postgres:5432/pedquiz` |
| `SECRET_KEY` | `change-me-...` | JWT signing key. Generate with `openssl rand -hex 32` |
| `ALGORITHM` | `HS256` | JWT algorithm |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | `1440` | Token lifetime (default 24 hours) |
| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
| `LITELLM_MODEL` | `gpt-4o-mini` | Default LLM model for extraction |
| `LITELLM_API_KEY` | (empty) | API key for LiteLLM proxy or provider |
| `LITELLM_API_BASE` | (empty) | Custom API base URL (for LiteLLM proxy) |
| `LITELLM_EMBEDDING_MODEL` | (empty) | Model for semantic search embeddings |
| `OPENAI_API_KEY` | (empty) | OpenAI API key (if using OpenAI models directly) |
| `ELEVENLABS_API_KEY` | (empty) | ElevenLabs API key for TTS voices |
| `GOOGLE_TTS_API_KEY` | (empty) | Google Cloud TTS API key |
| `AWS_ACCESS_KEY_ID` | (empty) | AWS credentials for Bedrock / Polly |
| `AWS_SECRET_ACCESS_KEY` | (empty) | AWS secret key |
| `AWS_REGION` | `us-east-1` | AWS region |
| `AWS_BEDROCK_REGION` | `us-east-1` | AWS Bedrock region |
| `EMBEDDING_DIMENSIONS` | `1024` | Vector embedding dimension size |
| `CHROMA_PERSIST_DIR` | `/app/chroma_data` | ChromaDB storage directory |
| `MAIL_USERNAME` | (empty) | SMTP username |
| `MAIL_PASSWORD` | (empty) | SMTP password |
| `MAIL_FROM` | (empty) | From address for emails |
| `MAIL_PORT` | `587` | SMTP port |
| `MAIL_SERVER` | `smtp.gmail.com` | SMTP server hostname |
| `MAIL_STARTTLS` | `true` | Use STARTTLS |
| `MAIL_SSL_TLS` | `false` | Use SSL/TLS |
| `UPLOAD_DIR` | `/app/uploads` | Upload storage directory |
| `MAX_UPLOAD_SIZE` | `524288000` | Max upload size in bytes (500MB) |
| `APP_URL` | `https://quiz.danvics.com` | Public URL (used in emails, links) |
| `TURNSTILE_SECRET_KEY` | (empty) | Cloudflare Turnstile secret key. Leave blank to disable captcha |
| `ADMIN_EMAIL` | (empty) | Email address for contact form submissions |

### Frontend (`frontend/.env`)

| Variable | Description |
|----------|-------------|
| `TURNSTILE_SITE_KEY` | Cloudflare Turnstile site key (public). Injected at runtime via `docker-entrypoint.sh` into `window.__APP_CONFIG__` |

---

## HTTPS Setup with Caddy

The frontend binds to `127.0.0.1:8081` (not exposed externally). Use a reverse proxy for HTTPS.

Example Caddyfile:

```caddyfile
quiz.example.com {
    reverse_proxy localhost:8081
}
```

Caddy handles automatic HTTPS certificate provisioning via Let's Encrypt. No additional TLS configuration is needed.

For setups where the backend needs to be accessed directly (rare):

```caddyfile
quiz.example.com {
    reverse_proxy localhost:8081

    # Optional: direct backend access for debugging
    handle_path /api-direct/* {
        reverse_proxy localhost:8000
    }
}
```

Note: The frontend's Nginx already proxies `/api` to the backend container internally, so only the frontend port needs to be exposed.

---

## Rebuilding

### When to Rebuild vs Restart

| Change | Action |
|--------|--------|
| Backend Python code | Rebuild backend: `docker compose build backend` |
| Frontend source code | Rebuild frontend: `docker compose build frontend` |
| Backend `.env` changes | Restart only: `docker compose restart backend celery` |
| Frontend `.env` changes | Restart only: `docker compose restart frontend` |
| `requirements.txt` changes | Rebuild backend: `docker compose build backend` |
| `package.json` changes | Rebuild frontend: `docker compose build frontend` |
| `docker-compose.yml` changes | `docker compose up -d` (recreates changed services) |

### Celery shares the backend image

The `celery` service builds from `./backend` — the same Dockerfile as `backend`. Rebuilding `backend` requires rebuilding `celery` as well:

```bash
docker compose build backend
docker compose up -d backend celery
```

Or build both explicitly:

```bash
docker compose build backend celery
docker compose up -d
```

### Stale code / cache issues

If code changes are not reflected after a rebuild, use `--no-cache`:

```bash
docker compose build --no-cache backend
docker compose build --no-cache frontend
```

This forces Docker to re-run all build steps, including `pip install` and `npm ci`.

---

## Database

### PostgreSQL with pgvector

The database uses `pgvector/pgvector:pg16` which includes the `vector` extension. On startup, the backend runs:

```sql
CREATE EXTENSION IF NOT EXISTS vector;
ALTER TABLE questions ADD COLUMN IF NOT EXISTS embedding vector(1024);
CREATE INDEX IF NOT EXISTS questions_embedding_hnsw
    ON questions USING hnsw (embedding vector_cosine_ops);
```

### Connection Pool

SQLAlchemy engine is configured with:

```python
engine = create_engine(settings.DATABASE_URL, pool_pre_ping=True, pool_recycle=300)
```

- `pool_pre_ping=True` — Tests connections before use. Prevents "connection closed" errors after database restarts.
- `pool_recycle=300` — Recycles connections every 5 minutes. Prevents stale connections in long-running processes (Celery workers).

### Backups

```bash
# Dump the database
docker compose exec postgres pg_dump -U pedquiz pedquiz > backup_$(date +%Y%m%d).sql

# Restore from backup
docker compose exec -T postgres psql -U pedquiz pedquiz < backup_20240101.sql
```

For automated backups, add a cron job on the host:

```bash
0 2 * * * cd /home/danvics/docker/quiz && docker compose exec -T postgres pg_dump -U pedquiz pedquiz | gzip > /backups/pedquiz_$(date +\%Y\%m\%d).sql.gz
```

---

## Monitoring

### Checking Logs

```bash
# Backend logs (FastAPI + uvicorn)
docker compose logs -f backend

# Celery worker logs (extraction jobs, embeddings)
docker compose logs -f celery

# All services
docker compose logs -f

# Last 100 lines
docker compose logs --tail=100 backend
```

### Common Errors and Fixes

| Error | Cause | Fix |
|-------|-------|-----|
| `connection already closed` | Stale DB connection in Celery | Already handled by `pool_pre_ping` and `pool_recycle`. If persistent, restart celery |
| `lock timeout` on startup | Previous killed process holding DDL locks | Backend auto-terminates stale connections and retries 3 times with 5s delay |
| `429 / rate limit` from LLM API | Too many extraction requests | Reduce celery concurrency or add rate limiting config |
| `embedding failed` | Embedding API rate limit or model issue | Check LITELLM_EMBEDDING_MODEL config, verify API key |
| `CORS error` in browser | Frontend not going through Nginx proxy | Ensure requests go through the frontend's Nginx (which proxies /api) |
| `502 Bad Gateway` | Backend not ready yet | Wait for backend to start, check `docker compose logs backend` |

---

## Troubleshooting

### Lock Timeout on Startup

The backend runs DDL migrations (ALTER TABLE, CREATE INDEX) at startup. If a previous instance was killed while holding a lock, the new instance will hang. The startup code handles this automatically:

1. Terminates stale `idle in transaction` connections older than 30 seconds.
2. Sets `lock_timeout = '15s'` before running DDL.
3. Retries up to 3 times with 5-second delays if a lock timeout occurs.

If this still fails, manually kill stale connections:

```bash
docker compose exec postgres psql -U pedquiz -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle in transaction' AND pid <> pg_backend_pid();"
```

### Singleton Lock for Startup Tasks

The backend uses a Redis-based singleton lock (`startup:singleton_lock`, 5-minute TTL) to ensure that only one of the 4 uvicorn workers runs the scheduler and backfill tasks. If Redis is unavailable, it assumes single-worker mode and runs everything.

### Stale DB Connections in Celery Tasks

Celery workers are long-running processes. Without `pool_recycle=300`, connections can go stale (especially after postgres restarts). The `pool_pre_ping=True` setting validates connections before use. If database errors appear in celery logs after a postgres restart, restart the celery service:

```bash
docker compose restart celery
```

### Embedding Batch Size / Rate Limits

Embedding generation happens during quiz extraction and on startup (backfill). If the embedding API has rate limits, failures may appear in the celery logs. The embedding service handles individual failures gracefully (logs and continues). To re-embed all questions:

```bash
docker compose exec backend python manage.py reembed
```

### Docker Build Cache Issues

If Python dependencies or JS packages have been updated but the build is reusing cached layers:

```bash
# Force fresh install of all dependencies
docker compose build --no-cache backend frontend
docker compose up -d
```

---

## Scaling

### Worker Counts

- **Backend (uvicorn)**: Runs with `--workers 4`. For higher throughput, increase in `docker-compose.yml`. Each worker runs the startup lifespan (idempotent DDL + singleton lock for scheduler).
- **Celery**: Runs with `--concurrency=2`. This controls how many extraction jobs run in parallel. Increase for faster throughput, but watch LLM API rate limits.

### Redis Memory

Redis stores:
- Quiz progress (per-user, per-quiz)
- Extraction job status and steps
- Celery broker messages
- Singleton lock
- User job lists

For most deployments, default Redis memory is sufficient. Monitor with:

```bash
docker compose exec redis redis-cli info memory
```

### ChromaDB Storage

ChromaDB stores question embeddings for semantic search. Storage grows with the number of questions. The `chroma_data` volume persists this data. Monitor disk usage:

```bash
docker system df -v | grep chroma
```

---

## Updating

### Standard Update Flow

```bash
cd /home/danvics/docker/quiz
git pull
docker compose build backend frontend
docker compose up -d
```

### Migration Safety

Database migrations run automatically on startup via `Base.metadata.create_all()` and explicit DDL in `setup_pgvector()`. These are idempotent:
- `CREATE TABLE IF NOT EXISTS`
- `ADD COLUMN IF NOT EXISTS`
- `CREATE INDEX IF NOT EXISTS`

DDL statements use `SET lock_timeout = '15s'` and retry up to 3 times to handle concurrent lock contention from Celery workers or other uvicorn processes.

### Zero-Downtime Updates

For zero-downtime updates (if needed):

```bash
# Build new images first
docker compose build backend frontend

# Restart one service at a time
docker compose up -d --no-deps backend
docker compose up -d --no-deps celery
docker compose up -d --no-deps frontend
```

---

## CLI Tools

The `manage.py` script provides management commands. Run inside the backend container:

```bash
docker compose exec backend python manage.py <command>
```

### Commands

#### `reset-password <email> <new-password>`

Reset a user's password. Password must be at least 8 characters.

```bash
docker compose exec backend python manage.py reset-password user@example.com newpassword123
```

#### `list-users`

List all users with ID, email, role, verified status, and name.

```bash
docker compose exec backend python manage.py list-users
```

Output:
```
  ID  Email                                     Role          Verified  Name
------------------------------------------------------------------------------------------
   1  admin@example.com                         admin         yes       Admin
   2  user@example.com                          user          yes       John
```

#### `reembed`

Regenerate all question embeddings. Useful after changing the embedding model. Clears existing embeddings and re-generates them one by one.

```bash
docker compose exec backend python manage.py reembed
```

#### `extract <section_id> [options]`

Run quiz extraction from a section. Can run inline (blocking) or in background (via Celery).

```bash
# Inline extraction (blocking, shows live output)
docker compose exec backend python manage.py extract 5

# With options
docker compose exec backend python manage.py extract 5 --title "Chapter 3 Quiz" --mode learning --user admin@example.com

# Background (via Celery)
docker compose exec backend python manage.py extract 5 --bg
```

Options:
- `--title` — Quiz title (default: auto-generated from section name)
- `--mode` — `timed` or `learning` (default: `timed`)
- `--user` — User email to assign the quiz to (default: first admin user)
- `--bg` — Run in background via Celery

#### `list-sections [doc_id]`

List all documents and their sections. Optionally filter by document ID.

```bash
# All documents
docker compose exec backend python manage.py list-sections

# Specific document
docker compose exec backend python manage.py list-sections 3
```

Output:
```
Doc 3: PREP_2024.pdf (ready, 450 pages)
  Section 5: 'Chapter 1'  pages 1-50
  Section 6: 'Chapter 2'  pages 51-120
```

#### `jobs [--user email]`

Show recent extraction jobs from Redis. Optionally filter by user email.

```bash
# All users' jobs
docker compose exec backend python manage.py jobs

# Specific user
docker compose exec backend python manage.py jobs --user admin@example.com
```

Output:
```
admin@example.com:
  [completed ] Quiz: Chapter 1                           steps= 12 quiz_id=7
              Saved 25 questions to quiz
  [running   ] Quiz: Chapter 2                           steps=  5
              Extracting questions from pages 51-80...
```