Document Alembic workflow and advisory-lock startup fix

README.md

@@ -356,7 +356,18 @@ For deep architecture documentation (database schema, request flow, background t
 
 ### Multi-Worker Setup
 
-The backend runs 4 uvicorn workers. A Redis SETNX lock (`startup:singleton_lock`, TTL 300s) ensures only one worker starts the APScheduler and backfill thread. Stale `idle in transaction` DB connections are killed at startup to prevent DDL migration hangs.
+The backend runs 4 uvicorn workers. Two coordination mechanisms prevent worker races at startup:
+
+- **Postgres advisory lock** (`pg_advisory_lock(8472931)`) serializes startup DDL (`Base.metadata.create_all()` + `setup_pgvector()`). Only one worker runs the schema operations; the others wait, then see the idempotent `IF NOT EXISTS` statements as no-ops. This eliminates the deadlock that used to kill one worker on boot when concurrent `ALTER TABLE`s acquired locks in different orders.
+- **Redis SETNX lock** (`startup:singleton_lock`, TTL 300s) ensures only one worker starts the APScheduler and backfill thread.
+
+Stale `idle in transaction` DB connections are killed at startup to prevent DDL migration hangs.
+
+### Database Migrations (Alembic)
+
+Schema changes go through Alembic (`backend/alembic/`). See [docs/migrations.md](docs/migrations.md) for the developer workflow (create → review → apply → rollback).
+
+Alembic reads `DATABASE_URL` from the container env; there is no hardcoded URL in `alembic.ini`. The live DB tracks its version in `alembic_version`. `Base.metadata.create_all()` remains as a fallback for fresh deploys and must not be removed without first generating a baseline migration.
 
 ### Landing Page
 

docs/migrations.md

@@ -0,0 +1,76 @@
+# Database Migrations (Alembic)
+
+## What's a migration?
+
+A migration is a versioned, ordered change to the database schema — adding a column, renaming a table, changing a type. Each change is a Python file in `backend/alembic/versions/`. Alembic tracks which ones have been applied in an `alembic_version` table in Postgres, so it knows what's new next time.
+
+Before Alembic was wired up in this project, schema was bootstrapped via `Base.metadata.create_all()` in `backend/app/main.py` — which only creates *missing tables*, never alters existing ones. Every column change required a manual `ALTER TABLE`. Alembic fixes that: schema changes become versioned, reversible, and reproducible across environments.
+
+## Current setup
+
+- `backend/alembic.ini` contains no hardcoded URL; `backend/alembic/env.py` injects `DATABASE_URL` from the container's environment.
+- The live production DB is stamped at the latest revision in `backend/alembic/versions/`.
+- `Base.metadata.create_all()` is still called at startup as a safety net for fresh deploys. Do **not** remove it without first generating a complete baseline migration from the current live schema.
+
+## Developer workflow
+
+```bash
+# where am I?
+docker compose exec backend alembic current
+docker compose exec backend alembic heads
+
+# create a new migration (auto-diffs your model against the live DB)
+docker compose exec backend alembic revision --autogenerate -m "add some column"
+# review the generated file under backend/alembic/versions/ BEFORE applying
+
+# apply pending migrations
+docker compose exec backend alembic upgrade head
+
+# roll back the last one
+docker compose exec backend alembic downgrade -1
+
+# mark the DB as being at a revision without running anything (use with care)
+docker compose exec backend alembic stamp <revision>
+```
+
+After adding a migration in dev, rebuild the backend image so the new file is in the container:
+
+```bash
+docker compose build backend celery
+docker compose up -d backend celery --force-recreate
+```
+
+## When to write a migration
+
+Any schema change:
+- New column, dropped column, renamed field
+- New table, dropped table
+- Altered index
+- New foreign key
+- Changed nullability or default value
+
+Always: edit model → generate migration → review → apply → commit both files together.
+
+## Gotchas
+
+- Migrations run inside a transaction. A failed migration rolls back cleanly, so the DB stays consistent.
+- `--autogenerate` doesn't catch everything. Things you must hand-edit into the file:
+  - `server_default` changes
+  - CHECK constraints
+  - Enum value additions
+  - Data migrations (moving rows around as part of a schema change)
+- The `alembic_version` table should only ever have one row. Multiple rows = branched heads; run `alembic merge` to reconcile.
+- Don't edit a migration after it's been applied to any shared environment — write a new one instead.
+
+## Rollback patterns
+
+```bash
+# Undo the last migration
+docker compose exec backend alembic downgrade -1
+
+# Jump to a specific revision (by hash prefix)
+docker compose exec backend alembic downgrade 9bac7bf02e38
+
+# Jump all the way back (rarely what you want)
+docker compose exec backend alembic downgrade base
+```