danvics/pediatric-ai-scribe-v3
1
0
Fork 0
Code Issues Pull requests Projects Releases 16 Packages Wiki Activity Actions
pediatric-ai-scribe-v3/docs/database.md
Daniel 025290d64a
Some checks failed
Forgejo Android APK / Root app tests (push) Successful in 47s
Details
Forgejo Docker Build / Root app tests (push) Successful in 45s
Details
Forgejo Android APK / Build signed APK (push) Successful in 2m1s
Details
Forgejo Docker Build / Build Docker image (push) Successful in 9s
Details
Forgejo Docker Build / Deploy to the host (push) Failing after 0s
Details
feat: retire Learning Hub 
My Resources generates better slides than Learning Hub ever did — a typed deck
the model fills in, rendered by python-pptx with fit-to-slide text, figures, a
vision review and themes, against Learning Hub's markdown-through-pandoc — and
the articles and quizzes now live in the quiz app. Keeping a second, weaker
generator and a whole CMS beside it was not earning its maintenance.

Removed: three routers, the Learning Hub and Content Manager tabs, their
components and frontend modules, the five database tables, the WebDAV browser,
the content embedding column and its vector index.

Content was exported first — every article as markdown plus a full SQL dump of
all five tables — to ops-backups/learning-hub-export-*. That export is the
restore path; the migration's down() can recreate the shape but never the rows,
and says so.

Two things this simplifies rather than merely deletes:

generated_image_links existed only to record which published content an image
appeared in, and it was the sole reason a generated image could be read by
someone who did not make it. Images are now owner-only — the visibility rule is
one WHERE clause instead of a join across two tables and a published flag.

embeddings.js keeps the model discovery the admin panel uses and loses
searchSimilar and generateContentEmbedding, which queried a table that no longer
exists.

Kept deliberately: Nextcloud connect, disconnect and export, which are how a
generated note reaches a real filesystem and have nothing to do with Learning
Hub; learningRetrieval, which despite its name is the clinical corpus search My
Resources depends on; and the pandoc reference deck, still the fallback when the
python renderer fails, moved from assets/learning to assets/deck now that the
old name misleads.

Tests: four Learning-Hub-only files removed, and the individual cases inside
shared files that asserted its behaviour. Where a test used a Learning endpoint
only as a convenient example — the account-boundary token test, the policy
matrix — it now uses one that still exists, so the property it proves is
unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
2026-09-12 20:14:20 +02:00

432 lines
16 KiB
Markdown
Raw Blame History

# Database schema
PostgreSQL 16 with `pgvector`. Image `pgvector/pgvector:pg16`, data in the
`pgdata` volume. Connection pool: 20 max, 30 s idle timeout, 5 s connect
timeout.
Schema is managed in two layers:
1. **Baseline init** — `src/db/database.js`. Idempotent
`CREATE TABLE IF NOT EXISTS` + `ALTER TABLE ADD COLUMN IF NOT EXISTS`.
Runs on every boot. Represents everything that predated the migration tool.
2. **Versioned migrations** — `migrations/` via `node-pg-migrate`. All new
schema changes go here. See `docs/migrations.md`.
## Extensions
```sql
CREATE EXTENSION IF NOT EXISTS vector;
```
## Tables
### `users`
Core accounts. Local-auth + OIDC federation + per-user preferences.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| email | TEXT UNIQUE NOT NULL | |
| password | TEXT NOT NULL | argon2id hash (primary) or bcrypt hash (legacy / rehashed on next login). For OIDC-auto-created users: random hex, not verifiable. |
| name | TEXT | |
| role | TEXT | `user` / `admin` / `moderator` |
| email_verified | BOOLEAN DEFAULT false | |
| verify_token, verify_expires | TEXT, BIGINT | Email verification |
| totp_secret, totp_enabled | TEXT, BOOLEAN DEFAULT false | 2FA |
| totp_backup_codes | TEXT | JSON array of bcrypt hashes of 10-character recovery codes. Consumed atomically on login. |
| oidc_sub | TEXT | IdP subject identifier (when linked) |
| disabled | BOOLEAN DEFAULT false | Soft disable |
| nextcloud_url, nextcloud_user, nextcloud_token, nextcloud_folder | TEXT | WebDAV credentials. `nextcloud_token` stored AES-256-GCM encrypted (prefix `enc1:`). |
| reset_token, reset_expires | TEXT, BIGINT | Password reset |
| stt_model, tts_voice | TEXT | Per-user STT/TTS override |
| webdav_learning_path | TEXT | Learning Hub file-browser root |
| created_at, updated_at | TIMESTAMPTZ DEFAULT NOW() | |
### `user_sessions`
Authoritative session registry.
| Column | Type | Notes |
|---|---|---|
| id | TEXT PK (UUID) | |
| user_id | INTEGER FK users.id | |
| token_hash | TEXT NOT NULL | SHA-256 of the JWT. Index `idx_sessions_token_hash` uses `COLLATE "C"` for ICU-drift immunity. |
| ip_address, user_agent | TEXT | |
| device_label | TEXT | Parsed from UA (`Chrome on Android`, `PedScribe (Android)`, etc.) |
| created_at, last_activity | TIMESTAMPTZ DEFAULT NOW() | `last_activity` only updated on POST/PUT/DELETE/PATCH, throttled to once per 10 min |
### `app_settings`
Key-value runtime config. 2-minute in-memory cache.
| Column | Type | Notes |
|---|---|---|
| key | TEXT PK | Also `COLLATE "C"` |
| value | TEXT | Plain or JSON |
| updated_at | TIMESTAMPTZ DEFAULT NOW() | |
| updated_by | INTEGER FK users.id | |
### `audit_log`
Human-level security and action audit. Writes are batched (1 s flush) by
`src/utils/auditQueue.js`.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER FK users.id | Null for unknown-user attempts |
| action | TEXT NOT NULL | e.g. `login`, `login_failed`, `session_idle_timeout`, `password_changed`, `generate_soap`, `2fa_backup_code_used` |
| category | TEXT DEFAULT 'general' | `auth`, `clinical`, `integration`, `export`, `documents`, `phi_access` |
| details | TEXT | Free-form, PHI-redacted via `src/utils/redact.js` |
| ip_address, user_agent | TEXT | |
| model_used, tokens_used, duration_ms | TEXT, INT, INT | LLM-call fields (optional) |
| status | TEXT DEFAULT 'success' | |
| timestamp | TIMESTAMPTZ DEFAULT NOW() | |
### `api_log`
Per-request AI-call telemetry.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER FK users.id | |
| endpoint | TEXT | Route path |
| method | TEXT | |
| status_code | INTEGER | |
| request_size, response_size | INTEGER | Bytes |
| model_used | TEXT | |
| tokens_input, tokens_output | INTEGER | |
| cost_estimate | NUMERIC | USD estimate (hardcoded rates; OpenRouter uses live pricing) |
| duration_ms | INTEGER | |
| ip_address, error | TEXT | |
| timestamp | TIMESTAMPTZ DEFAULT NOW() | |
### `access_log`
Auth-only event stream.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER FK users.id | |
| action | TEXT | `login`, `logout`, `failed_login`, … |
| ip_address, user_agent | TEXT | |
| success | BOOLEAN | |
| timestamp | TIMESTAMPTZ DEFAULT NOW() | |
### `saved_encounters`
Draft/complete encounter workspace. Auto-expires (default 7 d,
`site.auto_delete_days`).
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER FK users.id ON DELETE CASCADE | |
| label | TEXT NOT NULL DEFAULT 'Untitled' | Unique-per-user within active rows |
| enc_type | TEXT NOT NULL DEFAULT 'encounter' | `encounter`, `dictation`, `soap`, `sickvisit`, `wellvisit`, `hospital`, `chart`, `milestones` |
| transcript | TEXT | |
| generated_note | TEXT | |
| partial_data | TEXT | JSON of in-progress form state |
| status | TEXT DEFAULT 'active' | |
| version | INTEGER NOT NULL DEFAULT 1 | Optimistic lock. POST with `expected_version` mismatch returns 409. |
| idempotency_key | TEXT | Prevents duplicate creates from double-submit |
| created_at, updated_at | TIMESTAMPTZ DEFAULT NOW() | |
| expires_at | TIMESTAMPTZ | Default `NOW() + 7 days` |
### `user_memories`
Per-user template and preference rows. Only selected categories are injected
into AI generation through `/api/memories/context`; `custom` rows are stored
for the user but not included in prompt context.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER FK users.id ON DELETE CASCADE | |
| category | TEXT NOT NULL DEFAULT 'custom' | Valid categories: `physical_exam`, `ros`, `encounter_format`, `family_history`, `assessment_plan`, `custom`, `template_soap`, `template_hpi`, `template_wellvisit`, `template_sickvisit`, `template_ed`. Legacy `correction_*` rows may exist but are filtered out. |
| name | TEXT NOT NULL | Encrypted with `enc1:` for new rows |
| content | TEXT NOT NULL | Encrypted with `enc1:` for new rows |
| created_at, updated_at | TIMESTAMPTZ DEFAULT NOW() | |
### `audio_backups`
Optional 24-hour encrypted recovery store for recordings when transcription
fails, so users can retry without re-recording.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER FK users.id | |
| module | TEXT | `encounter`, `dictation`, etc. |
| mime_type | TEXT | |
| size_bytes, compressed_bytes | INTEGER | |
| audio_data | BYTEA | Gzip → AES-256-GCM (0x01 version prefix). Legacy rows (prefix `0x1F` = raw gzip) pass through. |
| created_at, expires_at | TIMESTAMPTZ | 24 h default |
### `user_documents`
Metadata for files in S3-compatible object storage. File bytes stay in S3.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER FK users.id | |
| s3_key | TEXT | Object storage key (prefixed with user id) |
| filename, mime_type | TEXT | |
| size_bytes | INTEGER | |
| description | TEXT | |
| created_at | TIMESTAMPTZ DEFAULT NOW() | |
### `developmental_milestones`
AAP-aligned pediatric milestone reference data. Age group + domain keyed.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| age_group | TEXT | `2 months`, `4 months`, `1 year`, … |
| domain | TEXT | `motor`, `language`, `social`, `cognitive` |
| milestone_text | TEXT | |
| sort_order | INTEGER | |
| created_at, updated_at | TIMESTAMPTZ DEFAULT NOW() | |
### `user_resources`
My Resources: a user's own teaching material. `deck` is the typed deck the
renderers read; `markdown` is the flattened form used for display and for the
markdown export. Export always renders from `deck` when it is present, so a
modification that edits only `markdown` will not change the exported file.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER NOT NULL | |
| title, topic | TEXT NOT NULL | |
| kind | TEXT NOT NULL | `deck` or `document` |
| markdown | TEXT NOT NULL | Flattened form |
| deck | JSONB | The typed deck; NULL for documents |
| image_ids | JSONB NOT NULL | Generated figures belonging to this resource |
| grounded_count | INTEGER NOT NULL | How many sources the generation was grounded in |
| created_at, updated_at | TIMESTAMPTZ NOT NULL | |
### `personal_notes`
Personal notes, with a trash lifecycle: `deleted_at` set means trashed, and
restore clears it. Emptying the trash is what actually deletes rows.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER NOT NULL | |
| title, body | TEXT NOT NULL | |
| created_at, updated_at | TIMESTAMPTZ NOT NULL | |
| deleted_at | TIMESTAMPTZ | NULL = live, set = in trash |
### `clinical_assistant_chats`
Saved Clinical Assistant conversations, one row per chat. `payload` holds the
serialized turns. Answers themselves are never cached for reuse — this is the
user's own history, not an answer cache.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER NOT NULL | |
| title | TEXT NOT NULL | |
| payload | TEXT NOT NULL | Serialized conversation |
| created_at, updated_at | TIMESTAMPTZ NOT NULL | |
### `clinical_prompt_pool_snapshots`
Point-in-time copies of the generated example-prompt pool, so a regeneration
that produces a worse pool can be rolled back from the Admin panel.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| payload | JSONB NOT NULL | The pool itself |
| generated_at | TIMESTAMPTZ | |
| target, count | INTEGER | Requested size and actual size |
| restored_from | INTEGER | The snapshot this one was restored from, if any |
| created_by | INTEGER | Admin user id |
| created_at | TIMESTAMPTZ | |
### `citation_audit`
Records how well an answer's citations matched its retrieved sources, so
citation quality can be watched over time rather than sampled by hand. Rows
expire.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER | |
| question | TEXT NOT NULL | |
| cited_count, source_count | INTEGER NOT NULL | |
| unverifiable | TEXT[] NOT NULL | Citations that matched no retrieved source |
| source_titles | TEXT[] NOT NULL | |
| created_at, expires_at | TIMESTAMPTZ NOT NULL | |
### `generated_image_jobs`, `generated_image_links`
Image generation jobs and their output. The prompt is stored encrypted
(`prompt_cipher`), and bytes are staged on the row until the job is claimed.
`lease_token` + `lease_until` are the worker lease, so a crashed worker's job
becomes claimable again instead of being stuck.
| Column | Type | Notes |
|---|---|---|
| id | UUID PK | |
| owner_id | INTEGER NOT NULL | Ownership is checked on every read |
| workflow | TEXT NOT NULL | Which feature asked, e.g. `clinical_assistant` |
| idempotency_key, input_hash | TEXT NOT NULL | Repeat submissions return the same job |
| prompt_cipher | TEXT NOT NULL | Encrypted prompt |
| model | TEXT NOT NULL | |
| prompt_revision, budget, prompt_units | INTEGER NOT NULL | |
| stage | TEXT NOT NULL | Job state |
| lease_token | UUID | Worker lease |
| lease_until | TIMESTAMPTZ | Lease expiry |
| staged_bytes | BYTEA | Image bytes |
| mime, checksum | TEXT | |
| byte_length | INTEGER | |
| error_code | TEXT | |
| context_included, context_total | INTEGER | How much context the prompt could carry |
| created_at, updated_at | TIMESTAMPTZ NOT NULL | |
`generated_image_links(asset_id, content_id)` ties an image to the Learning Hub
content that embeds it.
### `mermaid_diagrams`
Saved diagrams: the Mermaid source plus the user's own notes.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER NOT NULL | |
| title, source, notes | TEXT NOT NULL | |
| created_at, updated_at | TIMESTAMPTZ NOT NULL | |
### `login_codes`
Passwordless sign-in codes. Only the bcrypt hash is stored, so a code read out
of the database is not a working credential. Ten-minute TTL, single use
(`used_at`), five attempts (`attempts`). See
[`authentication.md`](authentication.md).
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER NOT NULL | |
| code_hash | TEXT NOT NULL | bcrypt |
| attempts | INTEGER NOT NULL | Refused at 5 |
| expires_at | TIMESTAMPTZ NOT NULL | 10 minutes |
| used_at | TIMESTAMPTZ | Set once; a used code never verifies again |
| created_at | TIMESTAMPTZ NOT NULL | |
### `registration_invites`
Invite codes for invite-only registration. Only the hash is stored; `code_hint`
is the fragment shown in the admin list so a code can be recognised without
being recoverable.
A code is *spent* when `used_at IS NOT NULL`, or when it has expired and was not
revoked. `DELETE /api/admin/invites/spent` deletes exactly that set — a live
code has to be revoked before it can be removed, so no code disappears while it
could still be redeemed.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| code_hash | TEXT NOT NULL | |
| code_hint | TEXT NOT NULL | Display fragment only |
| note | TEXT NOT NULL | Why it was issued |
| created_by, used_by, revoked_by | INTEGER | User ids |
| created_at, expires_at | TIMESTAMPTZ NOT NULL | |
| used_at, revoked_at | TIMESTAMPTZ | |
### `user_phone_extensions`
Phone extensions and pagers, with a trash lifecycle (`trashed_at`).
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| user_id | INTEGER | |
| location, name, number, type | TEXT | `type` distinguishes extension from pager |
| notes | TEXT | |
| trashed_at | TIMESTAMPTZ | NULL = live |
| created_at, updated_at | TIMESTAMPTZ | |
### `prompt_revisions`
History for admin prompt overrides, so a prompt edit can be reviewed and rolled
back. `was_default` records whether the value replaced the built-in.
| Column | Type | Notes |
|---|---|---|
| id | SERIAL PK | |
| prompt_key | TEXT NOT NULL | |
| value | TEXT NOT NULL | |
| was_default | BOOLEAN NOT NULL | |
| created_by | INTEGER | |
| restored_from | INTEGER | The revision this one was restored from |
| created_at | TIMESTAMPTZ NOT NULL | |
### `pgmigrations`
Created and managed by `node-pg-migrate`. Records applied migration filenames
+ run time. Never edit by hand.
## Indexes
Core btree indexes — see `database.js` for the full list.
- `users(email)` — **`COLLATE "C"`** (lookup-critical auth path)
- `user_sessions(token_hash)` — **`COLLATE "C"`**
- `audit_log(user_id)`, `audit_log(timestamp)`, `audit_log(action)`, `audit_log(category)`
- `api_log(user_id)`, `api_log(timestamp)`, `api_log(endpoint)`
- `access_log(user_id)`, `access_log(timestamp)`
- `saved_encounters(user_id)`, `saved_encounters(expires_at)`, `saved_encounters(idempotency_key)`
- `user_memories(user_id, category)`
- `audio_backups(user_id)`, `audio_backups(expires_at)`
- `user_documents(user_id)`
- `learning_content(category_id)`
- `learning_progress(user_id, content_id)`
- `developmental_milestones(age_group, domain)`
The `COLLATE "C"` indexes are immune to ICU library version changes between
Postgres image upgrades — silent index corruption from libc / ICU drift
cannot affect auth lookups.
## Collation drift handling
On startup, `src/db/database.js` compares `pg_database.datcollversion` with
`pg_database_collation_actual_version()`. On mismatch it runs
`REINDEX DATABASE` + `ALTER DATABASE … REFRESH COLLATION VERSION` and logs
the event. `npm run maint:reindex` runs the same operation manually.
## Auto-cleanup
Hourly job (plus 10 s after startup):
```sql
DELETE FROM saved_encounters WHERE expires_at < NOW();
DELETE FROM audio_backups WHERE expires_at < NOW();
DELETE FROM user_sessions WHERE last_activity < NOW() - INTERVAL '30 days';
```
(The session cleanup is optional safety — the idle middleware deletes stale
rows eagerly.)
## PHI at rest
| Column | Protection |
|---|---|
| `users.nextcloud_token` | AES-256-GCM via `src/utils/crypto.js`, prefix `enc1:` |
| `audio_backups.audio_data` | Gzip → AES-256-GCM, 0x01 version prefix |
| `audit_log.details` | Redacted (SSN, phone, email, DoB regex; 500-char cap; note-body heuristic truncation) |
| Error responses | Generic `'Request failed'` on 500s; full detail stays in `logger.error` / Loki |
Reference in a new issue View git blame Copy permalink