pediatric-ai-scribe-v3/docs/scaling.md

Scaling

This document describes how Ped-AI should scale without becoming harder to debug or maintain.

Current Scaling Model

Ped-AI is currently a single app container backed by PostgreSQL and Redis. That is acceptable for self-hosted use, but the code should keep moving toward a shape where multiple app containers can run safely.

reverse proxy
  -> pediatric-ai-scribe replica 1
  -> pediatric-ai-scribe replica 2
  -> shared PostgreSQL
  -> shared Redis
  -> LiteLLM
  -> MCP

Horizontal Scaling Requirements

Requirement	Why It Matters
Session state in PostgreSQL/Redis	Any app replica can handle the next request
No clinical state only in memory	Restarting or scaling containers should not lose required state
Shared uploads/storage if files grow	Local container disk does not scale across replicas
Idempotent migrations	Deploying more than one app container should not corrupt schema state
Request timeouts	Slow providers should not exhaust Node workers
Queue for slow jobs	Long work should not block interactive requests
Readiness endpoint	Load balancer should only send traffic to ready replicas

What Can Stay In Memory

Small process-local caches are acceptable when they are optional and short-lived.

Examples:

• settings cache with short TTL,
• provider model metadata cache,
• static configuration derived at boot.

Do not store required user workflow state only in memory if the action must survive restart or run across replicas.

Redis Use

Redis is appropriate for:

• prompt suggestion pools,
• rate-limit coordination if needed,
• queues and job status,
• short-lived provider metadata,
• operational locks.

Redis should not be used for final clinical answer response caching. Clinical answers should be generated live from current retrieval context.

Queue Candidates

Consider moving these to a queue when latency or concurrency becomes a problem:

• long transcription jobs,
• file import/export,
• Learning Hub AI generation from large files,
• image generation,
• bulk document operations,
• provider metadata refresh,
• long-running admin maintenance actions.

BullMQ with Redis is a natural fit if a queue is added.

Readiness And Health

Keep /api/health fast and simple for liveness.

Add a separate readiness endpoint when scaling:

GET /api/ready

It should check:

• PostgreSQL query works,
• Redis ping works if Redis is required for this deployment,
• core settings can be read,
• MCP health is reachable if Clinical Assistant is enabled,
• LiteLLM metadata or configured model endpoint is reachable if AI features are enabled.

Database Scaling

Priorities:

• confirm indexes on hot user/session/settings/log tables,
• keep migrations explicit and reversible where practical,
• monitor slow queries,
• cap admin log queries with safe limits,
• keep audit/log writes batched where possible,
• avoid long transactions around provider calls.

Provider Scaling

LiteLLM and MCP can become the bottlenecks before Ped-AI does.

Track:

• LiteLLM request latency,
• LiteLLM error rate by model,
• MCP search latency,
• MCP timeout/error rate,
• queue depth if async jobs are added,
• Postgres connections,
• app container memory and event-loop delay.

Scaling Order

1. Add request IDs across browser, Ped-AI, MCP, and LiteLLM calls.
2. Add /api/ready for dependency readiness.
3. Ensure sessions and settings are not process-local.
4. Add a queue for slow jobs if interactive requests block.
5. Run a second app replica behind the reverse proxy in a staging/test environment.
6. Add metrics and alerts around latency, errors, and resource saturation.