pediatric-ai-scribe-v3/docs/CLINICAL_ASSISTANT.md

# Clinical Assistant

The Clinical Assistant is a retrieval-grounded assistant for pediatric clinical reference questions. It is not the same as the app's note-generation/HPI workflow.

## Responsibilities

| Component | Responsibility |
|---|---|
| Browser UI | question input, source display, markdown/citation rendering, export |
| Ped-AI backend | settings, MCP search call, answer prompt construction, model call |
| MCP server | Nextcloud access, indexing, vector search, rerank, source metadata |
| LiteLLM | model routing and provider abstraction |

## Request Flow

```txt
User asks a question
  -> browser posts to Ped-AI
  -> Ped-AI calls MCP `nc_semantic_search`
  -> MCP returns source excerpts and metadata
  -> Ped-AI builds an answer prompt with source constraints
  -> LiteLLM model returns answer text
  -> browser renders answer and source cards
```

## Source Rules

- Prefer MCP `file_path` basename for displayed source titles when present.
- Do not relabel one source as another requested source.
- If the user names a source and retrieval does not return it, say that before using other sources.
- Use citations only for returned source numbers.
- Unknown citation numbers should remain plain text instead of being guessed.

## Table And Markdown Rendering

LLM output is not guaranteed to be valid markdown. The browser renderer defensively handles common problems:

- adjacent citation clusters,
- missing closing bracket in narrow citation cases,
- smashed bullet lists,
- inline headings,
- malformed pipe tables,
- bare source numbers in source/citation table columns,
- orphan markdown emphasis markers,
- code blocks that must not be modified.

Renderer fixes must be narrow. Do not add broad repairs that turn arbitrary clinical numbers into citations.

## Image Routing

Table lookup requests should stay in retrieval flow.

Examples that should use retrieval:

```txt
show me the table
show me Table 13.1
summarize the developmental table
```

Explicit visual creation/display requests can use image flow.

Examples:

```txt
create an infographic
generate a diagram
show me the image/figure
```

## Caching Policy

Clinical answer response caching is intentionally disabled. Redis can support prompt suggestions and operational metadata, but final answers should be generated from current retrieval context.

## Image Attachments

Users can attach up to 4 images (PNG, JPEG, WebP) to an outgoing clinical question. Attachments **ride the outgoing question only for inference** and **persist with the saved chat** once the question is sent:

- They are validated client-side and authoritatively on the server (MIME allowlist, canonical base64, ≤ 5 MiB per image, ≤ 4 images, ≤ 10 MiB decoded total). Invalid input is rejected with 400 before any retrieval or provider call.
- They are sent **only** with the outgoing clinical question for inference. Attaching images never disables retrieval: RAG/includeContext runs exactly as without images.
- The conversation budget counts text only: images are excluded from the UTF-16 code-unit count. The server still validates every request.
- Once sent, the message's attachments are stored in the saved chat payload (same bounded limits, re-validated on every save) and restored as thumbnails on load.
- Only OpenAI-compatible providers (LiteLLM, OpenRouter, Azure) receive them as multimodal content parts (`text` + `image_url` data URIs) on the latest user message; the system/retrieval/history structure is unchanged. Legacy direct adapters (Bedrock/Vertex) refuse with a clear 400 before contacting the provider.
- Attachments clear on a successful send and on New chat; a rejected send keeps them for correction.

## Autosave, titles and saved-chat updates

After each completed assistant turn (and on any change to the conversation), the chat is autosaved with an 800 ms debounce to `POST /api/clinical-assistant/chats`. New chats get a title derived from the first user message (first 60 characters); later saves include the chat `id` and update the same row in place. Failures surface once per change and never block chat flow; oversized saves keep the 8 MiB / 400 / 413 semantics and are retried only on the next change, never truncated. The raw transcript stays canonical. The generated sidebar image and per-message image jobs persist with the chat again.

## Translation

Every message offers Translate with a target-language picker and a provider choice. The default provider is the local LibreTranslate container (`LIBRETRANSLATE_URL`, default `http://libretranslate:5000`); DeepL is an admin-configured alternative (`DEEPL_API_KEY` + `DEEPL_API_BASE`, default `https://api.deepl.com/v2`, `api-free.deepl.com` also allowed). The admin default lives in `clinical_assistant.translate_provider` (libretranslate|deepl). Responses are cached per provider+message+lang; transient provider failures fall back to the other configured provider once; validation failures never fall back. No patient data leaves the local network unless the admin explicitly configures DeepL.

## Settings

Important settings include:

All are stored in `settings` and edited under Admin → Clinical Assistant /
Learning, except the image roster, which is written by the Image Generation
card. Every one is read through `getSetting`, so an unset key falls back to the
default in the right-hand column.

| Setting | Purpose |
|---|---|
| `clinical_assistant.chat_model` | Chat model for answers; falls back to `models.default` |
| `clinical_assistant.image_model` | Image model for explicit image generation; falls back to `CLINICAL_ASSISTANT_IMAGE_MODEL`, then `openai-gpt-image-1` |
| `clinical_assistant.fallback_image_model` | Single retry target when the image model fails |
| `clinical_assistant.allowed_models` | Comma-separated chat models a user may pick. Empty means no choice: the configured model is used. A non-empty list always includes the configured model; anything else is rejected with 400 `model_not_allowed` |
| `clinical_assistant.allowed_image_models` | The same, for image models |
| `clinical_assistant.image_model_roster` | Image models an admin added from Admin → Image Generation (**+ Add**). This is the pool the Image models tick-list offers; it is not itself an allowlist. Validated as up to 100 ids |
| `clinical_assistant.search_limit` | Number of MCP results requested |
| `clinical_assistant.context_chars` | Context characters requested from MCP |
| `clinical_assistant.conversation_chars` | Input budget in UTF-16 code units. Empty means use `CLINICAL_ASSISTANT_CONVERSATION_CHARS`; a value must be 1000-1000000 |
| `clinical_assistant.show_sources` | `true`/`false`. Display only: hides the Sources panel and the citation markers. The prompt, the retrieval and the stored answer are byte-for-byte identical either way, so it cannot bias an answer; turning it back on restores the citations |
| `clinical_assistant.preview_enabled` | `true`/`false`. Lets signed-out visitors try the assistant read-only; anything needing an account asks them to sign in |
| `clinical_assistant.system_behavior` | Admin-editable assistant behavior guidance |
| `clinical_assistant.image_behavior` | Guidance for the `generate_image` tool |
| `clinical_assistant.patient_takehome_behavior` | Guidance for patient take-home text |
| `clinical_assistant.prompt_model` | Model that generates the starter prompt pool |
| `clinical_assistant.translate_provider` | Translation provider. `libretranslate` is the only value the server accepts |
| `clinical_assistant.citations_enabled` | Legacy key, read only as a fallback for `show_sources` |

## Choosing a model

The composer shows a **Model** button rather than the model id, which can be as
long as `openrouter-gemini-3.1-flash-image-preview`; clicking it opens the list.
The button is a face for `#assistant-chat-model-select`, which stays in the DOM
as the state holder — so a choice made in the popup is saved by the same
delegated `change` listener as before, under an account-scoped storage key. The
whole control is hidden unless the allowlist offers more than one model.

For an image model to reach a user, an admin does two things: **+ Add** it under
Admin → Image Generation (which puts it in `image_model_roster`), then tick it
in the Clinical Assistant's Image models list (which puts it in
`allowed_image_models`). Discovery lists what the gateway advertises with mode
`image_generation`; it never adds anything on its own.

## Testing Priorities

Add or update tests when changing:

- citation rendering,
- source title cleanup,
- named-source provenance behavior,
- table rendering and table copy/CSV actions,
- image intent routing,
- image attachment validation, multimodal payload shape and saved-chat roundtrips,
- autosave debounce, title derivation and saved-chat updates,
- translation validation, caching and provider fallback,
- MCP result normalization,
- model discovery or settings behavior.