pediatric-ai-scribe-v3/docs/CLINICAL_ASSISTANT.md

Clinical Assistant

The Clinical Assistant is a retrieval-grounded assistant for pediatric clinical reference questions. It is not the same as the app's note-generation/HPI workflow.

Responsibilities

Component	Responsibility
Browser UI	question input, source display, markdown/citation rendering, export
Ped-AI backend	settings, MCP search call, answer prompt construction, model call
MCP server	Nextcloud access, indexing, vector search, rerank, source metadata
LiteLLM	model routing and provider abstraction

Request Flow

User asks a question
  -> browser posts to Ped-AI
  -> Ped-AI calls MCP `clinical_semantic_search`
  -> MCP returns source excerpts and metadata
  -> Ped-AI builds an answer prompt with source constraints
  -> LiteLLM model returns answer text
  -> browser renders answer and source cards

Source Rules

• Prefer MCP file_path basename for displayed source titles when present.
• Do not relabel one source as another requested source.
• If the user names a source and retrieval does not return it, say that before using other sources.
• Use citations only for returned source numbers.
• Unknown citation numbers should remain plain text instead of being guessed.

Table And Markdown Rendering

LLM output is not guaranteed to be valid markdown. The browser renderer defensively handles common problems:

• adjacent citation clusters,
• missing closing bracket in narrow citation cases,
• smashed bullet lists,
• inline headings,
• malformed pipe tables,
• bare source numbers in source/citation table columns,
• orphan markdown emphasis markers,
• code blocks that must not be modified.

Renderer fixes must be narrow. Do not add broad repairs that turn arbitrary clinical numbers into citations.

Image Routing

Table lookup requests should stay in retrieval flow.

Examples that should use retrieval:

show me the table
show me Table 13.1
summarize the developmental table

Explicit visual creation/display requests can use image flow.

Examples:

create an infographic
generate a diagram
show me the image/figure

Caching Policy

Clinical answer response caching is intentionally disabled. Redis can support prompt suggestions and operational metadata, but final answers should be generated from current retrieval context.

Image Attachments

Users can attach up to 4 images (PNG, JPEG, WebP) to an outgoing clinical question. Attachments ride the outgoing question only for inference and persist with the saved chat once the question is sent:

• They are validated client-side and authoritatively on the server (MIME allowlist, canonical base64, ≤ 5 MiB per image, ≤ 4 images, ≤ 10 MiB decoded total). Invalid input is rejected with 400 before any retrieval or provider call.
• They are sent only with the outgoing clinical question for inference. Attaching images never disables retrieval: RAG/includeContext runs exactly as without images.
• The conversation budget counts text only: images are excluded from the UTF-16 code-unit count. The server still validates every request.
• Once sent, the message's attachments are stored in the saved chat payload (same bounded limits, re-validated on every save) and restored as thumbnails on load.
• Only OpenAI-compatible providers (LiteLLM, OpenRouter, Azure) receive them as multimodal content parts (text + image_url data URIs) on the latest user message; the system/retrieval/history structure is unchanged. Legacy direct adapters (Bedrock/Vertex) refuse with a clear 400 before contacting the provider.
• Attachments clear on a successful send and on New chat; a rejected send keeps them for correction.

Autosave, titles and saved-chat updates

After each completed assistant turn (and on any change to the conversation), the chat is autosaved with an 800 ms debounce to POST /api/clinical-assistant/chats. New chats get a title derived from the first user message (first 60 characters); later saves include the chat id and update the same row in place. Failures surface once per change and never block chat flow; oversized saves keep the 8 MiB / 400 / 413 semantics and are retried only on the next change, never truncated. The raw transcript stays canonical. The generated sidebar image and per-message image jobs persist with the chat again.

Translation

Every message offers Translate with a target-language picker. Translation is the local LibreTranslate container (LIBRETRANSLATE_URL, default http://libretranslate:5000), which is the only provider there is. clinical_assistant.translate_provider is read but any unrecognised value silently falls back to LibreTranslate, and no DeepL client exists in the code at all. Responses are cached per provider+message+lang. Patient text therefore never leaves the local network.

Settings

Important settings include:

All are stored in settings and edited under Admin → Clinical Assistant / Learning, except the image roster, which is written by the Image Generation card. Every one is read through getSetting, so an unset key falls back to the default in the right-hand column.

Setting	Purpose
clinical_assistant.chat_model	Chat model for answers; falls back to models.default
clinical_assistant.image_model	Image model for explicit image generation; falls back to CLINICAL_ASSISTANT_IMAGE_MODEL, then openai-gpt-image-1
clinical_assistant.fallback_image_model	Single retry target when the image model fails
clinical_assistant.allowed_models	Comma-separated chat models a user may pick. Empty means no choice: the configured model is used. A non-empty list always includes the configured model; anything else is rejected with 400 model_not_allowed
clinical_assistant.allowed_image_models	The same, for image models
clinical_assistant.image_model_roster	Image models an admin added from Admin → Image Generation (+ Add). This is the pool the Image models tick-list offers; it is not itself an allowlist. Validated as up to 100 ids
clinical_assistant.search_limit	Number of MCP results requested
clinical_assistant.context_chars	Context characters requested from MCP
clinical_assistant.conversation_chars	Input budget in UTF-16 code units. Empty means use CLINICAL_ASSISTANT_CONVERSATION_CHARS; a value must be 1000-1000000
clinical_assistant.show_sources	true/false. Display only: hides the Sources panel and the citation markers. The prompt, the retrieval and the stored answer are byte-for-byte identical either way, so it cannot bias an answer; turning it back on restores the citations
clinical_assistant.preview_enabled	true/false. Lets signed-out visitors try the assistant read-only; anything needing an account asks them to sign in
clinical_assistant.system_behavior	Admin-editable assistant behavior guidance
clinical_assistant.image_behavior	Guidance for the generate_image tool
clinical_assistant.patient_takehome_behavior	Guidance for patient take-home text
clinical_assistant.prompt_model	Model that generates the starter prompt pool
clinical_assistant.translate_provider	Translation provider. libretranslate is the only value the server accepts
clinical_assistant.citations_enabled	Legacy key, read only as a fallback for show_sources

Choosing a model

The composer shows a Model button rather than the model id, which can be as long as openrouter-gemini-3.1-flash-image-preview; clicking it opens the list. The button is a face for #assistant-chat-model-select, which stays in the DOM as the state holder — so a choice made in the popup is saved by the same delegated change listener as before, under an account-scoped storage key. The whole control is hidden unless the allowlist offers more than one model.

For an image model to reach a user, an admin does two things: + Add it under Admin → Image Generation (which puts it in image_model_roster), then tick it in the Clinical Assistant's Image models list (which puts it in allowed_image_models). Discovery lists what the gateway advertises with mode image_generation; it never adds anything on its own.

Testing Priorities

Add or update tests when changing:

• citation rendering,
• source title cleanup,
• named-source provenance behavior,
• table rendering and table copy/CSV actions,
• image intent routing,
• image attachment validation, multimodal payload shape and saved-chat roundtrips,
• autosave debounce, title derivation and saved-chat updates,
• translation validation, caching and provider fallback,
• MCP result normalization,
• model discovery or settings behavior.