pediatric-ai-scribe-v3/docs/clinical-assistant.md

Clinical Assistant

The Clinical Assistant is a retrieval-grounded assistant for pediatric clinical reference questions. It is not the same as the app's note-generation/HPI workflow.

Responsibilities

Component	Responsibility
Browser UI	question input, source display, markdown/citation rendering, export
Ped-AI backend	settings, MCP search call, answer prompt construction, model call
MCP server	Nextcloud access, indexing, vector search, rerank, source metadata
LiteLLM	model routing and provider abstraction

Request Flow

User asks a question
  -> browser posts to Ped-AI
  -> Ped-AI calls MCP `clinical_semantic_search`
  -> MCP returns source excerpts and metadata
  -> Ped-AI builds an answer prompt with source constraints
  -> LiteLLM model returns answer text
  -> browser renders answer and source cards

Source Rules

• Prefer MCP file_path basename for displayed source titles when present.
• Do not relabel one source as another requested source.
• If the user names a source and retrieval does not return it, say that before using other sources.
• Use citations only for returned source numbers.
• Unknown citation numbers should remain plain text instead of being guessed.

Table And Markdown Rendering

LLM output is not guaranteed to be valid markdown. The browser renderer defensively handles common problems:

• adjacent citation clusters,
• missing closing bracket in narrow citation cases,
• smashed bullet lists,
• inline headings,
• malformed pipe tables,
• bare source numbers in source/citation table columns,
• orphan markdown emphasis markers,
• code blocks that must not be modified.

Renderer fixes must be narrow. Do not add broad repairs that turn arbitrary clinical numbers into citations.

Image Routing

Table lookup requests should stay in retrieval flow.

Examples that should use retrieval:

show me the table
show me Table 13.1
summarize the developmental table

Explicit visual creation/display requests can use image flow.

Examples:

create an infographic
generate a diagram
show me the image/figure

Caching Policy

Clinical answer response caching is intentionally disabled. Redis can support prompt suggestions and operational metadata, but final answers should be generated from current retrieval context.

Image Attachments

Users can attach up to 4 images (PNG, JPEG, WebP) to an outgoing clinical question. Attachments ride the outgoing question only for inference and persist with the saved chat once the question is sent:

• They are validated client-side and authoritatively on the server (MIME allowlist, canonical base64, ≤ 5 MiB per image, ≤ 4 images, ≤ 10 MiB decoded total). Invalid input is rejected with 400 before any retrieval or provider call.
• They are sent only with the outgoing clinical question for inference. Attaching images never disables retrieval: RAG/includeContext runs exactly as without images.
• The conversation budget counts text only: images are excluded from the UTF-16 code-unit count. The server still validates every request.
• Once sent, the message's attachments are stored in the saved chat payload (same bounded limits, re-validated on every save) and restored as thumbnails on load.
• Only OpenAI-compatible providers (LiteLLM, OpenRouter, Azure) receive them as multimodal content parts (text + image_url data URIs) on the latest user message; the system/retrieval/history structure is unchanged. The direct Bedrock adapter refuses with a clear 400 before contacting the provider.
• Attachments clear on a successful send and on New chat; a rejected send keeps them for correction.

Autosave, titles and saved-chat updates

After each completed assistant turn (and on any change to the conversation), the chat is autosaved with an 800 ms debounce to POST /api/clinical-assistant/chats. New chats get a title derived from the first user message (first 60 characters); later saves include the chat id and update the same row in place. Failures surface once per change and never block chat flow; oversized saves keep the 8 MiB / 400 / 413 semantics and are retried only on the next change, never truncated. The raw transcript stays canonical. The generated sidebar image and per-message image jobs persist with the chat again.

Translation

Every message offers Translate with a target-language picker. Translation is the local LibreTranslate container (LIBRETRANSLATE_URL, default http://libretranslate:5000), which is the only provider there is. clinical_assistant.translate_provider is read but any unrecognised value silently falls back to LibreTranslate, and no DeepL client exists in the code at all. Responses are cached per provider+message+lang. Patient text therefore never leaves the local network.

Settings

Important settings include:

All are stored in settings and edited under Admin → Clinical Assistant / Learning, except the image roster, which is written by the Image Generation card. Every one is read through getSetting, so an unset key falls back to the default in the right-hand column.

Setting	Purpose
clinical_assistant.chat_model	Chat model for answers; falls back to models.default
clinical_assistant.image_model	Image model for explicit image generation; falls back to CLINICAL_ASSISTANT_IMAGE_MODEL, then openai-gpt-image-1
clinical_assistant.fallback_image_model	Single retry target when the image model fails
clinical_assistant.allowed_models	Comma-separated chat models a user may pick. Empty means no choice: the configured model is used. A non-empty list always includes the configured model; anything else is rejected with 400 model_not_allowed
clinical_assistant.allowed_image_models	The same, for image models
clinical_assistant.image_model_roster	Image models an admin added from Admin → Image Generation (+ Add). This is the pool the Image models tick-list offers; it is not itself an allowlist. Validated as up to 100 ids
clinical_assistant.search_limit	Number of MCP results requested
clinical_assistant.context_chars	Context characters requested from MCP
clinical_assistant.conversation_chars	Input budget in UTF-16 code units. Empty means use CLINICAL_ASSISTANT_CONVERSATION_CHARS; a value must be 1000-1000000
clinical_assistant.show_sources	true/false. Display only: hides the Sources panel and the citation markers. The prompt, the retrieval and the stored answer are byte-for-byte identical either way, so it cannot bias an answer; turning it back on restores the citations
clinical_assistant.preview_enabled	true/false. Lets signed-out visitors try the assistant read-only; anything needing an account asks them to sign in
clinical_assistant.system_behavior	Admin-editable assistant behavior guidance
clinical_assistant.image_behavior	Guidance for the generate_image tool
clinical_assistant.patient_takehome_behavior	Guidance for patient take-home text
clinical_assistant.prompt_model	Model that generates the starter prompt pool
clinical_assistant.translate_provider	Translation provider. libretranslate is the only value the server accepts
clinical_assistant.citations_enabled	Legacy key, read only as a fallback for show_sources

search_limit and context_chars are capped by RERANKER_TOP_K in the MCP deployment, which is the real ceiling on every search. See retrieval-tuning.md.

Lending eyes to a text-only model

clinical_assistant.vision_model, when set, is the model shown an attachment that the chat model cannot be shown.

The chat model is offered a look_at_image tool alongside the image tool and decides when to use it, exactly as it decides about drawing. The attachment is withheld from its own request — sending an image to a model that cannot accept one is either refused by the provider or silently dropped, and an answer about a picture nobody looked at is worse than a refusal.

Delegation only happens when the gateway reports supports_vision: false for the chat model. An unknown is left alone: most of a roster carries no supports_vision at all, and treating silence as blindness would route perfectly good models through a detour they do not need. The capability is read from /model/info and cached for five minutes, never inferred from the model id.

The seeing model is told to report and not to diagnose: it has a picture and no conversation, no retrieved sources and no system prompt, so an opinion from it would carry weight it has not earned. Its description returns as a tool result and the chat model answers in its own voice, from words.

Saving the setting is refused if the gateway reports that model as text-only — the same check that guards the slide reviewer.

Environment variables

Settings above are the normal way to configure the assistant. These environment variables sit underneath them — connection details, timeouts, and the defaults a setting falls back to.

Variable	Default	Purpose
CLINICAL_ASSISTANT_MCP_URL	—	MCP endpoint. MCP_SERVER_URL is accepted as an older name.
CLINICAL_ASSISTANT_MCP_URLS	—	Comma-separated list, tried in order, ahead of the single-URL variable.
CLINICAL_ASSISTANT_SEARCH_TOOL	clinical_semantic_search	Tool name to call on the MCP server. Only this value is accepted; the nc_semantic_search alias was removed, and anything else throws at startup rather than failing per request.
CLINICAL_ASSISTANT_MCP_INITIALIZE_TIMEOUT_MS	30000	Session handshake timeout.
CLINICAL_ASSISTANT_MCP_REQUEST_TIMEOUT_MS	90000	Per-search timeout.
CLINICAL_ASSISTANT_MCP_SESSION_TTL_MS	600000	How long an MCP session is reused.
CLINICAL_ASSISTANT_MCP_WARMUP	on	Set to false to skip opening an MCP session at boot. Tests set this.
CLINICAL_ASSISTANT_MCP_WARMUP_DELAY_MS	5000	Delay before that warmup.
CLINICAL_ASSISTANT_CONVERSATION_CHARS	120000	Input budget in UTF-16 code units, when the setting is empty.
CLINICAL_ASSISTANT_IMAGE_MODEL	openai-gpt-image-1	Image model, when the setting is empty.
CLINICAL_ASSISTANT_PROMPT_MODEL	—	Model for the starter prompt pool, when the setting is empty.
CLINICAL_ASSISTANT_PROMPT_POOL_TARGET	1000	How many example prompts to generate.
CLINICAL_ASSISTANT_PROMPT_POOL_REFRESH_MS	7 days	How often the pool regenerates. 0 disables refresh.
CLINICAL_ASSISTANT_PROMPT_POOL_KEY	clinical-assistant:prompt-pool:v2	Redis key holding the pool.
CLINICAL_ASSISTANT_PROMPT_POOL_WARMUP_DELAY_MS	15000	Delay before the pool warms at boot.
CLINICAL_ASSISTANT_EXAMPLE_CACHE_MS	600000	How long the examples endpoint caches its answer.

Choosing a model

The composer shows a Model button rather than the model id, which can be as long as openrouter-gemini-3.1-flash-image-preview; clicking it opens the list. The button is a face for #assistant-chat-model-select, which stays in the DOM as the state holder — so a choice made in the popup is saved by the same delegated change listener as before, under an account-scoped storage key. The whole control is hidden unless the allowlist offers more than one model.

For an image model to reach a user, an admin does two things: + Add it under Admin → Image Generation (which puts it in image_model_roster), then tick it in the Clinical Assistant's Image models list (which puts it in allowed_image_models). Discovery lists what the gateway advertises with mode image_generation; it never adds anything on its own.

Testing Priorities

Add or update tests when changing:

• citation rendering,
• source title cleanup,
• named-source provenance behavior,
• table rendering and table copy/CSV actions,
• image intent routing,
• image attachment validation, multimodal payload shape and saved-chat roundtrips,
• autosave debounce, title derivation and saved-chat updates,
• translation validation, caching and provider fallback,
• MCP result normalization,
• model discovery or settings behavior.