Commit graph

53 commits

Author SHA1 Message Date
Daniel
8a6a4df121 refactor: citations are a markdown-it token, and the numbers you see are display order
The old renderer rewrote the text: it found "[n]" with regexes, renumbered
them, and swapped the result back in — which broke inside `arr[2][1]`, inside
HTML attributes, and whenever two turns disagreed about what "[3]" meant. It
also had a fallback markdown renderer of its own for when the rewrite
produced something markdown-it would not parse.

Now "[n]" is an inline rule registered on the same markdown-it instance that
renders everything else. The parser decides what is prose and what is code, a
link, or a URL, so the rule never sees "[1]" inside a code span, and it steps
aside for "[1](url)". Math is two more rules on the same parser instead of a
regex pre-pass, so "$" inside a URL is no longer math.

Identity vs display: the stored "[n]" and each card's id are the source's
identity (sourceNumber) and are never rewritten. The number a reader sees is
the order of first appearance, computed at render time from the token stream
(orderSourcesByCitation), so "one, then seven" cannot happen and a saved chat
re-opens pointing at the same cards it was saved with. Stored messages and
sources are untouched; export and the modal resolve by identity.

Translated HTML gets the same links through a TreeWalker over text nodes
(linkCitationsInHtml) rather than a regex over markup.

Deleted: renderCitationLinks, normalizeAdjacentCitationClusters, the
fallback renderer (fallbackMarkdown/renderMixedList/renderFallbackTable),
renderLatexText, CITATION_SCAN. Tests that asserted rewritten text now assert
token output; harnesses that render for real are given a parser.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
2026-09-13 04:41:40 +02:00
Daniel
e376f69502 fix: each TTS model offers only the voices it will accept
All checks were successful
Forgejo Docker Build / Root app tests (push) Successful in 47s
Forgejo Docker Build / Build Docker image (push) Successful in 21s
Forgejo Docker Build / End-to-end (browser) (push) Successful in 6s
"These settings don't work" — picking a model and testing with a voice
returned 500 every time. The gateway said why, once asked directly:

    voice must be one of the following voices: [autumn diana hannah austin daniel troy]

The screen was listing twelve Orpheus voices and six Kokoro ones in a
single flat list with no indication of which model would accept which,
because LITELLM_TTS_VOICES — written for one model — was treated as a
list that applied to all of them, and the Orpheus lists were pushed in
beside it. Choosing Orpheus and testing it with a Kokoro voice is not a
configuration; it is an error, and it was the default outcome.

LiteLLM cannot supply this. /model/info reports mode audio_speech for
all four models and carries no voice field for any of them. So the
mapping lives here, keyed by family so the gateway alias and the
upstream id resolve to one list, and every list was taken from the
provider rather than from documentation:

  Groq Orpheus English   autumn diana hannah austin daniel troy   (stated by Groq)
  Groq Orpheus Arabic    abdullah fahad sultan lulwa noura aisha  (stated by Groq)
  Fish s2.1-pro          alloy                                    (alloy returns audio; the rest 400)
  Kokoro                 sherpa/kokoro:* from LITELLM_TTS_VOICES  (the gateway's own list)

The environment still wins for the model it was written for, so the
local gateway's voices can change without a code change — but it
answers for that model only. A model with no list at all is offered
nothing rather than another model's voices, and a voice known to belong
to a different family is refused.

There were two copies of this knowledge before: getLiteLLMTTSVoicesForModel
branched by family and fell through to the env list for any model it did
not recognise — which is how Fish came to be offered six Kokoro voices.
One table now.

Also in this commit: citation renumbering skips fenced code, inline code
and math, so arr[2][1] in a code block is never rewritten. Renumbering at
render time was tried and reverted — it also has to skip HTML attributes,
and every such region is another regex branch. It stays at the answer
boundary, and the saved-chat boundary is next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
2026-09-13 03:54:31 +02:00
Daniel
0e1e74882e feat: sources are numbered in the order the answer cites them
Borrowed from the quiz app's AI Mode, where validating citations and
ordering them fall out of the same pass: it collects the sources an
answer actually used into an insertion-ordered map, so the list comes
back in first-citation order for free.

Ours listed sources in retrieval order — an order the reader never sees
and has no way to follow. An answer whose first citation was [7] opened
a list that began at [1], so matching a marker to a source meant
hunting. Reference lists in published writing are numbered by first
appearance for exactly this reason.

Cited sources now come first, renumbered by first appearance, and the
markers in the text are rewritten to match. Anything retrieved and not
cited keeps its place after them, labelled "not cited" — the panel is
also a view of what the search returned, which is worth keeping, but it
should not sit among the numbers the answer used.

The marker itself now shows its number instead of the word "src". Every
citation read identically, so the only way to tell one from another was
to hover it — which made the numbered list beneath useless to match
against. The export has shown numbers since the day "src" was
introduced, with no recorded reason for the difference.

Renumbering happens once the whole answer is known, never while
streaming: the order is the order of first citation, so a citation that
has not arrived yet cannot take its place, and numbers would shuffle
under the reader mid-sentence. The text is rewritten in a single pass —
number by number would turn 2 into 1 and then that 1 into whatever 1
maps to.

An invented citation reserves no position and is left exactly as it was.
It is still not turned into a link, and citation_audit still records it;
what matters here is that it cannot push a real source down the list.

Accuracy was already held: a marker with no matching source never
becomes a link. This changes what a reader can do with the ones that are
real.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
2026-09-13 03:26:17 +02:00
Daniel
6dcdf36c81 fix: resolve a citation by its number, not its position in the array
A marker names a source by the `number` dedupeSources assigns server-side.
Rendering looked it up as sources[n - 1], which works only while the array
order and the numbers agree. Nothing breaks that today, but it is an implicit
contract across a network boundary: any later filtering or reordering of the
list — hiding low-score sources, say — would point citations at the wrong
source silently, which is worse than not linking at all.

Matching on the number cannot drift. Positional lookup remains as the fallback
for a list whose entries carry no number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
2026-09-10 19:10:54 +02:00
Daniel
2096057ebd perf: cache generated images for the session and decode thumbnails, not originals
Generated assets are served `private, no-store`, which is right for a clinical
app on a shared workstation — but it meant every gallery render re-downloaded
every image. Measured: 26 images averaging 279kB, so 7.2MB fetched to draw 56px
tiles, on every open of the Create image popup.

Two changes, both reusable anywhere in the app:

- A session cache holding decoded blobs in MEMORY ONLY, so nothing is written to
  disk and the no-store posture is unchanged. Entries are keyed by owner ticket
  as well as asset, and cleared when the account boundary moves, so one account
  can never read another's bytes out of memory. Concurrent tiles asking for the
  same asset share one request rather than racing.
- Any img carrying data-image-thumb gets a downscaled copy instead of the
  original, so a 56px tile no longer decodes a 300kB image. The gallery asks for
  256px and the in-chat preview for 640px; opening the full view still gets the
  original. Browsers without OffscreenCanvas, and any decode failure, fall back
  to the full image rather than showing nothing.

This does not reduce the first fetch. Serving genuinely smaller bytes needs
server-side resizing, which needs an image library this project does not carry —
worth a deliberate decision rather than adding a native dependency in passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GmpYHPSLGmXGZMyLpn2Lbe
2026-09-10 05:20:10 +02:00
Daniel
9788b167f2 refactor: retrieval is text-only; admins can add model ids discovery never returns
Multimodal removal
The multimodal path called nc_multimodal_search against a second hardcoded
collection whose embedding service (multimodal-embeddings:7999) was never
deployed and ENABLE_MULTIMODAL_RAG has always been false, so it only ever logged
"multimodal search skipped". Removed rather than left as dead weight:

- clinicalRetrieval: normalizeMcpMultimodalResponse, isVisualSourceQuery,
  isRadiologyQuery, buildMultimodalSearchQuery, classifyAndRerankMultimodalResults,
  selectMultimodalResults, visualIntent, visualMetadataScore,
  shouldRejectVisualSource, allowsFrontMatterQuery, looksLikeFrontMatterPage,
  looksLikeTextOnlyPage and MULTIMODAL_CANDIDATE_LIMIT (~140 lines).
- clinicalMcpClient: multimodalSearch.
- The route's visual/text slot split is gone; the whole search limit is text.
- The "[visual PDF page match]" prompt label and the "visual PDF page" source
  badge are gone with it.

Adding models
Model availability could only be ticked from what the gateway advertised, so an
admin could never offer a model discovery did not list. Each list now has a text
field: a typed id joins the same checkbox list, is enabled by default, is
de-duplicated, and persists through the normal allowed_models save.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkfrkQwA4YGrGw9LZSpeAq
2026-09-09 23:35:06 +02:00
Daniel
d04a3fe53b fix: translate as HTML so tables and emphasis survive; stop repeat image generation
Translation formatting
LibreTranslate's text mode destroys markdown syntax. Verified against the live
container: table pipes come back as "←", the |---| delimiter row is translated
as prose ("Silencio."), and "**bold**" returns as "** bold**" which no longer
renders. Its html mode leaves tags — and bare [n] markers — completely intact.
Messages and the patient take home are now rendered to HTML, simplified (maths
and UI chrome flattened to text), and translated as HTML. Citation chips are
re-linked from the returned markers afterwards, which is the step the original
html path was missing. A text-mode fallback remains for builds that reject html.

Repeat image generation
Typing "Окей" or "Nice" after an image turn produced another image every time:
the model saw its own "I'll generate an educational image…" in the history and
repeated it. Recognising acknowledgements in every language is not possible, so
the rule is inverted — a short follow-up (<=3 words) that mentions nothing about
a picture does not get the image tool offered at all when the previous assistant
turn produced an image. Terse repeat requests ("again", "ещё", "another one")
still work. The worst case is that a terse question is answered in text.

In-chat images
Generated images render as a 320x240 thumbnail instead of filling the bubble,
and the image itself opens the full-resolution preview.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkfrkQwA4YGrGw9LZSpeAq
2026-09-09 18:49:58 +02:00
Daniel
653d6d6ed6 fix: strip [src]/[source] placeholder tokens from rendered answers instead of showing src
All checks were successful
Forgejo Android APK / Root app tests (push) Successful in 59s
Forgejo Android APK / Build signed APK (push) Successful in 2m15s
2026-09-09 04:22:15 +02:00
Daniel
73955dd5f9 feat: model selection in chat (composer pill + popup image model), deterministic image dispatch when the model writes text instead of calling the tool, markdown-rendered email, OWUI stacked composer, left mobile drawer, Go back in topbar, DeepL removed, image history inside the Create image popup 2026-09-09 02:55:28 +02:00
Daniel
982a089fcb fix: authHeaders must include Content-Type — Express never parsed JSON bodies, causing empty "Question is required" on every chat send. Also: Learning Hub back in the main menu, sources-only right column, image gallery as the user library with poll+preview, queued job reference handling 2026-09-09 02:13:17 +02:00
Daniel
7fdbcd51e6 feat: ChatGPT-style chat list — abbreviated rows with relative dates, pinned-first ordering, ⋮ menu (Rename/Pin/Export/Delete); clinical context always on; warn-only budget; mic live-transcript fallback; learning-hub page mode; composer autosize; include-context icon fix 2026-09-09 00:14:34 +02:00
Daniel
23788d63c5 feat: Open WebUI presentation — unboxed assistant messages, hairline OWUI tables without card containers
All checks were successful
Forgejo Android APK / Root app tests (push) Successful in 54s
Forgejo Android APK / Build signed APK (push) Successful in 2m2s
2026-09-08 23:46:29 +02:00
Daniel
bb218750d9 feat: Create image tool at rail top (describe or pick a chat), Images gallery of past creations, ChatGPT-style composer icons (headset/mic/attach/send), example tap asks immediately 2026-09-08 23:40:53 +02:00
Daniel
da86c7db79 fix: OWUI-style polish — fullscreen workspace replaces main menu, row-click saved chats (delete stays as hover), markdown-rendered patient handout, admin-only raw transcript, /api TTS URL, scrollable unclipped tables, action buttons hidden in exports, autosave-only copy 2026-09-08 22:41:08 +02:00
Daniel
79de7ccddb fix: reviewer findings — mhchem macro wrapping, regenerate history dedupe, strict safe image allowlist, stale docs 2026-09-08 19:25:31 +02:00
Daniel
8b072496e2 feat: Open WebUI-style assistant workspace — 3-column layout, markdown/math/code/tables, autosave with images, translation (LibreTranslate+DeepL), citation modal, Learning Hub moved in, handoff removed 2026-09-08 18:52:35 +02:00
Daniel
94210ac0a1 feat: attach images to clinical search with strict validation and multimodal routing 2026-09-08 15:53:05 +02:00
Daniel
6be2d1375a feat: integrate durable image jobs/private assets into current core 2026-09-07 16:53:42 +02:00
Daniel
40e7dd5206 fix: bind clinical image and export actions to their original account
All checks were successful
Forgejo Android APK / Root app tests (push) Successful in 31s
Forgejo Android APK / Build signed APK (push) Successful in 1m57s
2026-09-07 07:45:48 +02:00
Daniel
dc83f6e21b fix: preserve export navigation and clinical source boundaries 2026-09-07 07:10:47 +02:00
Daniel
b0bebe6970 fix: preserve saved clinical tables and per-turn citation targets 2026-09-07 05:58:12 +02:00
Daniel
cfaf8e957b feat: ship reviewed prompt history, conversation limits and account protections 2026-09-07 04:01:01 +02:00
Daniel
2942d4f6b3 Revert "Sanitize citations from clinical image prompts"
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m43s
This reverts commit 7833018695.
2026-09-02 01:50:44 +02:00
Daniel
7833018695 Sanitize citations from clinical image prompts
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m47s
2026-09-02 01:47:32 +02:00
Daniel
491f5e02b7 Render escaped clinical citations as links
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m49s
2026-09-01 19:19:11 +02:00
Daniel
e4dacbaf09 Rejoin detached clinical citations
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m50s
2026-09-01 19:11:18 +02:00
Daniel
34eef6ec6b Keep prose citations inline
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m50s
2026-09-01 18:51:39 +02:00
Daniel
e71a7e22e1 Keep clinical citations inline
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m54s
2026-09-01 18:45:44 +02:00
Daniel
3d4a95fea4 Revert the clinical assistant markdown changes and fix mermaid rendering only
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m50s
Reverts 126d7928 and 2d292d12 in full. The markdown normaliser and the trailing
emphasis and dollar handling return to exactly the state tagged
pre-citation-fixes-20260828, which has been rendering acceptably in use.

The one change kept is mermaid. DOMPurify 3.1.6 strips an attribute whose value
contains "-->", and every mermaid flowchart contains one, so the sanitiser
removed data-mermaid and querySelectorAll('[data-mermaid]') never matched: the
diagram sat on "Rendering graph..." forever. Reproduced against the exact pinned
build from cdnjs; sequence diagrams using "->>" were unaffected, which is why
only flowcharts failed.

The diagram source is now percent-encoded into the attribute and decoded when
read, so no arrow ever appears in an attribute value. Nothing about the
sanitiser configuration changes and no markup is newly allowed. The reader
tolerates an unencoded value, so anything already in flight still renders.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 19:55:03 +02:00
Daniel
2d292d12af fix: make the markdown repairs line aware so tables and ranges survive
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m58s
The earlier fix corrected two of the repair rules and left the rest with the same
flaw, and the punctuation boundary it introduced let the table case back in. A
review found the remainder.

Every repair rule inserted newlines with no idea whether it was inside a markdown
table row. A dose row such as "| Dexamethasone | 0.6 mg/kg PO (max 16 mg) - single
dose | 1 |" was split mid-row, which drops that row's citation and every row
below it out of the table. A "#" column destroyed the table outright. The rules
now skip table rows, dividers and headers entirely.

Ranges were still split in three other places: in a heading, so "### Dexamethasone
0.15 - 0.6 mg/kg" rendered as a heading reading "Dexamethasone 0.15"; after a
citation; and beside a hash, where "Room # 4" became a heading. A hyphen now
starts a list item only when a number does not follow, and a run-together heading
is recognised by the capital letter after the hashes rather than by position.

Adjacent citation merging could also cross a paragraph break, turning "[1]\n\n[2]
Vancomycin only if MRSA" into "[1, 2]Vancomycin only if MRSA" -- joining two
paragraphs and moving a citation onto a claim it never supported. Merging is now
limited to citations on the same line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 19:49:01 +02:00
Daniel
126d7928a2 fix: stop the markdown normaliser mangling doses, bold and dollar amounts
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m52s
Three rendering defects in the clinical assistant, all confirmed by running the
code rather than reading it.

A spaced hyphen anywhere in a sentence was rewritten as a list item, so every
numeric range written that way was split in two:

    "Give dexamethasone 0.15 - 0.6 mg/kg orally."
      -> "Give dexamethasone 0.15\n- 0.6 mg/kg orally."

which renders as a truncated sentence followed by a bullet, and a dose range
therefore reads as a different dose. The same applied to SpO2 targets, pH ranges
and age ranges. A hyphen now starts a list item only at the beginning of a line
or after sentence punctuation, which still catches the case the rule was written
for.

An answer ending in a bold phrase lost its closing marker, because trailing
emphasis was stripped unconditionally and the strip ran twice. Only an unpaired
marker is removed now.

Inline maths swallowed dollar amounts: "Costs $5 to $10 per dose" rendered the
text between the signs as an equation and dropped both signs. A $...$ span is
now treated as maths only when it contains something mathematical, so subscripts
and fractions still render.

Citation handling is unchanged and covered by the same tests: clusters still
merge and sort, unknown source numbers stay literal, links still resolve by index,
and bare numbers in a table's Source column still become bracketed tokens.

The state before these fixes is tagged pre-citation-fixes-20260828.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 19:20:39 +02:00
Daniel
c88cc6a547 Prevent citations in clinical image prompts
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m58s
2026-08-28 03:23:18 +02:00
Daniel
bee9361c1d Fix authenticated mobile image downloads
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 2m39s
2026-06-09 16:02:11 +02:00
Daniel
2ca969e099 Fix generated image mobile downloads
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 2m21s
2026-06-09 15:20:06 +02:00
Daniel
80139d9a82 Prevent mobile image download preview fallback
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 1m58s
2026-06-09 03:11:35 +02:00
Daniel
f7cfc6695d Fix clinical assistant image downloads
All checks were successful
Forgejo Android APK / Build signed APK (push) Successful in 2m3s
2026-06-08 22:15:34 +02:00
Daniel
2cef65fb1f Fix mobile assistant image downloads
Some checks failed
Forgejo Android APK / Build signed APK (push) Has been cancelled
2026-05-13 16:07:41 +02:00
Daniel
97ddd87449 fix mobile assistant table streaming
Some checks failed
Forgejo Android APK / Build signed APK (push) Has been cancelled
2026-05-11 16:15:45 +02:00
Daniel
6e8fae72e7 fix mobile image save to photos
Some checks failed
Forgejo Android APK / Build signed APK (push) Has been cancelled
2026-05-11 01:28:43 +02:00
Daniel
2c287bd1b3 fix mobile export save actions
Some checks failed
Forgejo Android APK / Build signed APK (push) Has been cancelled
2026-05-10 23:56:51 +02:00
Daniel
f871384063 fix mobile assistant export and image actions
Some checks failed
Forgejo Android APK / Build signed APK (push) Has been cancelled
2026-05-10 20:28:46 +02:00
Daniel
046b07a84a fix clinical assistant mobile exports 2026-05-10 17:23:54 +02:00
Daniel
a176e1b014 protect code blocks during citation repair 2026-05-09 20:54:05 +02:00
Daniel
83d9a77160 fix table source citation links 2026-05-09 20:50:54 +02:00
Daniel
795ad9ffae harden clinical assistant source handling 2026-05-09 19:59:04 +02:00
Daniel
326fb726a1 split assistant frontend modules 2026-05-08 01:19:12 +02:00
Daniel
f54b293d39 extract assistant frontend api client 2026-05-08 00:58:44 +02:00
Daniel
a8f364f177 stabilize assistant and add diagrams 2026-05-08 00:23:00 +02:00
Daniel
c8436d5e4c refactor clinical assistant prompt handling 2026-05-07 22:52:46 +02:00
Daniel
21fb631fb5 feat: improve clinical assistant export and citations 2026-05-07 17:15:26 +02:00