pediatric-ai-scribe-v3/docs/my-resources.md

# My Resources

Teaching material a signed-in user generates for themselves — a deck for
tomorrow's session, a handout, a summary — kept privately and exported as
PowerPoint, Word or PDF.

Deliberately personal: everything here belongs to whoever made it. It needs no
role beyond being signed in, and nothing is shared. Every statement filters on
`user_id`, and there is no route that returns another person's work. Sharing,
if it is ever wanted, should be a deliberate feature rather than something that
leaks out of a forgotten `WHERE` clause.

## What a resource is

| Column | |
|---|---|
| `markdown` | the readable artifact — what Word renders and what a text edit edits |
| `deck` | for a presentation, the slide structure the model designed (see below) |
| `image_ids` | the illustration jobs belonging to this resource, in the order they were made |
| `topic`, `grounded_count` | what it was asked for and how many library excerpts it was written from |

`MAX_PER_USER` caps how many a person may keep.

## Sources

One function, `gatherSources()`, answers "what is this written from" for both
generating and modifying, so the two cannot drift into offering different things
or searching them differently.

- **The clinical library** — on by default. Semantic retrieval over the indexed
  corpus. Budgets are in [retrieval-tuning.md](retrieval-tuning.md).
- **PubMed** — admin-enabled, optional API key. Returns structured records so a
  reference carries a PMID somebody can look up.
- **The web** — admin-enabled, provider-configurable (Tavily, Serper, Brave,
  SearXNG).
- **Illustrations** — see below.

Each option hides itself when an administrator has not enabled it, so nothing
appears that a person could tick and then be refused.

Nothing here may fail a generation. A retrieval or search that comes back empty
is reported as a reason and the resource is written from what is available.

### Searching is the route's job, not the model's

Both searches run up front on the topic, and their results go into the prompt as
findings. They are **not** offered as tools.

They were, once. Tested live against a question explicitly about recent trials,
the model never called them — with or without corpus grounding, and no matter
how the tool description was worded, because the prompt ends "Output ONLY Pandoc
markdown" and a model told to output only markdown does not emit a tool call.
Calling `callAI` with the tool directly produced a correct call, so the plumbing
was never the problem. The search only ever needed the topic, and the route
knows the topic before it calls the model.

### Keyword searches get the topic, retrieval gets more

PubMed ANDs every mapped term, so one unrecognised word takes a query to zero:
`"febrile seizures"` returns six results and `"febrile seizures in under-fives"`
returns none. A query that finds nothing is retried against progressively
shorter versions of itself, longest first, and the response says which query
actually worked. Those retries are spaced — three `esearch` calls back to back
trips NCBI's three-a-second limit without an API key.

When modifying, the library search gets the topic *plus* the instruction —
retrieval is semantic and benefits from context — while PubMed and the web get
the topic alone.

### Empty searches must not invite invention

When a search was asked for and came back empty, the prompt says not to invent a
citation, a PMID or a URL to fill the gap. Without that the model supplies them
from memory, and a fabricated PMID is indistinguishable from a real one.

## Presentations are designed, not written

A presentation is described as a **deck**: the model returns JSON naming a
layout per slide and the prompt for each figure it wants. Articles stay
markdown, which is what prose wants.

Markdown could express about five of the things the renderer can draw, so the
model had no way to say "put this figure beside these three bullets" or "make
this a comparison with two labelled columns" — the parser inferred a layout from
the shape of a list, and inferring is what made every deck look the same.

Layouts: `title`, `section`, `bullets`, `two`, `compare`, `table`, `callout`,
`figure`, `image`. See `src/utils/deckSchema.js` for what each accepts.

### When the named layouts are not enough

Those nine are a fixed vocabulary, so "lay the three severity levels out left to
right with arrows between them" had no expression at all. A `custom` slide
carries a list of shapes instead:

```json
{"type":"custom","heading":"Severity at a glance","shapes":[
  {"kind":"rect","x":6,"y":30,"w":26,"h":18,"fill":"DCFCE7","line":"16A34A",
   "runs":[{"text":"MILD","bold":true,"align":"center"}]},
  {"kind":"arrow","x":33,"y":37,"w":8,"h":5,"fill":"94A3B8"},
  {"kind":"chart","chart":"column","x":6,"y":28,"w":56,"h":60,
   "categories":["<6m","6-12m"],"series":[{"name":"Cases","values":[4,22]}]}
]}
```

Kinds: `text`, `rect`, `roundRect`, `ellipse`, `arrow`, `arrowDown`, `chevron`,
`diamond`, `hexagon`, `line`, `image`, `table`, `chart` (column, bar, line, pie,
doughnut — native PowerPoint charts, not pictures of charts).

Coordinates are percentages of the slide, 0–100, so a model can reason about
position without knowing anything about EMU. Shapes draw in array order, so a
later one sits on top.

**The model never emits Python.** It names shapes and the renderer draws them.
Running model-authored code to lay out a slide would be an enormous amount of
trust to buy a feature, on a server that holds clinical data.

Everything is validated in `src/utils/slideShapes.js`, which lives beside the
text that describes the vocabulary to the model so the two cannot drift: kinds
are an allowlist, colours must be six hex digits, coordinates are clamped inside
the slide, counts are capped, and a shape that cannot be understood is dropped.
A custom slide that loses every shape falls back to being a plain one rather than
a heading over an empty frame, and one bad shape never costs the slide it is on.

Verified live: asked to "lay the three severity levels out left to right as
coloured boxes with arrows between them", the model produced
`[rect arrow rect arrow rect]`, chose green/amber/red itself, and it rendered as
asked.

### When it wants something that is not there

The vocabulary is deliberately small, so it needs a way to find out what it is
missing. The model is told to say so:

```json
{"kind":"unsupported","need":"a SmartArt cycle of four stages"}
```

Nothing is drawn for that entry. It is recorded, along with the other signal —
reaching for a kind, chart type or slide type that does not exist, which is how
a model asks by trying. Both produce a log line:

```
[deck-vocabulary] wanted "smartart" (used as a shape kind) while generating: croup severity
```

and increment `ped_ai_deck_vocabulary_gap_total{wanted="smartart"}`, so it can be
counted over time rather than noticed once. Capped per generation, deduplicated,
and it can never fail anything — it is a note to whoever decides what to build
next.

That is the answer to "should this run model-authored code in a sandbox instead".
Maybe, one day, and the log says whether the gap is real. Today the model never
emits Python: running model-authored code to lay out a slide would be an enormous
amount of trust to buy a feature, on a server holding clinical data and secrets,
and it would need its own network-isolated container with dropped capabilities,
a read-only filesystem and hard resource limits before it was even safe to try.

Some of the delta is not closeable by any amount of sandboxing, because it is
python-pptx's own ceiling rather than this vocabulary's: **no SmartArt, no
animations or slide transitions**, and a limited set of chart types. Those are
library limits. A sandbox would let a model write code against the same library
and hit the same wall.

Markdown is still produced, serialised from the deck, so Word export and text
editing keep working and the stored artifact stays readable by a person. The
deck is stored alongside it because that serialisation is lossy by design:
round-tripping through markdown would discard the layout choices.

Nothing costs more than the thing that went wrong — a reply that is not a deck
falls back to asking for markdown; a malformed slide degrades to bullets; a
comparison with one column is not a comparison; JSON wrapped in fences or a
covering sentence is read rather than refused.

## Illustrations

`resourceImages.js`, **not** the shared `imageTool.dispatch` — that one permits
exactly one image per request, which is right for a chat reply and wrong for a
twelve-slide deck, and the clinical assistant depends on that rule. Same queue, same storage, same `my_resources` workflow, same asset
endpoint; only the number differs, bounded at `MAX_IMAGES` because each figure
is a paid request.

"Use 3 diagrams" in the instructions is read as the number it is. Writing
"include a diagram of the airway" switches the illustration option on and says
why, rather than the request being dropped in silence; switching it off by hand
sticks.

`my_resources` is its own image workflow rather than a reuse of `learning_hub`
because `generated_image_links` only accepts `learning_hub` assets — that is the
barrier keeping a private illustration out of published content.

### Getting a model to illustrate at all

Two things had to be right, both measured:

1. The illustration guidance is the **last** thing in the prompt. Placed before
   the output rules it lost: the model returned 3297 characters of markdown and
   zero tool calls, while the same tool and wording in a shorter prompt produced
   three calls.
2. Even last, it loses to a prompt carrying thirty library excerpts —
   deterministically: library off → three calls, library on → none. So when the
   author names a number the call is **required** rather than offered. With no
   number named the choice stays the model's.

A model that has just made three tool calls also tends to sign off instead of
writing — `"I'll create the presentation and the three teaching diagrams."` was
once returned as the resource, 61 characters, because only a completely empty
body counted as missing. A body with no title block and no heading is now
treated as missing whatever its length.

## Export

| Format | Built by |
|---|---|
| `pptx` | `scripts/render_pptx.py` (python-pptx) from the stored deck |
| `docx` | `scripts/render_docx.py` (python-docx) from the same typed source |
| `pdf` | Gotenberg (LibreOffice), from whichever office file above |

Both office formats come from `src/utils/docSpec.js` / `slideSpec.js` rather than
from markdown. Pandoc reads markdown, so a deck had to be flattened first — and a
flattened deck stops being one: a comparison became two headings and two lists, a
callout became bold text, and a figure became nothing at all. From the typed
source a comparison is a labelled two-column table, a callout is a shaded box,
and a figure is embedded at its own aspect ratio with its caption. An article,
which has no deck, is parsed from its markdown into the same blocks.

Pandoc is still installed and is still the fallback for Word.

Pandoc's pptx writer was the ceiling on how good a deck could be, and the model
on top made no difference to it: a handful of reference layouts, no per-slide
layout, no positioning, no control over how large an image is drawn. It also
leaves a bare `<a:bodyPr/>` on every shape, so slides overflowed until autofit
was injected into its emitted OOXML by hand.

The renderer sizes text to fit before writing the file rather than trusting
autofit — LibreOffice ignores `<a:normAutofit/>` when converting to PDF, which
is how slides were being cut off mid-sentence. Wrapped bullet lines hang under
the text. Images are drawn at their own aspect ratio.

If the renderer fails for any reason, pandoc still produces a deck: a plainer
deck beats a failed download. The log line is
`deck renderer failed, falling back to pandoc`.

Figures are fetched to a scratch directory for the renderer and removed
afterwards. One that cannot be fetched is left out rather than failing a
download that works without it.

**Runtime dependency:** the image carries `python3`, `py3-lxml`, `py3-pillow`
(apk — both are C extensions with no Alpine wheels), plus `python-pptx` 1.0.2 and
`python-docx` 1.1.2 from pip, and `poppler-utils` for slide review. Roughly 58MB
of Python. Both pip packages are pinned: unpinned, a rebuild from the same commit
could produce different documents.

## What a modification can and cannot change

The deck vocabulary describes *structure*, not *style*: `bullets`, `compare`,
`table`, `callout`, `figure`, `image`, `section`, `title`. None of them carries a
colour. The palette lives in `scripts/render_pptx.py` as fixed constants —
`ACCENT` (#2563EB), `INK`, `MUTED`, `RULE`, `PAPER` — and the model never sees
them.

So an instruction like "make it yellow" has nowhere to land. The model applies it
to the only colour lever it has: the `image_prompt` text, which produces yellow
*figures* on an otherwise blue deck. That is not the modification going only to
the images — it is the model using the one field that accepts a colour at all.

The exception is a `custom` slide, whose shapes take `fill` and `color` (see
`slideShapes.js`). A model can restyle those, but it will rarely rebuild an
ordinary slide as a custom one just to change a colour.

If deck styling should be changeable, the honest fix is a theme — an accent
colour on the resource, passed to the renderer — not a wider slide vocabulary.

## The image library

Library → **Images** is every picture the account has generated — from My
Resources and from the Clinical Assistant — newest first. A figure outlives the deck it was drawn for: the deck gets
replaced, the diagram is still good.

`GET /api/generated-images` returns only finished jobs, scoped by `owner_id` in
the statement rather than filtered afterwards. Paging is keyset (`created_at <
cursor`), not `OFFSET`, because a gallery that grows while you scroll repeats or
skips a row under `OFFSET`. The prompt is the only human-readable label an image
has — there is no filename and no title — so it is decrypted for the caption; a
prompt that cannot be decrypted costs the caption, never the picture.

Tiles request the stored 256px preview through `data-image-thumb`, so thirty
tiles cost a few kB each rather than thirty full-size downloads. Every fetch
goes through `hydrateImage`, never a bare `src`: assets are served `no-store`
and a bare `src` would not carry the session on a mobile client.

### Image model fallbacks

Every workflow tries its configured model first, then each fallback in order,
stopping at the first that produces an image. Primary plus two, capped — each
hop is a paid request. Set in **Admin → Models → Availability → Image generation, per workflow**.

A fallback is only tried where another model has a real chance:

| Failure | Next model? | Why |
|---|---|---|
| Timeout, 429, 5xx, network fault | yes | The provider said "not now", not "not ever" |
| 404 — the gateway does not have that model | yes | A configuration mistake the next model rescues |
| A content refusal | yes | Policy is a vendor decision, not a fact about the request |
| 401 / 403 | **no** | One gateway, one set of credentials; the next model fails identically |
| 413 — too large | **no** | It is too large everywhere |
| Any other 4xx | **no** | Malformed is malformed everywhere |
| Cancelled, or shutting down | **no** | Never start more paid work |

A refusal is recognised from the message, because no provider sends a
machine-readable reason and the status varies — 400 from some, 422 from others.

Each hop re-leases the job, so a chain cannot outlive its claim and let a second
worker repeat the same paid work; if the lease has gone the attempt stops there
rather than paying again. The row records the model actually being paid for, so
a picture made by the third model is not attributed to the first, and every hop
is logged with the reason it moved on.

This used to be the Clinical Assistant alone, with one fallback. My Resources
had none at all — which is where a missing picture is most visible, because it
leaves a hole in a slide.

### Deleting

`DELETE /api/generated-images/:id` removes the bytes before the row, and refuses
the whole operation if storage is unreachable. The other order would leave a row
pointing at a key that is gone — an image listed in the gallery that renders
broken — whereas failing between the two leaves a complete, working image and an
error worth retrying.

Both derived previews go with the original; they live under their own prefix in
the same bucket, and missing them would leave paid-for bytes behind that are
still readable. `THUMB_WIDTHS` is defined once, in `generatedImageStorage.js`,
because a width that is written but never deleted is exactly what two copies of
that list produces.

A resource that embedded the figure keeps working: a deck stores the job id and
renders without the figure when it has gone.

## Slide review

Off unless an administrator names a model, in **Admin → Models → Availability → Slide review**.

The model that writes a deck never sees it, so overflow, a figure on the wrong
slide and a nine-item list that wants two columns are invisible to it. With a
reviewer configured, each generated deck is rendered to PDF through Gotenberg,
rasterised to one PNG per slide with `pdftoppm`, and shown to a vision model.

One pass per change — on generation, and again on the result of a modification.
Modifying was excluded at first on the reasoning that refining is a text edit.
It is not: an edit is made against how the deck looked *before* it, so a slide
that gains two bullets only overflows once it is rendered again, which is
exactly what the reviewer exists to catch.

### Modifying can see the deck too

When a vision model is configured, modifying renders the current deck — with
its figures — and hands the model one image per slide alongside the JSON. Most
of what people ask for while modifying is about the rendered page: "that slide
is crowded", "the diagram is in the wrong place", "this one looks empty". None
of it is answerable from the JSON.

The vision model then does the editing, which is a second benefit measured
before this was built: on a real 20-slide deck, `ds-deepseek-v4-flash` returned
the deck unchanged for "make it better" and `openrouter-gemini-3.8-flash` did
not. A model the author picks explicitly still wins over both.

Sight is an upgrade, never a dependency. No vision model configured, Gotenberg
down, a render that fails — each falls through to editing the JSON blind, which
is what this did before it could see at all, and none of them may cost someone
their modification.

The reviewer must be able to see. Saving `my_resources.review_model` asks the
gateway what it reports for that model and refuses one whose `supports_vision`
is explicitly `false` — otherwise the mistake surfaces as a failed request on
every generation, long after the moment an administrator could have chosen
differently. A model the gateway says nothing about is allowed: most of a
roster carries no `supports_vision` at all, and silence is not proof of
blindness. An unreachable gateway is not evidence either, and never blocks the
save.

### It returns a patch, not a deck

```json
{"changes":[
  {"slide":2,"action":"two"},
  {"slide":4,"action":"split","after":3,"heading":"Management (continued)"},
  {"slide":6,"action":"compare","at":3,"labels":["MILD","SEVERE"]}
]}
```

Asking for the corrected deck back put the reply in proportion to the *deck*
rather than to the number of problems — a fourteen-slide deck came back cut off
mid-object every time, at any output budget the provider would honour.

The patch is better for a second reason. The reviewer names a slide and an
action; the server moves the text it already has. The words never pass through
the model at all, so a review cannot reword, drop or invent a single bullet —
which is a stronger guarantee than instructing it not to and checking
afterwards. The check still runs: body text must come out the same multiset,
figures the same set, and a heading may only be reused or extended. A
continuation heading is the reviewer's one piece of text, and it is replaced
with `"<original> (continued)"` if it does not continue anything.

Nothing here can fail a generation. No reviewer, an unreachable one, an
unparseable reply, a deck longer than `MAX_SLIDES`, or a patch that applies to
nothing — each returns the deck that was written.

### Cost

One image per slide on every presentation generated. Pick a cheap capable vision
model rather than the best one available; `openrouter-gemini-3.8-flash` is a
reasonable default. Measured on a three-slide deck: three images in, one change
out.

## Modify

`POST /api/my-resources/:id/refine` rewrites a resource in place, keeping its
id, its downloads and its References section. It offers the same four sources as
generating — it had none, so "add what the 2024 trial showed" was answered from
the model's memory rather than by looking anything up.

The previous version is replaced, not versioned.

## Sharing

A resource is its author's. `POST /api/my-resources/:id/shares` (by email,
exact — accounts are never listed) and `PUT …/shares/all` extend *reading* —
open, preview, download — to named people or to everyone signed in; the rows
live in `user_resource_shares` and `user_resources.shared_with_all`. Every
write route (modify, theme, delete, the share list itself) still filters on
`user_id`. The list route returns a person's own resources first, then what is
shared with them, each row saying `owned` and `shared_by_name`.

## Preview

`GET /api/my-resources/:id/preview` renders the resource the way its download
is built and turns it into one PNG per page (`src/utils/previewPages.js`:
Gotenberg to PDF, `pdftoppm` to pages), keyed on `updated_at` and theme and
kept under the OS temp directory; `…/preview/:page` serves a page. Theme
samples have the same pair under `theme-sample/:id/preview`.