feat: image fallback chains for every workflow, and a library worth looking at
Some checks failed
Forgejo Android APK / Root app tests (push) Successful in 55s
Forgejo Docker Build / Root app tests (push) Successful in 45s
Forgejo Android APK / Build signed APK (push) Successful in 2m5s
Forgejo Docker Build / Build Docker image (push) Successful in 19s
Forgejo Docker Build / Deploy to the host (push) Failing after 0s

**Fallbacks.** One image model meant a refusal, a rate limit or a model the
gateway had since dropped ended as a missing picture. Every workflow now tries
its model, then each fallback in order, stopping at the first that produces an
image. Primary plus two, capped: each hop is a paid request, and a chain long
enough to need a cap is long enough to surprise someone.

My Resources previously had no fallback at all — only the Clinical Assistant
did, and only one. That is backwards: a missing figure is most visible in a
deck, where it leaves a hole in a slide.

The retry rule is now a classifier that says *why*, rather than a boolean.
Transient faults, a 404 for a model the gateway does not have, and a content
refusal all move to the next model — a refusal because policy is a vendor
decision, not a fact about the request. 401/403 stop immediately (one gateway,
one set of credentials, the next model fails identically), as do 413 and any
other 4xx, which are malformed everywhere. Refusals are recognised from the
message: no provider sends a machine-readable reason and the status varies.

Each hop re-leases the job, so a chain cannot outlive its claim and let a second
worker repeat the same paid work, and the row records the model actually being
paid for so a picture made by the third model is not attributed to the first.

The old singular `fallback_image_model` is still read, so an existing
configuration keeps working without anyone re-entering it.

**Library.** Documents/Images tabs in My Resources, with a real grid: fixed
aspect tiles so the rows line up whatever shape the pictures are, a source badge
on the picture, two-line prompt, hover lift, shimmer skeletons while thumbnails
land, and a lightbox that closes on Escape or the backdrop and restores focus.
Actions are hidden on hover only behind `@media (hover:hover)` — hiding delete
behind :hover would put it out of reach on touch and keyboard.

Downloads go through privateImageBlob rather than a bare `<a download href>`: a
mobile client's session is a bearer token an anchor cannot send, and these
assets are served no-store on purpose.

The gallery lives in My Resources only. Assistant images appear in it, which was
the point; the assistant page does not grow a gallery of its own, and a test
asserts no assistant module lists the endpoint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dv6sqaY6Vq3ChZHMem3cnU
This commit is contained in:
Daniel 2026-09-12 14:53:04 +02:00
parent 259b4858be
commit 03621752e8
10 changed files with 653 additions and 105 deletions

View file

@ -272,6 +272,37 @@ tiles cost a few kB each rather than thirty full-size downloads. Every fetch
goes through `hydrateImage`, never a bare `src`: assets are served `no-store`
and a bare `src` would not carry the session on a mobile client.
### Image model fallbacks
Every workflow tries its configured model first, then each fallback in order,
stopping at the first that produces an image. Primary plus two, capped — each
hop is a paid request. Set in **Admin → Image models**.
A fallback is only tried where another model has a real chance:
| Failure | Next model? | Why |
|---|---|---|
| Timeout, 429, 5xx, network fault | yes | The provider said "not now", not "not ever" |
| 404 — the gateway does not have that model | yes | A configuration mistake the next model rescues |
| A content refusal | yes | Policy is a vendor decision, not a fact about the request |
| 401 / 403 | **no** | One gateway, one set of credentials; the next model fails identically |
| 413 — too large | **no** | It is too large everywhere |
| Any other 4xx | **no** | Malformed is malformed everywhere |
| Cancelled, or shutting down | **no** | Never start more paid work |
A refusal is recognised from the message, because no provider sends a
machine-readable reason and the status varies — 400 from some, 422 from others.
Each hop re-leases the job, so a chain cannot outlive its claim and let a second
worker repeat the same paid work; if the lease has gone the attempt stops there
rather than paying again. The row records the model actually being paid for, so
a picture made by the third model is not attributed to the first, and every hop
is logged with the reason it moved on.
This used to be the Clinical Assistant alone, with one fallback. My Resources
had none at all — which is where a missing picture is most visible, because it
leaves a hole in a slide.
### Deleting
`DELETE /api/generated-images/:id` removes the bytes before the row, and refuses

View file

@ -177,7 +177,7 @@
<div id="mr-images-panel" role="tabpanel" aria-labelledby="tab-mr-images" hidden
style="padding:12px 16px;max-height:420px;overflow-y:auto;">
<p id="mr-images-empty" style="margin:0;font-size:12px;color:var(--g500);">Loading…</p>
<div id="mr-images-grid" style="display:grid;grid-template-columns:repeat(auto-fill,minmax(150px,1fr));gap:10px;"></div>
<div id="mr-images-grid" class="img-grid"></div>
<div style="display:flex;justify-content:center;padding-top:10px;">
<button id="btn-mr-images-more" class="btn-sm btn-ghost" type="button" hidden>Load more</button>
</div>

View file

@ -1494,3 +1494,64 @@ button, a, .btn-sm, .btn-generate, .btn-send, .tab-btn, input, textarea, select,
@media (max-width:640px) {
.assistant-history .assistant-mode-switch { margin:6px 14px 10px; }
}
/* Image library
A grid of pictures, not a list of filenames: an image is recognised by
looking at it. Styles live here rather than inline so the hover and focus
states can exist at all an inline style cannot express either. */
.img-grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(168px,1fr));gap:12px;}
.img-tile{margin:0;display:flex;flex-direction:column;background:var(--white,#fff);
border:1px solid var(--g200);border-radius:10px;overflow:hidden;
transition:border-color .15s ease, box-shadow .15s ease, transform .15s ease;}
.img-tile:hover{border-color:var(--g300);box-shadow:0 6px 16px rgba(17,24,39,.08);transform:translateY(-1px);}
.img-tile:focus-within{border-color:var(--blue);box-shadow:0 0 0 3px var(--blue-light);}
/* A fixed ratio so the grid lines up whatever shape the pictures are, and a
neutral ground so a transparent PNG does not read as a hole in the page. */
.img-tile-frame{position:relative;aspect-ratio:4/3;background:var(--g100);
display:flex;align-items:center;justify-content:center;overflow:hidden;cursor:zoom-in;border:0;padding:0;width:100%;}
.img-tile-frame img{width:100%;height:100%;object-fit:cover;display:block;}
.img-tile-frame:focus-visible{outline:2px solid var(--blue);outline-offset:-2px;}
/* The source badge sits on the picture: it is one word, and a row of its own
would cost more space than it is worth. */
.img-tile-badge{position:absolute;top:8px;left:8px;padding:2px 7px;border-radius:999px;
font-size:10px;font-weight:600;letter-spacing:.02em;background:rgba(17,24,39,.72);color:#fff;
backdrop-filter:blur(2px);}
.img-tile-body{padding:9px 10px 10px;display:flex;flex-direction:column;gap:6px;min-width:0;}
.img-tile-prompt{font-size:11.5px;line-height:1.4;color:var(--g700);
display:-webkit-box;-webkit-line-clamp:2;-webkit-box-orient:vertical;overflow:hidden;}
.img-tile-meta{font-size:10.5px;color:var(--g500);font-variant-numeric:tabular-nums;}
/* Actions appear on hover on a pointer device, and are always present for
keyboard and touch hiding them behind hover would put delete out of reach. */
.img-tile-actions{display:flex;gap:6px;margin-top:2px;opacity:1;transition:opacity .15s ease;}
@media (hover:hover){
.img-tile-actions{opacity:0;}
.img-tile:hover .img-tile-actions,
.img-tile:focus-within .img-tile-actions{opacity:1;}
}
.img-tile-action{border:1px solid var(--g200);background:var(--white,#fff);color:var(--g600);
font:inherit;font-size:11px;padding:3px 9px;border-radius:6px;cursor:pointer;}
.img-tile-action:hover{background:var(--g50);border-color:var(--g300);}
.img-tile-action.danger{color:var(--red);}
.img-tile-action.danger:hover{background:var(--red-light);border-color:var(--red);}
.img-tile-action[disabled]{opacity:.5;cursor:default;}
/* Skeletons while the thumbnails arrive, so the grid does not reflow under the
reader as each one lands. */
.img-tile-frame.loading{background:linear-gradient(90deg,var(--g100) 25%,var(--g200) 37%,var(--g100) 63%);
background-size:400% 100%;animation:img-shimmer 1.4s ease infinite;}
@keyframes img-shimmer{0%{background-position:100% 0}100%{background-position:0 0}}
@media (prefers-reduced-motion:reduce){
.img-tile,.img-tile-actions{transition:none}
.img-tile-frame.loading{animation:none}
}
/* Full-size view. */
.img-lightbox{position:fixed;inset:0;z-index:9999;background:rgba(17,24,39,.82);
display:flex;flex-direction:column;align-items:center;justify-content:center;gap:12px;padding:24px;}
.img-lightbox img{max-width:100%;max-height:calc(100vh - 140px);object-fit:contain;
border-radius:8px;box-shadow:0 16px 48px rgba(0,0,0,.45);background:var(--g100);}
.img-lightbox-bar{display:flex;gap:8px;align-items:center;flex-wrap:wrap;justify-content:center;max-width:640px;}
.img-lightbox-caption{color:#e5e7eb;font-size:12px;line-height:1.5;text-align:center;}

View file

@ -1,8 +1,21 @@
// One image model per workflow: Clinical Assistant and Learning, each a dropdown of discovered models.
// One image model per workflow, plus the models to try when it fails.
//
// A single model means a refusal, a rate limit or a model the gateway has since
// dropped ends as a missing picture. The chain is primary first, then each
// fallback in order, and it stops at the first one that produces an image —
// every hop being a paid request is why it is capped rather than open-ended.
import { imageJson } from '../generatedImages.js';
const WORKFLOWS = [
{ key: 'clinical_assistant', label: 'Clinical Assistant' },
{ key: 'learning_hub', label: 'Learning Hub' },
{ key: 'my_resources', label: 'My Resources' }
];
let loading = false;
let loaded = false;
let clinicalSelect; let learningSelect; let clinicalBudget; let learningBudget; let save; let status;
let controls = {};
let save; let status; let maxModels = 3;
async function load() {
const root = document.getElementById('workflow-image-settings');
@ -14,50 +27,122 @@ async function load() {
imageJson('/api/admin/image-settings'),
imageJson('/api/admin/config/image-models/discover').catch(() => ({ success: false }))
]);
maxModels = Number(settings.maxModels) || 3;
const discovered = Array.isArray(models.models) ? models.models : [];
root.textContent = '';
const form = document.createElement('form');
const heading = document.createElement('h4'); heading.textContent = 'Image models';
clinicalSelect = makeSelect([settings.workflows.clinical_assistant.model], models.models);
learningSelect = makeSelect([settings.workflows.learning_hub.model], models.models);
clinicalBudget = makeBudget(settings.workflows.clinical_assistant.budget);
learningBudget = makeBudget(settings.workflows.learning_hub.budget);
save = document.createElement('button'); save.type = 'submit'; save.className = 'btn-sm btn-primary'; save.textContent = 'Save image settings';
const heading = document.createElement('h4');
heading.textContent = 'Image models';
const note = document.createElement('p');
note.style.cssText = 'margin:2px 0 10px;font-size:12px;color:var(--g500);';
note.textContent = 'Each workflow tries its model first, then its fallbacks in order, ' +
'stopping at the first that produces an image. A fallback is only tried when another ' +
'model has a real chance — a refusal, a rate limit, a provider fault, or a model the ' +
'gateway does not have. Bad credentials and a malformed request stop immediately.';
form.append(heading, note);
controls = {};
WORKFLOWS.forEach(workflow => {
const saved = settings.workflows[workflow.key] || {};
const group = document.createElement('fieldset');
group.style.cssText = 'border:1px solid var(--g200);border-radius:8px;padding:12px;margin:0 0 12px;';
const legend = document.createElement('legend');
legend.style.cssText = 'font-size:12px;font-weight:600;color:var(--g700);padding:0 6px;';
legend.textContent = workflow.label;
group.appendChild(legend);
// My Resources picks its model per request from the roster, so only the
// Learning Hub and the Assistant name a primary here.
const primary = makeSelect([saved.model], discovered);
primary.disabled = workflow.key === 'my_resources';
group.appendChild(row(workflow.key === 'my_resources' ? 'Model (set per request)' : 'Model', primary));
const fallbacks = [];
for (let i = 0; i < maxModels - 1; i++) {
const select = makeSelect([(saved.fallbacks || [])[i] || ''], discovered, true);
fallbacks.push(select);
group.appendChild(row('Fallback ' + (i + 1), select));
}
const budget = makeBudget(saved.budget);
group.appendChild(row('Image input budget', budget));
controls[workflow.key] = { primary, fallbacks, budget };
form.appendChild(group);
});
save = document.createElement('button');
save.type = 'submit'; save.className = 'btn-sm btn-primary';
save.textContent = 'Save image settings';
status = document.createElement('p'); status.setAttribute('role', 'status');
const row1 = document.createElement('div'); row1.className = 'admin-row';
const l1 = document.createElement('label'); l1.className = 'admin-row-label'; l1.textContent = 'Clinical Assistant image model'; row1.append(l1, clinicalSelect);
const row2 = document.createElement('div'); row2.className = 'admin-row';
const l2 = document.createElement('label'); l2.className = 'admin-row-label'; l2.textContent = 'Clinical image input budget'; row2.append(l2, clinicalBudget);
const row3 = document.createElement('div'); row3.className = 'admin-row';
const l3 = document.createElement('label'); l3.className = 'admin-row-label'; l3.textContent = 'Learning image model'; row3.append(l3, learningSelect);
const row4 = document.createElement('div'); row4.className = 'admin-row';
const l4 = document.createElement('label'); l4.className = 'admin-row-label'; l4.textContent = 'Learning image input budget'; row4.append(l4, learningBudget);
form.append(heading, row1, row2, row3, row4, save, status);
form.append(save, status);
form.onsubmit = async e => {
e.preventDefault(); if (save.disabled) return; save.disabled = true;
e.preventDefault();
if (save.disabled) return;
save.disabled = true;
try {
await imageJson('/api/admin/image-settings/clinical_assistant', { method: 'PUT', body: JSON.stringify({ model: clinicalSelect.value, budget: Number(clinicalBudget.value) }) });
await imageJson('/api/admin/image-settings/learning_hub', { method: 'PUT', body: JSON.stringify({ model: learningSelect.value, budget: Number(learningBudget.value) }) });
for (const workflow of WORKFLOWS) {
const control = controls[workflow.key];
const body = {
budget: Number(control.budget.value),
fallbacks: control.fallbacks.map(s => s.value).filter(Boolean)
};
// Only the workflows that own a primary send one; My Resources would
// be overwriting a per-request choice with a form field.
if (workflow.key !== 'my_resources') body.model = control.primary.value;
await imageJson('/api/admin/image-settings/' + workflow.key,
{ method: 'PUT', body: JSON.stringify(body) });
}
status.textContent = 'Saved. New jobs use these settings; existing jobs are unchanged.';
} catch (error) { status.textContent = error.message + ' Nothing was saved.'; } finally { save.disabled = false; }
} catch (error) {
status.textContent = error.message + ' Nothing was saved.';
} finally { save.disabled = false; }
};
root.append(form);
loaded = true;
} catch (_) { /* Next tab entry retries; no drafts are touched. */ }
finally { loading = false; }
}
function makeSelect(knownIds, models) {
const select = document.createElement('select'); select.className = 'admin-control';
const ids = new Set([...(Array.isArray(knownIds) ? knownIds : []), ...((Array.isArray(models) ? models : []).map(m => m && m.id).filter(Boolean))]);
[...ids].sort().forEach(id => { const o = document.createElement('option'); o.value = id; o.textContent = id; select.appendChild(o); });
if (knownIds[0]) select.value = knownIds[0];
function row(labelText, control) {
const wrap = document.createElement('div');
wrap.className = 'admin-row';
const label = document.createElement('label');
label.className = 'admin-row-label';
label.textContent = labelText;
wrap.append(label, control);
return wrap;
}
function makeSelect(knownIds, models, optional) {
const select = document.createElement('select');
select.className = 'admin-control';
if (optional) {
const none = document.createElement('option');
none.value = ''; none.textContent = 'None';
select.appendChild(none);
}
const ids = new Set([...(Array.isArray(knownIds) ? knownIds : []).filter(Boolean),
...((Array.isArray(models) ? models : []).map(m => m && m.id).filter(Boolean))]);
[...ids].sort().forEach(id => {
const option = document.createElement('option');
option.value = id; option.textContent = id;
select.appendChild(option);
});
select.value = knownIds[0] || '';
return select;
}
function makeBudget(value) {
const input = document.createElement('input'); input.className = 'admin-control';
input.type = 'number'; input.min = '1000'; input.max = '32000'; input.required = true; input.value = value;
const input = document.createElement('input');
input.className = 'admin-control';
input.type = 'number'; input.min = '1000'; input.max = '32000'; input.required = true;
input.value = value;
return input;
}
export function initImageSettings() {
document.addEventListener('tabChanged', e => { if (e.detail?.tab === 'admin') load(); });
}

View file

@ -331,53 +331,71 @@
// Built as elements, never innerHTML: the caption is a model-written prompt.
function imageTile(image, m) {
var tile = document.createElement('figure');
tile.style.cssText = 'margin:0;border:1px solid var(--g200);border-radius:8px;overflow:hidden;' +
'display:flex;flex-direction:column;background:var(--white);';
tile.className = 'img-tile';
// The picture is the control: clicking it opens the full size. A button
// rather than a div, so it is reachable and announced without inventing
// keyboard handling.
var frame = document.createElement('button');
frame.type = 'button';
frame.className = 'img-tile-frame loading';
frame.title = 'Open full size';
var frame = document.createElement('div');
frame.style.cssText = 'aspect-ratio:4/3;background:var(--g100);display:flex;align-items:center;justify-content:center;overflow:hidden;';
var img = document.createElement('img');
img.alt = image.prompt ? 'Generated illustration: ' + image.prompt.slice(0, 80) : 'Generated illustration';
img.setAttribute('data-image-thumb', '256');
img.style.cssText = 'width:100%;height:100%;object-fit:cover;display:block;';
img.loading = 'lazy';
frame.appendChild(img);
m.hydrateImage(img, image.imageUrl).catch(function () {
img.remove();
frame.textContent = 'Unavailable';
frame.style.fontSize = '11px';
frame.style.color = 'var(--g500)';
});
frame.addEventListener('click', function () { openImage(image, m); });
var caption = document.createElement('figcaption');
caption.style.cssText = 'padding:8px;display:flex;flex-direction:column;gap:6px;min-width:0;';
var badge = document.createElement('span');
badge.className = 'img-tile-badge';
badge.textContent = image.source;
frame.appendChild(badge);
var text = document.createElement('div');
text.style.cssText = 'font-size:11px;color:var(--g600);line-height:1.35;max-height:3.4em;overflow:hidden;';
text.textContent = image.prompt || 'No description recorded';
m.hydrateImage(img, image.imageUrl)
.then(function () { frame.classList.remove('loading'); })
.catch(function () {
frame.classList.remove('loading');
img.remove();
var gone = document.createElement('span');
gone.style.cssText = 'font-size:11px;color:var(--g500);';
gone.textContent = 'Unavailable';
frame.appendChild(gone);
});
var body = document.createElement('div');
body.className = 'img-tile-body';
var prompt = document.createElement('figcaption');
prompt.className = 'img-tile-prompt';
prompt.textContent = image.prompt || 'No description recorded';
prompt.title = image.prompt || '';
var meta = document.createElement('div');
meta.style.cssText = 'font-size:10px;color:var(--g500);display:flex;gap:6px;flex-wrap:wrap;';
meta.textContent = image.source + ' · ' + new Date(image.createdAt).toLocaleDateString();
meta.className = 'img-tile-meta';
meta.textContent = new Date(image.createdAt).toLocaleDateString(undefined,
{ year: 'numeric', month: 'short', day: 'numeric' }) +
(image.bytes ? ' · ' + Math.max(1, Math.round(image.bytes / 1024)) + ' kB' : '');
var actions = document.createElement('div');
actions.style.cssText = 'display:flex;gap:6px;';
var open = document.createElement('button');
open.type = 'button';
open.className = 'btn-sm btn-ghost';
open.style.cssText = 'font-size:11px;padding:3px 8px;';
open.textContent = 'Open';
open.addEventListener('click', function () { openImage(image, m); });
actions.className = 'img-tile-actions';
var download = document.createElement('button');
download.type = 'button';
download.className = 'img-tile-action';
download.textContent = 'Download';
download.addEventListener('click', function () { saveImage(image, m, download); });
var remove = document.createElement('button');
remove.type = 'button';
remove.className = 'btn-sm btn-ghost';
remove.style.cssText = 'font-size:11px;padding:3px 8px;color:var(--red);';
remove.className = 'img-tile-action danger';
remove.textContent = 'Delete';
remove.addEventListener('click', function () { deleteImage(image, tile, remove); });
actions.append(open, remove);
caption.append(text, meta, actions);
tile.append(frame, caption);
actions.append(download, remove);
body.append(prompt, meta, actions);
tile.append(frame, body);
return tile;
}
@ -386,26 +404,83 @@
// no-store on purpose.
function openImage(image, m) {
var overlay = document.createElement('div');
overlay.style.cssText = 'position:fixed;inset:0;background:rgba(0,0,0,.72);z-index:9999;' +
'display:flex;align-items:center;justify-content:center;padding:24px;';
overlay.className = 'img-lightbox';
overlay.setAttribute('role', 'dialog');
overlay.setAttribute('aria-modal', 'true');
overlay.setAttribute('aria-label', 'Generated illustration');
var full = document.createElement('img');
full.alt = image.prompt || 'Generated illustration';
full.style.cssText = 'max-width:100%;max-height:100%;object-fit:contain;border-radius:6px;';
overlay.appendChild(full);
overlay.addEventListener('click', function () { overlay.remove(); });
document.addEventListener('keydown', function escape(e) {
if (e.key !== 'Escape') return;
var bar = document.createElement('div');
bar.className = 'img-lightbox-bar';
var caption = document.createElement('p');
caption.className = 'img-lightbox-caption';
caption.textContent = image.prompt || 'No description recorded';
var close = document.createElement('button');
close.type = 'button';
close.className = 'img-tile-action';
close.textContent = 'Close';
var save = document.createElement('button');
save.type = 'button';
save.className = 'img-tile-action';
save.textContent = 'Download';
save.addEventListener('click', function () { saveImage(image, m, save); });
bar.append(save, close);
overlay.append(full, caption, bar);
// Escape and a click on the backdrop both close it; a click on the picture
// or the buttons does not.
function dismiss() {
overlay.remove();
document.removeEventListener('keydown', escape);
});
document.removeEventListener('keydown', onKey);
if (lastFocus && lastFocus.focus) lastFocus.focus();
}
function onKey(e) { if (e.key === 'Escape') dismiss(); }
var lastFocus = document.activeElement;
overlay.addEventListener('click', function (e) { if (e.target === overlay) dismiss(); });
close.addEventListener('click', dismiss);
document.addEventListener('keydown', onKey);
document.body.appendChild(overlay);
close.focus();
m.hydrateImage(full, image.imageUrl).catch(function () {
full.remove();
overlay.textContent = 'That image could not be loaded.';
overlay.style.color = 'var(--white)';
caption.textContent = 'That image could not be loaded.';
});
}
// Fetched, then saved from memory. A plain <a download href> would work in a
// browser carrying the session cookie and fail on a mobile client, whose
// session is a bearer token an <a> cannot send — and these assets are served
// no-store precisely so they are not sitting in a cache to be linked to.
function saveImage(image, m, button) {
var label = button.textContent;
button.disabled = true;
button.textContent = 'Saving…';
m.privateImageBlob(image.imageUrl)
.then(function (blob) {
var url = URL.createObjectURL(blob);
var link = document.createElement('a');
link.href = url;
link.download = 'illustration-' + image.id.slice(0, 8) + '.' +
({ 'image/png': 'png', 'image/jpeg': 'jpg', 'image/webp': 'webp' })[image.mime] || 'png';
document.body.appendChild(link);
link.click();
link.remove();
// Revoked on the next frame: revoking immediately races the download in
// some browsers.
setTimeout(function () { URL.revokeObjectURL(url); }, 1000);
})
.catch(function () {
if (typeof showToast === 'function') showToast('That image could not be downloaded.', 'error');
})
.finally(function () { button.disabled = false; button.textContent = label; });
}
function deleteImage(image, tile, button) {
// Asked, because it removes the picture from every deck that used it. The
// deck keeps working; the figure is simply no longer there.

View file

@ -147,26 +147,66 @@ learningRouter.post('/jobs', async (req, res) => {
learningRouter.get('/jobs/:id', async (req, res) => {
try { res.json(await images.service().get(req.params.id, req.user.id, 'learning_hub')); } catch (e) { fail(res, e); }
});
const IMAGE_WORKFLOWS = ['clinical_assistant', 'learning_hub', 'my_resources'];
// The old single key, still read so an existing deployment keeps its fallback
// without anyone re-entering it. Writing through the new key supersedes it.
async function fallbackModels(workflow) {
const list = String(await db.getSetting(workflow + '.fallback_image_models') || '').trim();
if (list) return list.split(',').map(s => s.trim()).filter(Boolean);
const legacy = String(await db.getSetting(workflow + '.fallback_image_model') || '').trim();
return legacy ? [legacy] : [];
}
router.get('/admin/image-settings', adminMiddleware, async (req, res) => {
try {
const workflows = {};
for (const workflow of ['clinical_assistant', 'learning_hub']) workflows[workflow] = {
for (const workflow of IMAGE_WORKFLOWS) workflows[workflow] = {
model: await db.getSetting(workflow + '.image_model') || '',
fallbacks: await fallbackModels(workflow),
budget: images.budgetLimit(await db.getSetting(workflow + '.image_budget')), unit: 'UTF-16 code units'
};
res.json({ success: true, workflows });
res.json({ success: true, workflows, maxModels: images.MAX_IMAGE_MODELS });
} catch (e) { fail(res, e); }
});
const MODEL_ID = /^[a-zA-Z0-9_.:/-]{1,200}$/;
router.put('/admin/image-settings/:workflow', adminMiddleware, async (req, res) => {
try {
const workflow = req.params.workflow;
if (!['clinical_assistant', 'learning_hub'].includes(workflow)) throw images.failure(404, 'Workflow not found');
if (!IMAGE_WORKFLOWS.includes(workflow)) throw images.failure(404, 'Workflow not found');
const budget = images.budgetLimit(req.body.budget);
if (workflow === 'learning_hub' && (typeof req.body.model !== 'string' || !/^[a-zA-Z0-9_.:/-]{1,200}$/.test(req.body.model))) throw images.failure(400, 'Enter an image model ID enabled at the configured image gateway');
if (workflow === 'learning_hub' && (typeof req.body.model !== 'string' || !MODEL_ID.test(req.body.model))) throw images.failure(400, 'Enter an image model ID enabled at the configured image gateway');
// Fallbacks are optional, and only sent when the caller means to change
// them: an absent field leaves the saved chain alone rather than clearing it.
let fallbacks = null;
if (req.body.fallbacks !== undefined) {
const list = Array.isArray(req.body.fallbacks) ? req.body.fallbacks
: String(req.body.fallbacks || '').split(',');
fallbacks = list.map(id => String(id).trim()).filter(Boolean);
for (const id of fallbacks) {
if (!MODEL_ID.test(id)) throw images.failure(400, 'Not a model ID: ' + id.slice(0, 60));
}
// The primary counts toward the cap, and naming it again would pay twice
// for the same refusal.
const primary = workflow === 'learning_hub' ? req.body.model
: String(await db.getSetting(workflow + '.image_model') || '');
fallbacks = fallbacks.filter((id, i) => id !== primary && fallbacks.indexOf(id) === i);
if (fallbacks.length > images.MAX_IMAGE_MODELS - 1) {
throw images.failure(400, 'At most ' + (images.MAX_IMAGE_MODELS - 1) +
' fallback models; each one is a paid retry');
}
}
const client = await db.pool.connect();
try {
await client.query('BEGIN');
for (const [key, value] of Object.entries(workflow === 'learning_hub' ? { image_model: req.body.model, image_budget: budget } : { image_budget: budget })) {
const changes = workflow === 'learning_hub'
? { image_model: req.body.model, image_budget: budget }
: { image_budget: budget };
if (fallbacks !== null) changes.fallback_image_models = fallbacks.join(',');
for (const [key, value] of Object.entries(changes)) {
await client.query('INSERT INTO app_settings(key,value) VALUES($1,$2) ON CONFLICT(key) DO UPDATE SET value=$2,updated_at=NOW()', [workflow + '.' + key, String(value)]);
}
await client.query('COMMIT');

View file

@ -1,6 +1,12 @@
const crypto = require('crypto');
const { DEFAULT_IMAGE_BEHAVIOR, imagePromptForCanvas } = require('./clinicalPrompts');
const storageUtil = require('./generatedImageStorage');
// fileLog rather than logger: logger requires the database at module load, and
// this module is exercised by tests that never open one.
const fileLog = require('./fileLog');
// Primary plus two. More is a bill, not a safety net: each hop is a paid request
// and a chain long enough to be worth capping is long enough to surprise someone.
const MAX_IMAGE_MODELS = 3;
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const failure = (statusCode, message) => Object.assign(new Error(message), { statusCode });
const workflows = ['clinical_assistant', 'learning_hub', 'my_resources'];
@ -30,14 +36,57 @@ function isDefiniteImageError(err) {
function isAbortImageError(err, signal) {
return !!(signal && signal.aborted) || !!(err && (err.name === 'AbortError' || err.code === 'ERR_CANCELED'));
}
// A model that declined on content grounds. Providers disagree about the status
// for this — some send 400, some 422 — and none of them send a machine-readable
// reason, so the message is what is left to read. Worth separating because it is
// the one 4xx where a different model genuinely may succeed: policy is a vendor
// decision, not a fact about the request.
var REFUSAL = /(safety|safety_?system|content[_ ]?polic|moderation|blocked|flagged|violat|not allowed|cannot generate|refus|prohibit|inappropriate|sensitive)/i;
function isContentRefusal(err) {
var status = err && err.response ? Number(err.response.status) : NaN;
if (!Number.isFinite(status) || status < 400 || status >= 500) return false;
var body = err && err.response && err.response.data;
var text = [err && err.message,
typeof body === 'string' ? body : body && JSON.stringify(body)].filter(Boolean).join(' ');
return REFUSAL.test(text);
}
// Why a failure is or is not worth handing to the next model. One function, so
// the reasoning is in one place and is reportable rather than implied by a
// boolean.
//
// retry the next model has a real chance: transient faults, a model the
// gateway does not have, and a refusal — another vendor's policy is
// not this one's
// stop nothing downstream can help: the caller went away, we are shutting
// down, the credentials are wrong (every model shares them), or the
// request itself is malformed or too large
//
// 408 is Request Timeout and 429 is Too Many Requests; both are the server
// saying "not now", not "not ever", which is why they never counted as definite.
function classifyImageFailure(err, signal, stopping) {
if (stopping) return { retry: false, reason: 'shutting down' };
if (isAbortImageError(err, signal)) return { retry: false, reason: 'the request was cancelled' };
var status = err && err.response ? Number(err.response.status) : NaN;
if (!Number.isFinite(status)) return { retry: true, reason: 'network or timeout' };
if (status === 401 || status === 403) {
// Same gateway, same credentials: the next model fails identically.
return { retry: false, reason: 'the gateway rejected our credentials' };
}
if (status === 413) return { retry: false, reason: 'the request is too large' };
if (status === 404) return { retry: true, reason: 'the gateway does not have that model' };
if (status === 408 || status === 429) return { retry: true, reason: 'the provider was busy' };
if (status >= 500) return { retry: true, reason: 'the provider failed' };
if (isContentRefusal(err)) return { retry: true, reason: 'the model declined the prompt' };
return { retry: false, reason: 'the provider rejected the request' };
}
function shouldRetryImageFallback(state) {
state = state || {};
if (state.workflow !== 'clinical_assistant') return false;
if (!state.fallback || state.fallback === state.jobModel) return false;
if (state.stopping) return false;
if (isAbortImageError(state.error, state.signal)) return false;
if (isDefiniteImageError(state.error)) return false;
return true;
return classifyImageFailure(state.error, state.signal, state.stopping).retry;
}
// Previews are generated once and stored beside the original, so a 56px tile
@ -156,24 +205,67 @@ function createImageService({ db, storage, generate = provider, encryption = req
await client.query('COMMIT'); return result.rows[0];
} catch (e) { await client.query('ROLLBACK').catch(() => {}); throw e; } finally { client.release(); }
}
async function generateWithFallback(job, prompt, signal) {
// The models to try, in order, for this workflow: the configured one first,
// then its fallbacks. Every workflow has this now — it used to be the Clinical
// Assistant alone, so a My Resources figure that failed simply had no picture,
// which is the case where a missing figure is most visible.
async function modelChain(workflow, primary) {
var chain = [primary];
try {
return await generate(job, prompt, signal);
} catch (err) {
var fallback = '';
if (job.workflow === 'clinical_assistant' && !stopping && !isAbortImageError(err, signal) && !isDefiniteImageError(err)) {
try {
const row = await db.query('SELECT value FROM app_settings WHERE key = $1', ['clinical_assistant.fallback_image_model']);
fallback = String(row.rows[0] && row.rows[0].value || '');
} catch (_) { fallback = ''; }
// The plural key is the current one; the old singular is still read so a
// deployment that configured a fallback before this keeps it without
// anyone re-entering it.
const row = await db.query(
'SELECT key, value FROM app_settings WHERE key = ANY($1::text[])',
[[workflow + '.fallback_image_models', workflow + '.fallback_image_model']]);
const byKey = {};
row.rows.forEach(function (r) { byKey[r.key] = String(r.value || ''); });
var configured = (byKey[workflow + '.fallback_image_models'] || '').trim()
|| (byKey[workflow + '.fallback_image_model'] || '').trim();
configured.split(',')
.map(function (id) { return id.trim(); })
.filter(Boolean)
.forEach(function (id) { if (chain.indexOf(id) === -1) chain.push(id); });
} catch (_) { /* no fallbacks configured is the normal case */ }
return chain.slice(0, MAX_IMAGE_MODELS);
}
// Walk the chain until one produces a picture. Each hop re-leases the job, so
// a chain cannot outlive its claim and let a second worker start the same paid
// work; if the lease has gone, the attempt stops there rather than paying
// again. The error that ends the chain is the one the caller sees, and every
// hop is recorded with the reason it moved on.
async function generateWithFallback(job, prompt, signal) {
const chain = await modelChain(job.workflow, job.model);
let lastError = null;
for (let attempt = 0; attempt < chain.length; attempt++) {
const model = chain[attempt];
if (attempt > 0) {
const lease = await db.query(
"UPDATE generated_image_jobs SET lease_until=NOW()+interval '3 minutes' WHERE id=$1 AND lease_token=$2 AND stage='generating' RETURNING id",
[job.id, job.lease_token]);
if (!lease.rows.length) throw lastError;
// The row records which model is actually being paid for, so a picture
// made by the third model is not attributed to the first.
await db.query('UPDATE generated_image_jobs SET model=$3,updated_at=NOW() WHERE id=$1 AND lease_token=$2',
[job.id, job.lease_token, model]).catch(function () {});
}
try {
return await generate(Object.assign({}, job, { model: model }), prompt, signal);
} catch (err) {
lastError = err;
const verdict = classifyImageFailure(err, signal, stopping);
const next = chain[attempt + 1];
fileLog.write(next && verdict.retry ? 'warn' : 'error',
'[generated-images] ' + model + ' failed: ' + verdict.reason +
(next && verdict.retry ? '; trying ' + next : '; giving up'),
{ jobId: job.id, workflow: job.workflow, attempt: attempt + 1, of: chain.length,
status: err && err.response ? err.response.status : null });
if (!verdict.retry || !next) throw err;
}
if (!shouldRetryImageFallback({ workflow: job.workflow, fallback: fallback, jobModel: job.model, stopping: stopping, error: err, signal: signal })) throw err;
console.warn('[generated-images] image generation failed; retrying once with the fallback image model', { jobId: job.id, model: job.model, fallbackModel: fallback });
const lease = await db.query("UPDATE generated_image_jobs SET lease_until=NOW()+interval '3 minutes' WHERE id=$1 AND lease_token=$2 AND stage='generating' RETURNING id",
[job.id, job.lease_token]);
if (!lease.rows.length) throw err;
return await generate(Object.assign({}, job, { model: fallback }), prompt, signal);
}
throw lastError;
}
async function tick() {
@ -303,4 +395,4 @@ function requestKey(body) {
}
return crypto.createHash('sha256').update(JSON.stringify(body)).digest('hex');
}
module.exports = { createImageService, service, budgetLimit, THUMB_WIDTHS, thumbWidth, args, imageContext, IMAGE_OUTPUT_RULE, publicJob, requestKey, UUID, failure, isDefiniteImageError, isAbortImageError, shouldRetryImageFallback };
module.exports = { createImageService, service, budgetLimit, THUMB_WIDTHS, MAX_IMAGE_MODELS, classifyImageFailure, isContentRefusal, thumbWidth, args, imageContext, IMAGE_OUTPUT_RULE, publicJob, requestKey, UUID, failure, isDefiniteImageError, isAbortImageError, shouldRetryImageFallback };

View file

@ -377,12 +377,15 @@ test('assistant config pending, rejected, HTTP failure and malformed payloads ne
});
});
test('single image-model dropdown keeps saved selection through discovery failures and saves both workflows', async t => {
test('image-model dropdowns keep saved selections through discovery failures and save every workflow', async t => {
const pending = [];
const ui = await browser(t, 'admin', url => {
if (url === '/api/admin/config') return json(assistantConfig());
if (url === '/api/models') throw Error('chat discovery offline');
if (url === '/api/admin/image-settings') return json({ success: true, workflows: { clinical_assistant: { model: 'saved-image', budget: 32000 }, learning_hub: { model: 'saved-image', budget: 32000 } } });
if (url === '/api/admin/image-settings') return json({ success: true, maxModels: 3, workflows: {
clinical_assistant: { model: 'saved-image', fallbacks: ['saved-backup'], budget: 32000 },
learning_hub: { model: 'saved-image', fallbacks: [], budget: 32000 },
my_resources: { model: '', fallbacks: [], budget: 32000 } } });
if (url.endsWith('/image-models/discover')) return new Promise((resolve, reject) => { pending.push({ resolve, reject }); });
});
await tick(); await tick();
@ -391,12 +394,23 @@ test('single image-model dropdown keeps saved selection through discovery failur
const select = ui.document.querySelector('#workflow-image-settings select');
assert.ok(select, 'image-model dropdown rendered');
assert.equal(select.value, 'saved-image', 'saved selection kept while discovery failed');
ui.document.querySelector('#workflow-image-settings form button').click(); await tick();
// The saved fallback survives a discovery failure the same way the primary
// does: it is seeded into the list rather than only offered from discovery.
const clinical = ui.document.querySelectorAll('#workflow-image-settings fieldset')[0];
assert.equal(clinical.querySelectorAll('select')[1].value, 'saved-backup', 'saved fallback kept');
ui.document.querySelector('#workflow-image-settings form button[type="submit"]').click();
// One await per workflow, so the queue needs draining more than once.
for (let i = 0; i < 6; i++) await tick();
const puts = () => ui.calls.filter(c => c.options.method === 'PUT' && c.url.includes('/api/admin/image-settings/'));
assert.equal(puts().length, 2);
for (const workflow of ['clinical_assistant', 'learning_hub']) {
assert.deepEqual(puts().find(c => c.url.endsWith(workflow)).body, { model: 'saved-image', budget: 32000 });
}
assert.equal(puts().length, 3, 'every workflow is saved, My Resources included');
assert.deepEqual(puts().find(c => c.url.endsWith('clinical_assistant')).body,
{ model: 'saved-image', budget: 32000, fallbacks: ['saved-backup'] });
assert.deepEqual(puts().find(c => c.url.endsWith('learning_hub')).body,
{ model: 'saved-image', budget: 32000, fallbacks: [] });
// My Resources chooses its model per request, so the form must not send one.
assert.deepEqual(puts().find(c => c.url.endsWith('my_resources')).body,
{ budget: 32000, fallbacks: [] });
assert.equal(select.value, 'saved-image', 'save never reverts the selection');
});

View file

@ -0,0 +1,100 @@
// An image model that fails should hand the job to the next one — for every
// feature, not only the Clinical Assistant, and for the failures where another
// model genuinely has a chance.
const test = require('node:test');
const assert = require('node:assert/strict');
const { classifyImageFailure, isContentRefusal, MAX_IMAGE_MODELS,
shouldRetryImageFallback } = require('../src/utils/generatedImages');
const httpError = (status, body) => Object.assign(new Error(typeof body === 'string' ? body : 'request failed'),
{ response: { status, data: body } });
test('the provider saying "not now" is always worth another model', () => {
// 408 is Request Timeout, 429 is Too Many Requests. Both are a server saying
// not now, never not ever.
for (const status of [408, 429, 500, 502, 503, 504]) {
assert.equal(classifyImageFailure(httpError(status), null, false).retry, true, 'status ' + status);
}
// No response at all: a socket hang-up or a DNS failure.
assert.equal(classifyImageFailure(new Error('socket hang up'), null, false).retry, true);
});
test('bad credentials stop the chain, because every model shares them', () => {
for (const status of [401, 403]) {
const verdict = classifyImageFailure(httpError(status), null, false);
assert.equal(verdict.retry, false, 'status ' + status);
assert.match(verdict.reason, /credentials/);
}
});
test('a model the gateway does not have moves on rather than failing the job', () => {
// This is a configuration mistake, and the next model is exactly the thing
// that rescues it.
const verdict = classifyImageFailure(httpError(404), null, false);
assert.equal(verdict.retry, true);
assert.match(verdict.reason, /does not have that model/);
});
test('a content refusal tries the next model — policy is a vendor decision', () => {
const refusals = [
httpError(400, { error: { message: 'Your request was rejected by our safety system' } }),
httpError(400, 'content_policy_violation'),
httpError(422, { message: 'The prompt was flagged as sensitive' }),
httpError(400, { error: 'This request was blocked by moderation' })
];
for (const err of refusals) {
assert.equal(isContentRefusal(err), true, err.message);
assert.equal(classifyImageFailure(err, null, false).retry, true);
}
});
test('a malformed request is not retried, because it is malformed everywhere', () => {
const verdict = classifyImageFailure(httpError(400, { error: 'size must be one of 1024x1024' }), null, false);
assert.equal(verdict.retry, false);
assert.match(verdict.reason, /rejected the request/);
assert.equal(classifyImageFailure(httpError(413), null, false).retry, false, 'too large everywhere too');
});
test('a cancelled request and a shutdown never start more paid work', () => {
assert.equal(classifyImageFailure(new Error('x'), { aborted: true }, false).retry, false);
assert.equal(classifyImageFailure(Object.assign(new Error('x'), { name: 'AbortError' }), null, false).retry, false);
assert.equal(classifyImageFailure(httpError(503), null, true).retry, false, 'shutting down');
});
test('every workflow gets fallbacks now, not the Clinical Assistant alone', () => {
// A My Resources figure that failed used to have no second chance at all,
// which is the case where a missing picture is most visible — a slide with a
// hole in it.
for (const workflow of ['clinical_assistant', 'my_resources', 'learning_hub']) {
assert.equal(shouldRetryImageFallback({
workflow, fallback: 'model-b', jobModel: 'model-a', error: httpError(503)
}), true, workflow);
}
});
test('a fallback identical to the model that just failed is not a fallback', () => {
assert.equal(shouldRetryImageFallback({
workflow: 'my_resources', fallback: 'model-a', jobModel: 'model-a', error: httpError(503)
}), false);
assert.equal(shouldRetryImageFallback({
workflow: 'my_resources', fallback: '', jobModel: 'model-a', error: httpError(503)
}), false);
});
test('the chain is capped, because every hop is a paid request', () => {
assert.equal(MAX_IMAGE_MODELS, 3);
const lib = require('fs').readFileSync(require('path').join(__dirname, '..', 'src/utils/generatedImages.js'), 'utf8');
assert.match(lib, /chain\.slice\(0, MAX_IMAGE_MODELS\)/);
// Duplicates collapse: naming the primary again as a fallback would pay twice
// for the same refusal.
assert.match(lib, /if \(chain\.indexOf\(id\) === -1\) chain\.push\(id\)/);
});
test('each hop re-leases, and records which model is actually being paid for', () => {
const lib = require('fs').readFileSync(require('path').join(__dirname, '..', 'src/utils/generatedImages.js'), 'utf8');
const fn = lib.slice(lib.indexOf('async function generateWithFallback'));
assert.match(fn.slice(0, 1600), /lease_until=NOW\(\)\+interval '3 minutes'/,
'a chain must not outlive its claim and let a second worker repeat the work');
assert.match(fn.slice(0, 1600), /UPDATE generated_image_jobs SET model=\$3/,
'a picture made by the third model must not be attributed to the first');
});

View file

@ -82,7 +82,7 @@ test('the grid asks for the stored preview, not the original', () => {
test('the caption is a model-written prompt and never reaches the page as HTML', () => {
const ui = read('public/js/myResources.js');
const tile = ui.slice(ui.indexOf('function imageTile'), ui.indexOf('function openImage'));
assert.match(tile, /text\.textContent = image\.prompt/);
assert.match(tile, /prompt\.textContent = image\.prompt/);
assert.doesNotMatch(tile, /innerHTML/);
});
@ -91,3 +91,53 @@ test('the images tab loads on first open, not on page load', () => {
const ui = read('public/js/myResources.js');
assert.match(ui, /if \(!docs && !imagesLoaded\) loadImages\(true\)/);
});
test('the library is in My Resources and never in the Clinical Assistant', () => {
// Assistant images appear here — that was the point — but the assistant page
// does not grow a gallery of its own.
const ui = require('fs').readFileSync(require('path').join(__dirname, '..', 'public/js/myResources.js'), 'utf8');
assert.match(ui, /function loadImages/);
const assistantFiles = require('fs').readdirSync(require('path').join(__dirname, '..', 'public/js/assistant'));
for (const file of assistantFiles) {
const src = require('fs').readFileSync(require('path').join(__dirname, '..', 'public/js/assistant', file), 'utf8');
assert.doesNotMatch(src, /\/api\/generated-images\?/, file + ' must not list the library');
}
const component = require('fs').readFileSync(require('path').join(__dirname, '..', 'public/components/my-resources.html'), 'utf8');
assert.match(component, /id="mr-images-panel"/);
});
test('downloading goes through the authenticated blob, not a bare link', () => {
// A mobile client's session is a bearer token an <a> cannot send, and the
// asset is served no-store on purpose.
const ui = require('fs').readFileSync(require('path').join(__dirname, '..', 'public/js/myResources.js'), 'utf8');
const save = ui.slice(ui.indexOf('function saveImage'), ui.indexOf('function deleteImage'));
assert.match(save, /m\.privateImageBlob\(image\.imageUrl\)/);
assert.match(save, /URL\.revokeObjectURL/, 'the object URL is released');
assert.doesNotMatch(save, /href = image\.downloadUrl/);
});
test('the tile actions stay reachable without a pointer', () => {
// Hiding delete behind :hover puts it out of reach on touch and keyboard.
const css = require('fs').readFileSync(require('path').join(__dirname, '..', 'public/css/styles.css'), 'utf8');
const block = css.slice(css.indexOf('.img-tile-actions'));
assert.match(block.slice(0, 400), /@media \(hover:hover\)/, 'only hidden where hover exists');
assert.match(css, /\.img-tile:focus-within \.img-tile-actions/);
assert.match(css, /prefers-reduced-motion/);
});
test('the picture itself is a button, so opening it needs no invented key handling', () => {
const ui = require('fs').readFileSync(require('path').join(__dirname, '..', 'public/js/myResources.js'), 'utf8');
const tile = ui.slice(ui.indexOf('function imageTile'), ui.indexOf('function openImage'));
assert.match(tile, /frame\.type = 'button'/);
assert.match(tile, /loading = 'lazy'/, 'a grid of thumbnails should not all fetch at once');
});
test('the lightbox closes on Escape and on the backdrop, and restores focus', () => {
const ui = require('fs').readFileSync(require('path').join(__dirname, '..', 'public/js/myResources.js'), 'utf8');
const box = ui.slice(ui.indexOf('function openImage'), ui.indexOf('function saveImage'));
assert.match(box, /aria-modal/);
assert.match(box, /e\.key === 'Escape'/);
assert.match(box, /if \(e\.target === overlay\) dismiss\(\)/, 'a click on the picture must not close it');
assert.match(box, /lastFocus\.focus\(\)/);
assert.match(box, /removeEventListener\('keydown', onKey\)/, 'no listener left behind');
});