Sixty-three comments said what the next statement already said: # Connect to
LanceDB above connect_lancedb, # Path object above isinstance(source, Path),
# Get page numbers from provenance above the prov loop, # Clear and populate
results above list_view.clear(). They cost a read and carry nothing.
The line is whether a comment restates one statement or labels a phase. Phase
labels stay: the migrations keep # Create staging table with new schema and
# Copy from staging to final table in batches, each heading ten lines of a
long procedure. So do comments carrying a fact the code cannot: the
merge_insert update-only note on document_meta, why the poller builds sources
eagerly, why create_document_from_source returns a list for directories, that
indexes need training data, the field-group markers in the config models, and
the file:// URL-encoding note in create_document_from_source.
capabilities/ is untouched. Its docstrings sit next to prompt surface, and
changing them needs an eval to back it.
The cassette-recording docs were wrong three ways. They named
tests/test_qa.py::test_qa_anthropic, which no longer exists; they targeted
whole modules, so a rewrite would re-record cassettes for services the
recorder is not running; and they used COHERE_API_KEY where the SDK reads
CO_API_KEY. docs/development.md now names exact tests with -n0, and the keyed
example is test_cohere_reranker, which owns the one cassette recording
api.cohere.com.
Collapsing the caption text into `get_pictures_grouped` served the enrichment
path, which uses it, but the multimodal reranker discards the second return value
while still paying to read the column. That is the widest fan-out in the codebase,
`limit * 10` candidates, and it previously projected self_ref and picture_data
alone.
`with_text` is opt-in and off by default, so the cheap projection is what a caller
gets unless it asks for more. The reranker test asserts the projection as well as
the query count, since a count alone would not notice the column coming back.
Splitting the assembly into a passthrough pass and an expandable pass reordered
equal-scored results: the score sort that follows is stable, so the order results
arrive in is the tiebreak. Results are assembled in document_groups order again,
after the batched fetch rather than around it.
Also ports the caption negative cases the removed single-document test carried: a
table's caption and an ordinary text reference map to no picture.
`_attach_picture_data` fetched picture bytes once per document, over the
`limit * 10` candidates reranking asks for, so it was the per-document fetch with
the most candidates behind it. It now issues one query however many documents the
candidates span: one for ten documents, as for one.
Removes `get_text_for_refs`, whose only caller now gets the text back with the
bytes from `get_pictures_grouped`.
`test_client_search_include_images_false_skips_lookup` returned no search
results, so asserting the picture accessor went uncalled held whatever the code
did. It now returns a picture-carrying result, making "did not fetch" the
assertion rather than "had nothing to fetch".
`expand_with_items` fetched its own inputs per document: one query to resolve
refs to positions, one for the window of items around them. A result set spanning
N documents cost 2N queries, which was 10 of the 18 measured for a limit=5 search
on a remote object-store corpus.
`expand_context` now does both fetches once for every document it is expanding,
and `expand_with_items` takes the positions and items it needs. Two queries for
one document, and two for five.
Each document keeps its own inclusive window in `get_items_in_ranges`. Positions
repeat across documents, so a shared range would splice one document's items into
another's context.
`_populate_image_data` ran its stages once per result document, so a result set
spanning N documents cost 4N `document_items` queries. Measured on a remote
object-store corpus, a limit=5 search with expansion was 18 queries, 16 of them
against `document_items`.
The stages now run once each across every document, and flat in document count:
two queries for the dependent caption-to-picture mapping when results ranked on a
caption, one for the picture bytes. Two queries for a picture-ref result set,
three at most.
Picture text comes back with the bytes rather than from a second query, since it
is on the same rows.
Predicates are per document, `(document_id = 'a' AND self_ref IN (…)) OR (…)`,
rather than `self_ref IN (union)`. self_ref and position values repeat across
documents, so a union predicate would return other documents' rows: for
picture_data that fetches blobs nobody asked for, and it can hand one document
another document's picture.
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
