engine.py held four unrelated things: what the tables are, how to open a
connection, how to read a database's state, and the Store that coordinates
writes. At 1240 lines the Store's own concerns — locks, migrations, vacuum,
tags — were hard to find among them.
Table records, Arrow schemas, index_specs, ensure_indexes, REQUIRED_TABLES
and query_to_pydantic move to store/schema.py, which imports nothing from
haiku.rag: it describes the tables and never opens or mutates one.
gather_database_info, get_database_stats, DatabaseInfo and its result models
move to store/info.py. Nothing in Store calls them — they are read paths for
the CLI, doctor, inspector and ingester API — so info depends on engine and
not the reverse.
engine.py keeps the Store, ConnectionMode, connect_lancedb, the tag helpers
and the restore-order and retention constants. No re-exports: importers
point at the new modules.
test_app_info_uses_connect_lancedb_for_remote patched
haiku.rag.store.engine.connect_lancedb; gather_database_info now binds that
name in info.py, so the patch targets where the call is looked up.
Four call sites repeated lock, snapshot, try/except, restore. Each used
restore_table_versions, which restores in _tables() order — documents
first, contradicting RESTORE_TABLE_ORDER — and each caught Exception, so
a cancellation mid-write skipped rollback and left the earlier table
writes committed.
Store.write_transaction() holds the lock, snapshots under it, and rolls
back through _rollback_to_snapshot: RESTORE_TABLE_ORDER, shielded from
cancellation, absorbed cancellation re-delivered, rollback failure raised
with the original as cause. The two single-table update_meta sites keep
the bare lock.
The batch documents write moves inside the guarded body; it was outside
the try, so a failure there was never rolled back. Auto-vacuum is
scheduled after the transaction rather than inside it.
restore_table_versions is removed; those four sites were its only callers.
Collapsing the caption text into `get_pictures_grouped` served the enrichment
path, which uses it, but the multimodal reranker discards the second return value
while still paying to read the column. That is the widest fan-out in the codebase,
`limit * 10` candidates, and it previously projected self_ref and picture_data
alone.
`with_text` is opt-in and off by default, so the cheap projection is what a caller
gets unless it asks for more. The reranker test asserts the projection as well as
the query count, since a count alone would not notice the column coming back.
Splitting the assembly into a passthrough pass and an expandable pass reordered
equal-scored results: the score sort that follows is stable, so the order results
arrive in is the tiebreak. Results are assembled in document_groups order again,
after the batched fetch rather than around it.
Also ports the caption negative cases the removed single-document test carried: a
table's caption and an ordinary text reference map to no picture.
`resolve_refs`, `get_items_in_range`, `get_caption_picture_refs` and
`get_all_items_grouped` have no callers left: the grouped equivalents serve every
path that used them. `get_all_items_grouped` had none even before this branch.
Tests whose subject was a removed method go with it. Tests that only used one to
fetch a fixture now use the grouped call, so what they assert is unchanged.
`_attach_picture_data` fetched picture bytes once per document, over the
`limit * 10` candidates reranking asks for, so it was the per-document fetch with
the most candidates behind it. It now issues one query however many documents the
candidates span: one for ten documents, as for one.
Removes `get_text_for_refs`, whose only caller now gets the text back with the
bytes from `get_pictures_grouped`.
`test_client_search_include_images_false_skips_lookup` returned no search
results, so asserting the picture accessor went uncalled held whatever the code
did. It now returns a picture-carrying result, making "did not fetch" the
assertion rather than "had nothing to fetch".
`_populate_image_data` ran its stages once per result document, so a result set
spanning N documents cost 4N `document_items` queries. Measured on a remote
object-store corpus, a limit=5 search with expansion was 18 queries, 16 of them
against `document_items`.
The stages now run once each across every document, and flat in document count:
two queries for the dependent caption-to-picture mapping when results ranked on a
caption, one for the picture bytes. Two queries for a picture-ref result set,
three at most.
Picture text comes back with the bytes rather than from a second query, since it
is on the same rows.
Predicates are per document, `(document_id = 'a' AND self_ref IN (…)) OR (…)`,
rather than `self_ref IN (union)`. self_ref and position values repeat across
documents, so a union predicate would return other documents' rows: for
picture_data that fetches blobs nobody asked for, and it can hand one document
another document's picture.
Opening a database ran `list_tables` three times, opened the settings table
twice, and read and parsed the same settings row three times: once for the stored
vector dimension, once for the version behind the migration check, and once for
config validation. On object storage each of those is a round trip.
`_initialize` now reads both once and threads them down. `_init_tables` and
`_check_migrations` take what it read instead of fetching their own copy, and
`validate_config_compatibility` accepts the settings it should compare against,
still reading for itself when called directly.
Passing the pre-init read to validation is equivalent: nothing between the read
and the validation rewrites `embeddings`, which is all it compares.
The settings read no longer swallows every exception. It did before, when the
only consequence was falling back to the configured vector dimension; now the
same empty result feeds the migration check, where it would read as version
0.0.0 and declare every migration pending. Only decode failures are tolerated,
and a decoded non-object normalizes to {} rather than reaching callers that
expect a mapping.
Also drops two things that were stale rather than merely verbose: the CLI docs
note for 0.75.0, which was the only release-tagged note in the docs tree while
the CHANGELOG already records that existing databases need `haiku-rag migrate`;
and "(created or corrected)" from the migration log line, left over from the
earlier behaviour that replaced wrong-typed indexes.
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.
`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:
- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
be deliberate, serving different query shapes, so an index this function did
not declare survives even on a column it does. The one thing it overwrites is
an index at LanceDB's default name, `{column}_idx`, which is the name it
creates itself.
- A column already carrying the declared type is skipped, so a database with the
full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.
It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.
The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.
`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
`_init_tables` left `chunks.id`, `chunks.document_id` and `documents.id`
unindexed, so those lookups scanned the column. On object storage that is
network I/O per query, on paths that run per document: citation lookup,
delete-by-document, the re-ingest merge, and every dedup probe.
Declare the index set per table in `index_specs()` and apply it through
`ensure_indexes()`, which skips a column only when it is already indexed with
the declared type. Both halves of that are load-bearing. Skipping is required
because `create_index(replace=True)` rebuilds an identical index, writing a new
index and a new table version and orphaning the old files until the next vacuum.
Comparing the type is required because column coverage alone would let a
wrong-typed index stand, and a BTree on `label` silently loses the
low-cardinality equality lookup the Bitmap is there for.
Columns not declared for a table are left alone, so an externally created index
such as a vector index on `chunks` survives.
`_init_tables`, `recreate_embeddings_table`, `ChunkRepository.delete_all` and
`DocumentRepository.delete_all` now all route through it instead of repeating
their own subsets.
Also recreate `document_items` from `get_document_items_arrow_schema()` in
`DocumentRepository.delete_all`, which was using the LanceModel and so returned
`picture_data` as 32-bit `binary`.
Existing databases are unchanged; the migration follows separately.
Import Store from haiku.rag.store.engine directly instead of re-exporting it
through the package __init__, and defer HaikuRAGApp's import in cli.py behind
TYPE_CHECKING/lazy imports, avoiding an eager import at CLI startup.
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.
Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.
Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.
Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.
Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.
Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
Add vector-index creation tests including the warned failure, chunk
repository get_by_id, list_all pagination, blank-query and precomputed-vector
search, the unknown-score-column guard, settings row recreation, and
replace_for_document with no items.
