haiku.rag/tests/store
Yiorgis Gozadinos 8e93b639bc
Migrate existing databases to the full index set
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.

`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:

- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
  the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
  be deliberate, serving different query shapes, so an index this function did
  not declare survives even on a column it does. The one thing it overwrites is
  an index at LanceDB's default name, `{column}_idx`, which is the name it
  creates itself.
- A column already carrying the declared type is skipped, so a database with the
  full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.

It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.

The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.

`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
2026-08-17 16:34:58 +03:00
..
__init__.py Split meta document attributes into a document_meta table 2026-06-12 10:17:17 +03:00
legacy_documents.py Serialize and roll back the multi-table document delete 2026-06-12 10:17:18 +03:00
test_citation.py Search results and citations carry document metadata 2026-07-23 10:23:48 +03:00
test_compression.py Collapse docling compression to a single function 2026-06-24 10:43:27 +03:00
test_database_info.py Extract gather_database_info shared by the info command 2026-06-09 11:09:28 +03:00
test_document_items.py Reach and enforce 100% coverage 2026-07-26 20:11:02 +03:00
test_document_meta_split.py Rename document_meta identity column document_id to id 2026-07-08 16:32:55 +03:00
test_engine.py Similar fix for haiku.rag.ingester, and formatting 2026-08-12 15:20:11 +01:00
test_indexes.py Migrate existing databases to the full index set 2026-08-17 16:34:58 +03:00
test_migrations.py Stop eagerly importing Store from haiku.rag.store 2026-08-11 17:26:51 +01:00
test_read_only.py Stop eagerly importing Store from haiku.rag.store 2026-08-11 17:26:51 +01:00
test_restore.py Stop eagerly importing Store from haiku.rag.store 2026-08-11 17:26:51 +01:00
test_tags.py Stop eagerly importing Store from haiku.rag.store 2026-08-11 17:26:51 +01:00
test_v0_40_0_migration.py Similar fix for haiku.rag.ingester, and formatting 2026-08-12 15:20:11 +01:00
test_v0_45_0_migration.py Stop eagerly importing Store from haiku.rag.store 2026-08-11 17:26:51 +01:00
test_v0_48_0_migration.py Collapse docling compression to a single function 2026-06-24 10:43:27 +03:00
test_v0_50_0_migration.py Reach and enforce 100% coverage 2026-07-26 20:11:02 +03:00
test_v0_58_0_migration.py Load document blobs only when asked 2026-08-06 13:04:51 +03:00
test_v0_64_0_migration.py Rename document_meta identity column document_id to id 2026-07-08 16:32:55 +03:00
test_v0_75_0_migration.py Migrate existing databases to the full index set 2026-08-17 16:34:58 +03:00