haiku.rag/docs
Yiorgis Gozadinos 8e93b639bc
Migrate existing databases to the full index set
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.

`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:

- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
  the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
  be deliberate, serving different query shapes, so an index this function did
  not declare survives even on a column it does. The one thing it overwrites is
  an index at LanceDB's default name, `{column}_idx`, which is the name it
  creates itself.
- A column already carrying the declared type is skipped, so a database with the
  full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.

It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.

The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.

`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
2026-08-17 16:34:58 +03:00
..
capabilities Refuse to compact evidence the host kept no record of 2026-08-13 15:46:23 +03:00
configuration Pin the eval judge sampling and standardise on Qwen3-Reranker 2026-08-06 13:17:58 +03:00
img Add ingester screenshot 2026-05-27 15:53:01 +03:00
stylesheets Landing page hero with TUI screen recording 2026-05-20 16:56:25 +03:00
apps.md Bind app example backend to loopback and document its lack of auth 2026-07-08 11:37:03 +03:00
benchmarks.md Publish the paired statistics behind the MTRAG compaction claims 2026-08-17 10:56:27 +03:00
changelog.md Improve documentation 2025-11-28 13:10:03 +02:00
chat.md Update caps documentation 2026-08-13 14:08:14 +03:00
cli.md Migrate existing databases to the full index set 2026-08-17 16:34:58 +03:00
custom-pipelines.md Always use the Store-owned embedder 2026-05-29 11:36:53 +03:00
development.md Docs 2026-04-24 14:42:52 +03:00
index.md Landing page hero with TUI screen recording 2026-05-20 16:56:25 +03:00
ingester.md Classify obstore config errors as permanent 2026-08-06 12:53:45 +03:00
installation.md Fix packaging and settings facts in the docs 2026-07-28 18:22:05 +03:00
mcp.md Accept images on CLI ask/analyze and MCP tools 2026-07-25 10:35:25 +03:00
overview.md Register the optional capabilities where agents are composed 2026-08-13 15:04:05 +03:00
python.md Load document blobs only when asked 2026-08-06 13:04:51 +03:00
remote-processing.md Store docling-serve chunk bodies via raw_text 2026-06-23 16:35:21 +03:00
tools.md replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
tuning.md Remove the wix evaluation dataset 2026-08-14 14:46:51 +03:00
tutorial.md replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00