haiku.rag/tests
Yiorgis Gozadinos 8e93b639bc
Migrate existing databases to the full index set
Adds the 0.75.0 upgrade, which brings a pre-existing database up to the index
set `_init_tables` now creates. It rewrites no table data, so unlike the earlier
data migrations its cost is the index builds alone, each of which reads the
column it indexes.

`ensure_indexes` ensures an index of the declared *type* covers each declared
column, rather than checking that the column is indexed at all. The distinction
is what makes it safe to run against a database of unknown provenance:

- A wrong-typed index no longer satisfies the check. A BTree on `label` covers
  the column while losing the low-cardinality equality lookup the Bitmap is for.
- Nothing is dropped or converted away from. Two index types over one column can
  be deliberate, serving different query shapes, so an index this function did
  not declare survives even on a column it does. The one thing it overwrites is
  an index at LanceDB's default name, `{column}_idx`, which is the name it
  creates itself.
- A column already carrying the declared type is skipped, so a database with the
  full set migrates instantly rather than re-sorting every indexed column.
- Undeclared columns are untouched, so a vector index on `chunks` survives.

It returns the columns it acted on, because a change is not always visible from
outside: adding a Bitmap beside an existing BTree leaves the column indexed
before and after.

The version bump to 0.75.0 is required, not incidental: `_set_initial_version`
stamps a new database with the installed package version, so a migration
numbered above it would be pending the moment the database was created.

`test_client_update_document_replaces_rows_with_bounded_versions` turns
auto_vacuum off. Indexing `documents` means a background vacuum now has an index
to maintain on that table, so `optimize()` writes a version where it previously
had nothing to do, and it landed inside the window the test measures. The
document update itself is still one version, so the bound stays exact.
2026-08-17 16:34:58 +03:00
..
capabilities Judge each capability's evidence by its own record 2026-08-13 15:57:33 +03:00
cassettes Recover from worker death and deny metadata writes 2026-07-28 19:11:19 +03:00
chat Give the capsule to one return, and the run its own copy of state 2026-08-13 13:00:02 +03:00
data Remove repliqa from test setup 2026-06-01 10:40:52 +03:00
docker Add a docker-compose for integration test services 2026-06-04 11:13:16 +03:00
ingester Move ingester telemetry setup behind argument parsing 2026-08-13 11:37:36 +03:00
sandbox Point the limit comments at the right limits 2026-07-28 19:11:37 +03:00
store Migrate existing databases to the full index set 2026-08-17 16:34:58 +03:00
tools Name the search result each page image belongs to 2026-08-13 13:00:01 +03:00
__init__.py
conftest.py Consolidate duplicated client-side tests 2026-07-26 13:30:59 +03:00
json_body_serializer.py Keep cross-encoder rerank scores apart when they saturate 2026-08-07 14:07:53 +03:00
services.py Add a docker-compose for integration test services 2026-06-04 11:13:16 +03:00
test_ask_images.py Accept images on ask/analyze 2026-07-25 10:23:55 +03:00
test_cascade_delete.py Skip embedding validation on document deletion 2026-07-22 10:48:00 +03:00
test_chunk.py Fix defects found reviewing the coverage work 2026-07-27 10:44:32 +03:00
test_chunker.py Fix defects found reviewing the coverage work 2026-07-27 10:44:32 +03:00
test_cli.py Move ingester telemetry setup behind argument parsing 2026-08-13 11:37:36 +03:00
test_client.py Migrate existing databases to the full index set 2026-08-17 16:34:58 +03:00
test_client_analyze.py replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
test_config.py Cover the lazy config init and restore the uv cache in CI 2026-07-27 13:35:50 +03:00
test_context.py Fix defects found reviewing the coverage work 2026-07-27 10:44:32 +03:00
test_context_enhancement.py Shrink the docling-serve context-expansion cassette 2026-06-29 15:15:26 +03:00
test_converters.py Remove the wix evaluation dataset 2026-08-14 14:46:51 +03:00
test_database_autocreate.py Test vacuum against real table state 2026-07-27 12:11:01 +03:00
test_docling_serve_client.py Cover MCP, ingester and converter error paths 2026-07-26 19:14:26 +03:00
test_doctor.py Reach and enforce 100% coverage 2026-07-26 20:11:02 +03:00
test_document.py Load document blobs only when asked 2026-08-06 13:04:51 +03:00
test_download_models.py Reach and enforce 100% coverage 2026-07-26 20:11:02 +03:00
test_duplicate_ingestion.py Re-check URI under the write lock to prevent duplicate ingestion 2026-07-08 11:26:45 +03:00
test_embedder.py Pool HTTP clients for vLLM embedding and vLLM/Jina reranking 2026-07-09 10:32:44 +03:00
test_embedder_config.py Fix defects found reviewing the coverage work 2026-07-27 10:44:32 +03:00
test_filter.py Rename document_meta identity column document_id to id 2026-07-08 16:32:55 +03:00
test_info.py Harden history against tag annotation failures; document tag restore 2026-07-16 13:11:34 +03:00
test_inspector.py Surface database-open errors in the TUIs 2026-07-25 11:27:29 +03:00
test_lancedb_connection.py Fix defects found reviewing the coverage work 2026-07-27 10:44:32 +03:00
test_logging.py Additional tests 2026-02-13 18:08:46 +02:00
test_mcp.py Cover MCP, ingester and converter error paths 2026-07-26 19:14:26 +03:00
test_pdf_attachments.py compare off-loop work against actual event-loop thread, fix ty 2026-06-22 10:17:02 +03:00
test_pdf_concurrency.py Serialize all in-process pdfium access under a shared lock 2026-06-14 10:50:42 +03:00
test_pdf_split.py Cover MCP, ingester and converter error paths 2026-07-26 19:14:26 +03:00
test_picture_description.py cover picture-description provider and rebuild edge cases 2026-05-05 14:20:37 +03:00
test_picture_in_context.py Name the search result each page image belongs to 2026-08-13 13:00:01 +03:00
test_processing.py Normalize pragma comments to the codebase's single-line form 2026-07-26 20:18:19 +03:00
test_reader.py async versions for convert_file() convert_text() 2025-11-26 17:46:25 +02:00
test_rebuild.py Stop loading page rasters on the title and update paths 2026-08-06 13:04:51 +03:00
test_reranker.py Keep cross-encoder rerank scores apart when they saturate 2026-08-07 14:07:53 +03:00
test_s3_integration.py Clean up S3 test data per-test to keep SeaweedFS writable 2026-06-26 16:41:20 +03:00
test_s3_source.py canonical metadata keys: source_revision + content_type, bump to 0.50.0 2026-05-26 11:43:30 +03:00
test_search.py Cover get_model provider branches and visualize_chunk edge paths 2026-07-26 19:23:54 +03:00
test_settings.py Cover store engine and repository paths 2026-07-26 19:36:37 +03:00
test_telemetry.py Route eval scripts through the shared telemetry configure 2026-07-10 11:38:33 +03:00
test_title_generation.py Reach and enforce 100% coverage 2026-07-26 20:11:02 +03:00
test_utils.py Map thinking onto Pydantic AI's unified setting 2026-07-27 18:26:42 +03:00
test_vacuum_debounce.py Serialize tag operations, vacuum, and metadata refresh with writes 2026-07-16 12:32:57 +03:00
test_versioning.py Throttle background auto-vacuum to at most once per 5 minutes 2026-06-12 10:17:18 +03:00