From c0cf612d6813e4f78bfb1df7a8b807a22cb8fef8 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Tue, 1 Sep 2026 15:18:29 +0300 Subject: [PATCH] Document vector index state and measured cost Benchmarks state that every published number is exact brute-force kNN; no benchmark database carries a vector index. The Vector Indexing section gains the measured with/without IVF_PQ comparison (hotpotqa, orb_multimodal_nemotron, frames: free to ~121k chunks, 0.0044 MAP at 426k, ~30 s / 4 GB build) and corrects the re-indexing story: optimize() folds new chunks into the index as delta parts via auto_vacuum, so a rebuild is about retraining centroids, not covering new rows. --- CHANGELOG.md | 5 +++++ docs/benchmarks.md | 2 ++ docs/configuration/storage.md | 18 ++++++++++++++---- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d514f471..1309ffc7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ ## [Unreleased] +### Documentation + +- `docs/configuration/storage.md` "Vector Indexing" carries the measured with/without IVF_PQ retrieval comparison; `docs/benchmarks.md` states the published numbers are measured without a vector index. +- Re-indexing note corrected: `optimize()` adds new chunks to an existing vector index, a rebuild retrains centroids. + ## [0.80.0] - 2026-08-31 ### Changed diff --git a/docs/benchmarks.md b/docs/benchmarks.md index fc69b6aa..66b1b10d 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -6,6 +6,8 @@ We evaluate `haiku.rag` on a small set of datasets that exercise different parts Numbers below were measured on a recent `haiku.rag` version. Most rows were judged by `Qwen3.6-35B-A3B-NVFP4`; the `Qwen3.8-27B` rows were judged by the currently pinned `qwen3.8`, as their footnote states. Rows are not re-judged when the pinned judge changes, so compare rows judged by the same judge and treat cross-judge differences as unmeasured. +No benchmark database carries a vector index, so every number below reflects exact brute-force kNN rather than approximate search. A vector index is never built automatically. `haiku-rag create-index` builds one, and `haiku-rag doctor` reports whether a database has it. For the measured effect of indexing on retrieval, see [Vector Indexing](configuration/storage.md#vector-indexing). + ### OpenRAG Bench (ORB) [OpenRAG Bench](https://huggingface.co/datasets/vectara/open_ragbench) contains ArXiv research papers with multimodal question-answering pairs. Queries include both text-based and image-based questions, testing retrieval and reasoning over visual content like figures, charts, and diagrams. Each query maps to one relevant document. diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md index a94b2b3c..02bfc02c 100644 --- a/docs/configuration/storage.md +++ b/docs/configuration/storage.md @@ -317,6 +317,16 @@ For search behavior settings (`limit`, `max_context_chars`), see [Search and Que !!! note Vector indexes are only necessary for large datasets with over 100,000 chunks. For smaller datasets, LanceDB's brute-force kNN search provides exact results with good performance. Only create an index if you notice search performance degradation on large datasets. +Retrieval MAP with and without an index, measured on copies of the benchmark databases with no reranker: + +| Dataset | Chunks | Dim | Exact | Indexed | Delta | Build | Peak RSS | +|---------|-------:|----:|------:|--------:|------:|------:|---------:| +| `hotpotqa` | 70,527 | 2560 | 0.6978 | 0.6979 | +0.0001 | 29.3 s | 3.19 GB | +| `orb_multimodal_nemotron` | 121,168 | 2048 | 0.9799 | 0.9800 | +0.0001 | 25.8 s | 3.38 GB | +| `frames` | 425,940 | 2560 | 0.5431 | 0.5387 | -0.0044 | 34.1 s | 4.02 GB | + +An index costs no accuracy at 70k and 121k chunks and 0.0044 MAP at 426k. A larger corpus holds more IVF partitions, so the default number of probes covers a smaller fraction of the space, and `vector_refine_factor` can only re-score what those probes returned. Build cost is near-flat in row count because training samples the data rather than scanning it, and vector dimension drives it more than corpus size. + **Index creation:** Vector indexes are **not created automatically** during document ingestion to avoid slowing down the process. After you've added documents (at least 256 chunks required), create the index manually: @@ -332,12 +342,12 @@ This command: **Re-indexing:** -Indexes are not automatically updated when you add new documents. After adding a significant amount of new data: +New chunks reach the index without a rebuild. `optimize()`, which runs after writes while `auto_vacuum` is on, adds them as a delta part. Between a write and the next optimize, LanceDB serves ANN over the indexed rows and a brute-force scan over the remainder, then combines the results. + +A rebuild retrains the centroids, which are fitted once at build time and never recomputed. As a corpus grows past the distribution it was trained on the partitioning fits it less well, and delta parts accumulate. Rebuild after substantial growth: ```bash -haiku-rag create-index # Rebuilds the index with all data +haiku-rag create-index ``` -Searches still work with stale indexes - LanceDB uses the index for old data (fast ANN) and brute-force kNN for new unindexed rows, then combines the results. However, performance degrades as more unindexed data accumulates. - For datasets with fewer than 256 chunks, searches use brute-force kNN scans (exact nearest neighbors, 100% recall) which work well for small datasets but don't scale beyond a few hundred thousand vectors.