From 5477446c5dfd019df77736c1b1d89b57cd3b8463 Mon Sep 17 00:00:00 2001
From: Yiorgis Gozadinos <ggozadinos@gmail.com>
Date: Thu, 20 Nov 2025 14:58:50 +0200
Subject: [PATCH] Update docs

---
 docs/cli.md           | 28 ++++++++++++++++++++++++++--
 docs/configuration.md |  6 +++---
 2 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/docs/cli.md b/docs/cli.md
index 53d629b0..ec7a164b 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -215,14 +215,38 @@ Shows:
 - path to the database
 - stored haiku.rag version (from settings)
 - embeddings provider/model and vector dimension
-- number of documents
+- number of documents and chunks (with storage sizes)
+- vector index status (exists/not created, indexed/unindexed chunks)
 - table versions per table (documents, chunks)
 
-At the end, a separate “Versions” section lists runtime package versions:
+At the end, a separate "Versions" section lists runtime package versions:
 - haiku.rag
 - lancedb
 - docling
 
+### Create Vector Index
+
+Create a vector index on the chunks table for fast approximate nearest neighbor search:
+
+```bash
+haiku-rag create-index [--db /path/to/your.lancedb]
+```
+
+**Requirements:**
+- Minimum 256 chunks required for index creation (LanceDB training data requirement)
+- Creates an IVF_PQ index using the configured `search.vector_index_metric` (cosine/l2/dot)
+
+**When to use:**
+- After ingesting documents (indexes are not created automatically)
+- After adding significant new data to rebuild the index
+- Use `haiku-rag info` to check index status and see how many chunks are indexed/unindexed
+
+**Search behavior:**
+- Without index: Brute-force kNN search (exact nearest neighbors, slower for large datasets)
+- With index: Fast ANN (approximate nearest neighbors) using IVF_PQ
+- With stale index: LanceDB combines indexed results (fast ANN) + brute-force kNN on unindexed rows
+- Performance degrades as more unindexed data accumulates
+
 ### Vacuum (Optimize and Cleanup)
 
 Reduce disk usage by optimizing and pruning old table versions across all tables:
diff --git a/docs/configuration.md b/docs/configuration.md
index d7463d0e..6849125c 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -751,7 +751,7 @@ haiku-rag create-index
 
 This command:
 - Checks if you have enough data (minimum 256 chunks)
-- Creates an IVF_PQ index for fast approximate nearest neighbor search
+- Creates an IVF_PQ index for fast approximate nearest neighbor (ANN) search
 - Uses LanceDB's automatic parameter calculation based on your dataset size and vector dimensions
 
 **Re-indexing:**
@@ -762,9 +762,9 @@ Indexes are not automatically updated when you add new documents. After adding a
 haiku-rag create-index  # Rebuilds the index with all data
 ```
 
-Searches still work with stale indexes - LanceDB uses the index for old data and brute-force for new unindexed rows, then combines the results. However, performance degrades as more unindexed data accumulates.
+Searches still work with stale indexes - LanceDB uses the index for old data (fast ANN) and brute-force kNN for new unindexed rows, then combines the results. However, performance degrades as more unindexed data accumulates.
 
-For datasets with fewer than 256 chunks, searches use brute-force scans which are slower but still functional.
+For datasets with fewer than 256 chunks, searches use brute-force kNN scans (exact nearest neighbors, 100% recall) which work well for small datasets but don't scale beyond a few hundred thousand vectors.
 
 ### Document Processing