From dbb2837550519b03e910b48080d2df3da4363e4c Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Fri, 21 Nov 2025 14:11:31 +0200 Subject: [PATCH] Add a note explaining when indexing is necessary --- docs/configuration.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 6849125c..98f3a698 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -741,6 +741,9 @@ search: - `dot`: Dot product similarity - **vector_refine_factor**: Retrieve `refine_factor * limit` candidates and re-rank in memory for better accuracy. Higher values increase accuracy but slow down queries. Default: 10 +!!! note + Vector indexes are only necessary for large datasets with over 100,000 chunks. For smaller datasets, LanceDB's brute-force kNN search provides exact results with good performance. Only create an index if you notice search performance degradation on large datasets. + **Index creation:** Vector indexes are **not created automatically** during document ingestion to avoid slowing down the process. After you've added documents (at least 256 chunks required), create the index manually: