diff --git a/CHANGELOG.md b/CHANGELOG.md index f9cb1696..d8964b94 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,7 +52,11 @@ - `content` and `docling_document` are mutually exclusive - **BREAKING: Chunker Interface**: `DocumentChunker.chunk()` now returns `list[Chunk]` instead of `list[str]` - Chunks include structured metadata (doc_item_refs, labels, headings, page_numbers) -- **BREAKING: Config Renamed**: `context_chunk_radius` renamed to `text_context_radius` +- **Search Config**: New settings in `search` section for search behavior and context expansion + - `search.limit` - Default number of search results (default: 5). Used by CLI, MCP server, and API when no limit specified + - `search.context_radius` - DocItems before/after to include for text content expansion (default: 0) + - `search.max_context_items` - Maximum items in expanded context (default: 25) + - `search.max_context_chars` - Maximum characters in expanded context (default: 10000) - **Rebuild Performance**: Batched database writes during `rebuild` command reduce LanceDB versions by ~98% - All rebuild modes (FULL, RECHUNK, EMBED_ONLY) now batch writes across documents - Eliminates redundant per-document chunk deletions and vacuum calls diff --git a/docs/configuration/index.md b/docs/configuration/index.md index 8b58db58..b2063844 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -100,6 +100,9 @@ research: search: limit: 5 # Default number of results to return + context_radius: 0 # DocItems before/after to include for text content + max_context_items: 25 # Maximum items in expanded context + max_context_chars: 10000 # Maximum characters in expanded context vector_index_metric: cosine # cosine, l2, or dot vector_refine_factor: 30 @@ -116,9 +119,6 @@ processing: chunker: docling-local # docling-local or docling-serve chunker_type: hybrid # hybrid or hierarchical chunk_size: 256 - text_context_radius: 0 - max_context_items: 25 - max_context_chars: 10000 chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B" chunking_merge_peers: true chunking_use_markdown_tables: false diff --git a/docs/configuration/processing.md b/docs/configuration/processing.md index af298ed3..a4b43895 100644 --- a/docs/configuration/processing.md +++ b/docs/configuration/processing.md @@ -11,11 +11,6 @@ processing: # Chunking configuration chunk_size: 256 # Maximum tokens per chunk - # Context expansion for search results - text_context_radius: 0 # Radius for text chunk expansion - max_context_items: 25 # Maximum items in expanded context - max_context_chars: 10000 # Maximum characters in expanded context - # Converter selection converter: docling-local # docling-local or docling-serve @@ -137,27 +132,14 @@ processing: - `false`: Tables as narrative text ("Value A, Column 2 = Value B") - `true`: Tables as markdown (preserves table structure) -### Chunk Size and Context Expansion +### Chunk Size ```yaml processing: - # Chunk size for document processing - chunk_size: 256 - - # Context expansion settings - # Controls how search results are expanded with surrounding content - text_context_radius: 0 # Chunks before/after to include for text content - max_context_items: 25 # Maximum doc items to include in expansion - max_context_chars: 10000 # Maximum characters in expanded content + chunk_size: 256 # Maximum tokens per chunk ``` -Context expansion enriches search results with surrounding content from the source document: - -- **text_context_radius**: For text content (paragraphs), includes N chunks before and after. Set to 0 to disable expansion (default). -- **max_context_items**: Limits how many document items (paragraphs, list items, etc.) can be included in expanded context. -- **max_context_chars**: Hard limit on total characters in expanded content. - -Structural content (tables, code blocks, lists) uses type-aware expansion that automatically includes the complete structure regardless of how it was chunked. For example, if a table was split across multiple chunks, expansion retrieves the complete table. +Context expansion settings (for enriching search results with surrounding content) are configured in the `search` section. See [Search Settings](storage.md#search-settings). ## File Monitoring diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md index f8182bdf..97a10c8f 100644 --- a/docs/configuration/storage.md +++ b/docs/configuration/storage.md @@ -59,16 +59,32 @@ This prevents the common mistake where a search query accidentally creates an em ## Search Settings -Configure search behavior: +Configure search behavior and context expansion: ```yaml search: limit: 5 # Default number of results to return + context_radius: 0 # DocItems before/after to include for text content + max_context_items: 25 # Maximum items in expanded context + max_context_chars: 10000 # Maximum characters in expanded context vector_index_metric: cosine # cosine, l2, or dot vector_refine_factor: 30 # Re-ranking factor for accuracy ``` - **limit**: Default number of search results to return when no limit is specified. Used by CLI, MCP server, and API. Default: 5 + +### Context Expansion + +Context expansion enriches search results with surrounding content from the source document: + +- **context_radius**: For text content (paragraphs), includes N DocItems before and after. Set to 0 to disable expansion (default). +- **max_context_items**: Limits how many document items (paragraphs, list items, etc.) can be included in expanded context. +- **max_context_chars**: Hard limit on total characters in expanded content. + +Structural content (tables, code blocks, lists) uses type-aware expansion that automatically includes the complete structure regardless of how it was chunked. For example, if a table was split across multiple chunks, expansion retrieves the complete table. + +### Vector Indexing + - **vector_index_metric**: Distance metric for vector similarity: - `cosine`: Cosine similarity (default, best for most embeddings) - `l2`: Euclidean distance diff --git a/evaluations/evaluations/benchmark.py b/evaluations/evaluations/benchmark.py index 9cc391cf..b6d260b8 100644 --- a/evaluations/evaluations/benchmark.py +++ b/evaluations/evaluations/benchmark.py @@ -45,9 +45,10 @@ def build_experiment_metadata( "embedder_model": config.embeddings.model.name, "embedder_dim": config.embeddings.model.vector_dim, "chunk_size": config.processing.chunk_size, - "text_context_radius": config.processing.text_context_radius, - "max_context_items": config.processing.max_context_items, - "max_context_chars": config.processing.max_context_chars, + "search_limit": config.search.limit, + "context_radius": config.search.context_radius, + "max_context_items": config.search.max_context_items, + "max_context_chars": config.search.max_context_chars, "rerank_provider": config.reranking.model.provider if config.reranking.model else None, diff --git a/haiku_rag_slim/haiku/rag/client.py b/haiku_rag_slim/haiku/rag/client.py index 3466bdbc..33101e14 100644 --- a/haiku_rag_slim/haiku/rag/client.py +++ b/haiku_rag_slim/haiku/rag/client.py @@ -880,8 +880,8 @@ class HaikuRAG: Expansion is type-aware based on content: - Tables, code blocks, and lists expand to include complete structures - - Text content uses the configured radius (text_context_radius) - - Expansion is limited by max_context_items and max_context_chars + - Text content uses the configured radius (search.context_radius) + - Expansion is limited by search.max_context_items and search.max_context_chars Args: search_results: List of SearchResult objects from search. @@ -889,9 +889,9 @@ class HaikuRAG: Returns: List of SearchResult objects with expanded content and resolved provenance. """ - radius = self._config.processing.text_context_radius - max_items = self._config.processing.max_context_items - max_chars = self._config.processing.max_context_chars + radius = self._config.search.context_radius + max_items = self._config.search.max_context_items + max_chars = self._config.search.max_context_chars # Group by document_id for efficient processing document_groups: dict[str | None, list[SearchResult]] = {} diff --git a/haiku_rag_slim/haiku/rag/config/models.py b/haiku_rag_slim/haiku/rag/config/models.py index 8809ea81..4b3b0faf 100644 --- a/haiku_rag_slim/haiku/rag/config/models.py +++ b/haiku_rag_slim/haiku/rag/config/models.py @@ -110,9 +110,6 @@ class ConversionOptions(BaseModel): class ProcessingConfig(BaseModel): chunk_size: int = 256 - text_context_radius: int = 0 - max_context_items: int = 25 - max_context_chars: int = 10000 converter: str = "docling-local" chunker: str = "docling-local" chunker_type: str = "hybrid" @@ -124,6 +121,9 @@ class ProcessingConfig(BaseModel): class SearchConfig(BaseModel): limit: int = 5 + context_radius: int = 0 + max_context_items: int = 25 + max_context_chars: int = 10000 vector_index_metric: Literal["cosine", "l2", "dot"] = "cosine" vector_refine_factor: int = 30 diff --git a/tests/test_context_enhancement.py b/tests/test_context_enhancement.py index 240a0ef6..36df90f6 100644 --- a/tests/test_context_enhancement.py +++ b/tests/test_context_enhancement.py @@ -81,8 +81,8 @@ def small_chunk_config() -> AppConfig: """Config with small chunk size to force splitting.""" config = AppConfig() config.processing.chunk_size = 32 - config.processing.max_context_items = 25 - config.processing.max_context_chars = 10000 + config.search.max_context_items = 25 + config.search.max_context_chars = 10000 return config @@ -228,7 +228,7 @@ async def test_text_expansion_uses_radius(temp_db_path): """Text content expansion should use radius, not structural boundaries.""" config = AppConfig() config.processing.chunk_size = 32 - config.processing.text_context_radius = 1 # Small radius + config.search.context_radius = 1 # Small radius # Create a document with longer paragraphs that will split doc = DoclingDocument(name="text_test") @@ -312,7 +312,7 @@ async def test_max_items_limit_caps_expansion(temp_db_path): """Expansion should respect max_context_items limit.""" config = AppConfig() config.processing.chunk_size = 32 - config.processing.max_context_items = 2 # Very restrictive + config.search.max_context_items = 2 # Very restrictive docling_doc = create_list_document() @@ -410,7 +410,7 @@ async def test_expand_context_radius_zero(temp_db_path): async def test_expand_context_multiple_documents(temp_db_path): """Test expand_context with results from multiple documents.""" config = AppConfig() - config.processing.text_context_radius = 1 + config.search.context_radius = 1 async with HaikuRAG(temp_db_path, config=config, create=True) as client: # Create first document with manual chunks @@ -471,7 +471,7 @@ async def test_expand_context_multiple_documents(temp_db_path): async def test_expand_context_merges_overlapping_chunks(temp_db_path): """Test that overlapping expanded chunks are merged into one.""" config = AppConfig() - config.processing.text_context_radius = 1 + config.search.context_radius = 1 async with HaikuRAG(temp_db_path, config=config, create=True) as client: # Create document with 5 chunks @@ -526,7 +526,7 @@ async def test_expand_context_merges_overlapping_chunks(temp_db_path): async def test_expand_context_keeps_separate_non_overlapping(temp_db_path): """Test that non-overlapping expanded chunks remain separate.""" config = AppConfig() - config.processing.text_context_radius = 1 + config.search.context_radius = 1 async with HaikuRAG(temp_db_path, config=config, create=True) as client: # Create document with chunks far apart @@ -591,7 +591,7 @@ async def test_expand_context_keeps_separate_non_overlapping(temp_db_path): async def test_expand_context_with_docling_merges_overlapping(temp_db_path): """Test that expand_context with DoclingDocument merges overlapping results.""" config = AppConfig() - config.processing.text_context_radius = 3 + config.search.context_radius = 3 markdown_content = """# Chapter 1 @@ -646,7 +646,7 @@ This is paragraph four about topic C. async def test_expand_context_docling_merges_metadata(temp_db_path): """Test that expand_context properly merges metadata from multiple results.""" config = AppConfig() - config.processing.text_context_radius = 10 + config.search.context_radius = 10 markdown_content = """# Introduction