From ed5519b38d08bab6dd4829e5cc55e854ba35866b Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Wed, 19 Aug 2026 15:52:50 +0300 Subject: [PATCH 1/2] Make the documented configuration match the code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit search.limit was documented as 10 in three places while the default is 5. The documented way to disable reranking, provider: "", is a valid ModelConfig, so it raised "Unknown reranking provider" — disabling means omitting reranking.model or setting it to null. The inline provider list named four of the six rerankers. prompts.picture_description: null fails validation, since the field is a non-optional str. storage.data_dir: "" coerced to Path("") — the working directory — while two doc pages promise the platform default and soliplex's example config relies on it. Empty or whitespace now resolves to the platform directory; an explicit "." is still honoured, so a config that wants the working directory says so. Three tests keep this from drifting again: every fenced yaml block in the docs validates against AppConfig, every value in the complete example either equals its default or is listed as a deliberate deviation, and empty data_dir resolves to the platform default. init-config's test reimplemented the command body instead of invoking it, which is why the command carried a coverage pragma. It now goes through CliRunner, with the refuse-to-overwrite guard covered too. --- CHANGELOG.md | 4 + docs/configuration/index.md | 7 +- docs/configuration/prompts.md | 6 +- docs/configuration/providers.md | 2 +- docs/configuration/qa.md | 4 +- haiku_rag_slim/haiku/rag/cli.py | 2 +- haiku_rag_slim/haiku/rag/config/models.py | 13 ++ tests/test_config.py | 142 +++++++++++++++++++--- 8 files changed, 153 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bdd5aeda..3c0bc477 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,10 @@ ### Fixed +- `storage.data_dir: ""` resolves to the platform data directory, as documented; it coerced to `Path("")`, so the database was created in the process's working directory. Set `data_dir: .` to keep the old placement. +- Documented `search.limit` default is 5, was stated as 10. +- The documented way to disable reranking is omitting `reranking.model` or setting it to `null`; the previous `provider: ""` example raised `Unknown reranking provider`. +- `prompts.picture_description: null` is not valid; the documented example sets a string or omits the key. - `create_document_from_source` closes the source adapter it builds for the call; adapters passed in through `sources` are left to their owner. - Directory ingestion skips symlinked files resolving outside the given directory, matching `FSSource.discover`. - A FULL rebuild no longer deletes a source-backed document before re-ingesting it: it refreshes the document in place, so the document id is preserved and a failed fetch or conversion falls back to rebuilding from stored content instead of losing the document. diff --git a/docs/configuration/index.md b/docs/configuration/index.md index 7b475081..d2434af7 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -96,9 +96,10 @@ embeddings: vector_dim: 2560 reranking: + # Omit this section, or set `model: null`, to disable reranking. model: - provider: "" # Empty to disable, or cross-encoder, cohere, zeroentropy, vllm - name: "" + provider: cross-encoder # cross-encoder, cohere, zeroentropy, vllm, jina, jina-local + name: cross-encoder/ms-marco-MiniLM-L-6-v2 multimodal: false # vllm only: send picture chunks to the reranker as images qa: @@ -110,7 +111,7 @@ qa: max_searches: 5 search: - limit: 10 # Default number of results to return + limit: 5 # Default number of results to return max_context_chars: 5000 # Maximum characters in expanded context vector_index_metric: cosine # cosine, l2, or dot vector_refine_factor: 30 diff --git a/docs/configuration/prompts.md b/docs/configuration/prompts.md index b495d445..754925cc 100644 --- a/docs/configuration/prompts.md +++ b/docs/configuration/prompts.md @@ -12,8 +12,10 @@ prompts: system, including installation manuals, maintenance procedures, and safety guidelines. Questions about "the system" or unqualified specs refer to the Helios panel. - # VLM prompt for image description during conversion (optional) - picture_description: null # Uses default prompt + # VLM prompt for image description during conversion. + # Omit the key to use the built-in prompt. + picture_description: | + Describe this figure in two sentences, naming any axis labels and units. ``` ## Domain Preamble diff --git a/docs/configuration/providers.md b/docs/configuration/providers.md index 2da41a14..b08065c6 100644 --- a/docs/configuration/providers.md +++ b/docs/configuration/providers.md @@ -391,7 +391,7 @@ See the [Pydantic AI documentation](https://ai.pydantic.dev/models/) for the com Reranking improves search quality by re-ordering the initial search results using specialized models. When enabled, the system retrieves more candidates (10x the requested limit) and then reranks them to return the most relevant results. -Reranking is **disabled by default** (`provider: ""`) for faster searches. You can enable it by configuring one of the providers below. +Reranking is **disabled by default** for faster searches: there is no `reranking.model`. Enable it by configuring one of the providers below, and disable it again by removing the section or setting `model: null`. ### Cohere diff --git a/docs/configuration/qa.md b/docs/configuration/qa.md index bb20f893..9716c4d1 100644 --- a/docs/configuration/qa.md +++ b/docs/configuration/qa.md @@ -6,11 +6,11 @@ Configure search behavior and context expansion: ```yaml search: - limit: 10 # Default number of results to return + limit: 5 # Default number of results to return max_context_chars: 5000 # Maximum characters in expanded context ``` -- **limit**: Default number of search results to return when no limit is specified. Used by CLI, MCP server, and QA. Default: 10 +- **limit**: Default number of search results to return when no limit is specified. Used by CLI, MCP server, and QA. Default: 5 - **max_context_chars**: Hard limit on total characters in expanded content. Default: 5000. Context expansion is automatic and section-aware. For structured documents (with section headers), expansion includes the entire section containing the match. For sections that exceed the budget or are too small (e.g., a title+authors area), expansion grows outward item-by-item from the match center, skipping noise labels (footnotes, page headers). This naturally crosses into adjacent sections until the budget is filled. Picture and table matches are exempt: they return their enclosing section as-is and never cross section boundaries. For unstructured documents, expansion grows outward item-by-item. Results without `doc_item_refs` (e.g., custom chunks passed to `import_document`) pass through unexpanded. diff --git a/haiku_rag_slim/haiku/rag/cli.py b/haiku_rag_slim/haiku/rag/cli.py index f746d149..8757eb67 100644 --- a/haiku_rag_slim/haiku/rag/cli.py +++ b/haiku_rag_slim/haiku/rag/cli.py @@ -414,7 +414,7 @@ def settings(): # pragma: no cover @_cli.command("init-config", help="Generate a YAML configuration file") -def init_config( # pragma: no cover +def init_config( output: Path = typer.Argument( Path("haiku.rag.yaml"), help="Output path for the config file", diff --git a/haiku_rag_slim/haiku/rag/config/models.py b/haiku_rag_slim/haiku/rag/config/models.py index baf71b49..4428d695 100644 --- a/haiku_rag_slim/haiku/rag/config/models.py +++ b/haiku_rag_slim/haiku/rag/config/models.py @@ -70,6 +70,19 @@ class StorageConfig(ConfigModel): auto_vacuum: bool = True vacuum_retention_seconds: int = Field(default=86400, ge=0) + @field_validator("data_dir", mode="before") + @classmethod + def _platform_default_when_empty(cls, value: Any) -> Any: + """An empty data_dir means the platform default directory. + + Without this it coerces to Path("") — the working directory — so a + config carrying `data_dir: ""` would put the database wherever the + process happened to start. + """ + if value is None or (isinstance(value, str) and not value.strip()): + return get_default_data_dir() + return value + class LanceDBConfig(ConfigModel): """LanceDB connection settings. diff --git a/tests/test_config.py b/tests/test_config.py index ef1de525..118a90c5 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -1,3 +1,4 @@ +import re from pathlib import Path import pytest @@ -206,29 +207,36 @@ def test_generate_default_config_completeness(): assert config.reranking.model is None -def test_init_config_creates_valid_yaml(tmp_path): - """Test that generated config can be written to YAML and loaded back.""" +def test_init_config_writes_a_loadable_config(tmp_path): + """`haiku-rag init-config` output has to load back as an AppConfig.""" + from typer.testing import CliRunner + + from haiku.rag.cli import _cli as cli + config_file = tmp_path / "test-config.yaml" + result = CliRunner().invoke(cli, ["init-config", str(config_file)]) - # Generate and write config - config_data = generate_default_config() - - with open(config_file, "w") as f: - f.write("# haiku.rag configuration file\n") - f.write( - "# See https://ggozad.github.io/haiku.rag/configuration/ for details\n\n" - ) - yaml.dump(config_data, f, default_flow_style=False, sort_keys=False) - - # Load it back - with open(config_file) as f: - loaded_data = yaml.safe_load(f) - - # Validate it - config = AppConfig.model_validate(loaded_data) + assert result.exit_code == 0, result.output + config = AppConfig.model_validate(yaml.safe_load(config_file.read_text())) assert config.environment == "production" +def test_init_config_refuses_to_overwrite(tmp_path): + """Overwriting a config in place would lose an operator's settings.""" + from typer.testing import CliRunner + + from haiku.rag.cli import _cli as cli + + config_file = tmp_path / "test-config.yaml" + config_file.write_text("environment: development\n") + + result = CliRunner().invoke(cli, ["init-config", str(config_file)]) + + assert result.exit_code == 1 + assert "already exists" in result.output + assert config_file.read_text() == "environment: development\n" + + def _write(tmp_path, body: str): p = tmp_path / "haiku.rag.yaml" p.write_text(body) @@ -693,3 +701,101 @@ def test_finite_switches_reject_unknown_values(data): def test_out_of_range_numbers_are_rejected(data): with pytest.raises(ValidationError): AppConfig.model_validate(data) + + +_DOCS_ROOT = Path(__file__).resolve().parent.parent + + +def _documented_config_blocks() -> list[tuple[str, int, str]]: + """Every fenced yaml block in the docs that looks like an AppConfig fragment.""" + known = set(AppConfig.model_fields) + blocks = [] + sources = sorted(_DOCS_ROOT.glob("docs/**/*.md")) + [ + _DOCS_ROOT / "README.md", + _DOCS_ROOT / "haiku_rag_slim" / "README.md", + ] + for path in sources: + if not path.exists(): + continue + text = path.read_text() + for match in re.finditer(r"```yaml\n(.*?)```", text, re.S): + data = yaml.safe_load(match.group(1)) + if not isinstance(data, dict) or not (set(data) & known): + continue + line = text[: match.start()].count("\n") + 1 + blocks.append((str(path.relative_to(_DOCS_ROOT)), line, match.group(1))) + return blocks + + +def test_documented_config_blocks_found(): + """Guard against the regex silently matching nothing.""" + assert len(_documented_config_blocks()) > 20 + + +@pytest.mark.parametrize( + "rel_path, line, block", + _documented_config_blocks(), + ids=[f"{rel}:{line}" for rel, line, _ in _documented_config_blocks()], +) +def test_documented_config_block_validates(rel_path, line, block): + """A config example a reader can copy has to load. Unknown keys, wrong types + and removed settings all fail here rather than on their first run.""" + AppConfig.model_validate(yaml.safe_load(block)) + + +def test_documented_search_limit_matches_the_default(): + """Prose that states a default drifts silently; pin the ones that are stated.""" + qa_doc = (_DOCS_ROOT / "docs" / "configuration" / "qa.md").read_text() + assert f"Default: {AppConfig().search.limit}" in qa_doc + + +@pytest.mark.parametrize("value", ["", " "]) +def test_empty_data_dir_means_the_platform_default(value): + """Both doc pages promise this, and soliplex's example config relies on it. + Without it the value coerces to Path("") and the database lands in whatever + directory the process started from.""" + from haiku.rag.utils import get_default_data_dir + + config = AppConfig.model_validate({"storage": {"data_dir": value}}) + + assert config.storage.data_dir == get_default_data_dir() + + +def test_explicit_relative_data_dir_is_kept(): + """`.` is a deliberate choice and must not be rewritten.""" + config = AppConfig.model_validate({"storage": {"data_dir": "."}}) + + assert config.storage.data_dir == Path(".") + + +# Values the complete example shows deliberately rather than as defaults. +_EXAMPLE_DEVIATIONS = {"storage.data_dir", "ingester.sources"} + + +def _flatten(data: dict, prefix: str = "") -> dict: + flat = {} + for key, value in (data or {}).items(): + path = f"{prefix}{key}" + if isinstance(value, dict): + flat.update(_flatten(value, path + ".")) + else: + flat[path] = value + return flat + + +def test_complete_example_matches_the_defaults(): + """The complete configuration example doubles as the default reference, so + every value in it either is the default or is listed as a deliberate + deviation. This is what catches `limit: 10` when the default is 5.""" + text = (_DOCS_ROOT / "docs" / "configuration" / "index.md").read_text() + blocks = re.findall(r"```yaml\n(.*?)```", text, re.S) + documented = _flatten(yaml.safe_load(max(blocks, key=len))) + defaults = _flatten(AppConfig().model_dump(mode="json")) + + drifted = { + key: (value, defaults[key]) + for key, value in documented.items() + if key in defaults and defaults[key] != value and key not in _EXAMPLE_DEVIATIONS + } + + assert not drifted, f"documented value != default: {drifted}" From 48a94f7793741c521078fb07f9eef1cacdc730e5 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Wed, 19 Aug 2026 16:00:27 +0300 Subject: [PATCH 2/2] Move unreleased changelog entries out of 0.75.0 The 0.75.0 section was closed after the branch behind #558 was cut, and every entry since anchored itself to a marker line inside it, so fifteen entries for unreleased work were filed under a released version: the three correctness fixes from #558, write_transaction from #559, the Config removal from #560, the strict-config group from #561, and this PR's four documentation fixes. 0.75.0 keeps only what it shipped. --- CHANGELOG.md | 42 +++++++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c0bc477..0c37dfa7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,17 +1,10 @@ # Changelog ## [Unreleased] -## [0.75.0] - 2026-08-19 - ### Added - The `haiku.rag` package declares the `jina` extra, so `provider: jina-local` is supported by declaration rather than through `cross-encoder`'s transitive `transformers` and `torch`. Raises the full package's torch floor to 2.0. - `providers.docling_serve.timeout` (default 300 seconds), forwarded to the docling-serve client's per-request timeout. -- `evaluations run --filter/-f CLAUSE`: SQL `WHERE` clause over document columns, applied to the retrieval benchmark's searches and to every capability search during QA. Recorded as `document_filter` in experiment metadata. -- `mtrag_clapnq` / `mtrag_clapnq_rewrite` / `mtrag_clapnq_live` / `mtrag_clapnq_live_uncompacted` evaluation datasets: multi-turn QA with gold-prefix and live-session conversation replay, live arms with and without `EvidenceCompactionCapability`, Recall@k/nDCG@k retrieval metrics, eligibility-aware citation scoring, refusal precision/recall, per-turn tool-traffic attributes, and `citation_status` / `turn_citation_status` eval attributes. -- Raw chunk metadata is now exposed to search and citation results, through `SearchResult.chunk_meta` and `Citation.chunk_meta`. For context-expanded results, the metadata is that of the anchor chunk. -- BTree indexes on `chunks.id`, `chunks.document_id` and `documents.id`, and a Bitmap index on `document_items.label`. Existing databases need `haiku-rag migrate`. -- `lancedb.read_consistency_interval_seconds` (default 30), `lancedb.index_cache_size_bytes` and `lancedb.metadata_cache_size_bytes`. The LanceDB session is shared across connections in a process, so its index and metadata caches survive a connection being closed. ### Changed @@ -20,18 +13,10 @@ - Numeric settings carry bounds: sizes, limits, dimensions, token budgets, attempt counts, breaker thresholds and `min_chunks` must be positive; retention, delays, intervals and cooldowns non-negative; `doctor.duplicates.similarity_threshold` within 0-1; `ingester.api.port` within 0-65535 (0 keeps its OS-assigned meaning); `ingester.workers.worker_count` allows 0 for an API-and-reaper-only process. - A configured reranker whose optional dependency is missing raises instead of silently disabling reranking, and names the extra to install (`uv pip install 'haiku.rag-slim[cohere]'`). A failure raised from inside an installed dependency propagates untouched rather than being reported as a missing package. - Multi-table writes (document create, update, batch import, cascade delete) go through `Store.write_transaction()`. Rollback restores in `RESTORE_TABLE_ORDER` and is shielded from cancellation, so a cancelled write rolls back instead of committing part of itself. `Store.restore_table_versions()` is removed. -- Search enrichment, the multimodal reranker's picture fetch, and context expansion each issue a fixed number of `document_items` queries regardless of how many documents a result set spans, instead of one set per document. `expand_with_items` takes the items to expand from rather than fetching them, and `DocumentItemRepository` gains `resolve_refs_grouped`, `get_items_in_ranges`, `get_pictures_grouped` and `get_caption_picture_refs_grouped`. The superseded methods are removed: `resolve_refs`, `get_items_in_range`, `get_caption_picture_refs`, `get_text_for_refs` and `get_all_items_grouped`. `get_pictures_grouped` returns each picture's text alongside its bytes under `with_text`, off by default so the reranker's blob fetch does not read a column it discards. -- Eval judge pinned to `qwen3.8`: `DEFAULT_JUDGE_MODEL` is `ollama:qwen3.8`, and the reference configs use `Inferact/Qwen3.8-27B-NVFP4` with `extra_body.chat_template_kwargs.reasoning_effort: low`. Results in `docs/benchmarks.md` were judged by `Qwen3.6-35B-A3B-NVFP4` and are not re-judged. -- `create_capability(rag=...)` lends a capability an open client rather than having it open its own; `client.ask`/`client.analyze` now pass theirs. -- The MCP server opens one database client for its lifetime instead of one per tool call, and fails at startup if the database cannot be opened. `delete_document` no longer opens its own connection with `skip_validation=True`, so the server no longer opts out of embedding-config validation: drift that validation rejects (any `vector_dim` mismatch, or identity drift on a writable server) now fails MCP startup instead of serving a delete-only server. Use the CLI to delete under drift. -- `import_documents` embeds chunks across the whole batch in one pass instead of per document. -- `haiku.rag` and `haiku.rag-slim` summaries and keywords; both packages now publish `[project.urls]`. -- `server.json` declares `title` and `websiteUrl`, and drops the `keywords` and `license` keys, which are not in the server schema. ### Removed - `haiku.rag.config.Config`, the eagerly loaded configuration instance. Use `get_config()` for the current global config, or pass an `AppConfig`. Every internal default (`get_embedder`, `get_converter`, `get_chunker`, `get_reranker`, `embed_chunks`, `HaikuRAG`, `Store`, `HaikuRAGApp`, `create_mcp_server`) now takes `config: AppConfig | None = None` and resolves it per call, so `set_config()` reaches them. `RerankerBase._model` no longer defaults to the configured reranker name; `CohereReranker` takes its model name as an argument. -- `wix` evaluation dataset and its reference config `evaluations/configs/wix.yaml`. ### Fixed @@ -42,6 +27,33 @@ - `create_document_from_source` closes the source adapter it builds for the call; adapters passed in through `sources` are left to their owner. - Directory ingestion skips symlinked files resolving outside the given directory, matching `FSSource.discover`. - A FULL rebuild no longer deletes a source-backed document before re-ingesting it: it refreshes the document in place, so the document id is preserved and a failed fetch or conversion falls back to rebuilding from stored content instead of losing the document. + +## [0.75.0] - 2026-08-19 + +### Added + +- `evaluations run --filter/-f CLAUSE`: SQL `WHERE` clause over document columns, applied to the retrieval benchmark's searches and to every capability search during QA. Recorded as `document_filter` in experiment metadata. +- `mtrag_clapnq` / `mtrag_clapnq_rewrite` / `mtrag_clapnq_live` / `mtrag_clapnq_live_uncompacted` evaluation datasets: multi-turn QA with gold-prefix and live-session conversation replay, live arms with and without `EvidenceCompactionCapability`, Recall@k/nDCG@k retrieval metrics, eligibility-aware citation scoring, refusal precision/recall, per-turn tool-traffic attributes, and `citation_status` / `turn_citation_status` eval attributes. +- Raw chunk metadata is now exposed to search and citation results, through `SearchResult.chunk_meta` and `Citation.chunk_meta`. For context-expanded results, the metadata is that of the anchor chunk. +- BTree indexes on `chunks.id`, `chunks.document_id` and `documents.id`, and a Bitmap index on `document_items.label`. Existing databases need `haiku-rag migrate`. +- `lancedb.read_consistency_interval_seconds` (default 30), `lancedb.index_cache_size_bytes` and `lancedb.metadata_cache_size_bytes`. The LanceDB session is shared across connections in a process, so its index and metadata caches survive a connection being closed. + +### Changed + +- Search enrichment, the multimodal reranker's picture fetch, and context expansion each issue a fixed number of `document_items` queries regardless of how many documents a result set spans, instead of one set per document. `expand_with_items` takes the items to expand from rather than fetching them, and `DocumentItemRepository` gains `resolve_refs_grouped`, `get_items_in_ranges`, `get_pictures_grouped` and `get_caption_picture_refs_grouped`. The superseded methods are removed: `resolve_refs`, `get_items_in_range`, `get_caption_picture_refs`, `get_text_for_refs` and `get_all_items_grouped`. `get_pictures_grouped` returns each picture's text alongside its bytes under `with_text`, off by default so the reranker's blob fetch does not read a column it discards. +- Eval judge pinned to `qwen3.8`: `DEFAULT_JUDGE_MODEL` is `ollama:qwen3.8`, and the reference configs use `Inferact/Qwen3.8-27B-NVFP4` with `extra_body.chat_template_kwargs.reasoning_effort: low`. Results in `docs/benchmarks.md` were judged by `Qwen3.6-35B-A3B-NVFP4` and are not re-judged. +- `create_capability(rag=...)` lends a capability an open client rather than having it open its own; `client.ask`/`client.analyze` now pass theirs. +- The MCP server opens one database client for its lifetime instead of one per tool call, and fails at startup if the database cannot be opened. `delete_document` no longer opens its own connection with `skip_validation=True`, so the server no longer opts out of embedding-config validation: drift that validation rejects (any `vector_dim` mismatch, or identity drift on a writable server) now fails MCP startup instead of serving a delete-only server. Use the CLI to delete under drift. +- `import_documents` embeds chunks across the whole batch in one pass instead of per document. +- `haiku.rag` and `haiku.rag-slim` summaries and keywords; both packages now publish `[project.urls]`. +- `server.json` declares `title` and `websiteUrl`, and drops the `keywords` and `license` keys, which are not in the server schema. + +### Removed + +- `wix` evaluation dataset and its reference config `evaluations/configs/wix.yaml`. + +### Fixed + - `DocumentRepository.delete_all` recreated `document_items` from `DocumentItemRecord` instead of `get_document_items_arrow_schema()`, returning `picture_data` as `binary` rather than `large_binary`. - `server.json` runtime arguments are `mcp --stdio`, was `serve --mcp`.