diff --git a/CHANGELOG.md b/CHANGELOG.md index 49af0a4e..1341c7ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,12 @@ # Changelog ## [Unreleased] +### Fixed + +- **Configuration**: Fixed `init-config` command generating invalid configuration files (#165) + - Refactored `generate_default_config()` to use Pydantic model serialization instead of manual dict construction + - Updated `qa`, `research`, and `reranking` sections to use new `ModelConfig` structure + ## [0.19.0] - 2025-11-25 ### Added diff --git a/README.md b/README.md index 2246d96a..4325c308 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ haiku-rag rebuild haiku-rag serve --monitor ``` -To customize settings, create a `haiku.rag.yaml` config file (see [Configuration](https://ggozad.github.io/haiku.rag/config-index/)). +To customize settings, create a `haiku.rag.yaml` config file (see [Configuration](https://ggozad.github.io/haiku.rag/configuration/)). ## Python Usage @@ -162,7 +162,7 @@ See the [examples directory](examples/) for working examples: Full documentation at: https://ggozad.github.io/haiku.rag/ - [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup -- [Configuration](https://ggozad.github.io/haiku.rag/config-index/) - YAML configuration +- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML configuration - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs - [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA agent and multi-agent research diff --git a/docs/cli.md b/docs/cli.md index a166de81..690d57b1 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -69,7 +69,7 @@ haiku-rag add-src /path/to/documents/ ``` !!! note - When adding a directory, the same content filters configured for [file monitoring](processing.md#filtering-monitored-files) are applied. This means `ignore_patterns` and `include_patterns` from your configuration will be used to filter which files are added. + When adding a directory, the same content filters configured for [file monitoring](configuration/processing.md#filtering-monitored-files) are applied. This means `ignore_patterns` and `include_patterns` from your configuration will be used to filter which files are added. !!! note As you add documents to `haiku.rag` the database keeps growing. By default, LanceDB supports versioning @@ -158,7 +158,7 @@ haiku-rag research "How does haiku.rag organize and query documents?" --verbose Flags: - `--verbose`: Show planning, searching previews, evaluation summary, and stop reason -Research parameters like `max_iterations`, `confidence_threshold`, and `max_concurrency` are configured in your [configuration file](config-index.md) under the `research` section. +Research parameters like `max_iterations`, `confidence_threshold`, and `max_concurrency` are configured in your [configuration file](configuration/index.md) under the `research` section. When `--verbose` is set, the CLI consumes the research graph's AG-UI event stream, displaying step events and activity snapshots as agents progress through planning, search, evaluation, and synthesis. Without `--verbose`, only the final research report is displayed. diff --git a/docs/config-index.md b/docs/configuration/index.md similarity index 99% rename from docs/config-index.md rename to docs/configuration/index.md index 55a77366..0394c1ba 100644 --- a/docs/config-index.md +++ b/docs/configuration/index.md @@ -3,7 +3,7 @@ Configuration is done through YAML configuration files. !!! note - If you create a db with certain settings and later change them, `haiku.rag` will detect incompatibilities (for example, if you change embedding provider) and will exit. You can **rebuild** the database to apply the new settings, see [Rebuild Database](./cli.md#rebuild-database). + If you create a db with certain settings and later change them, `haiku.rag` will detect incompatibilities (for example, if you change embedding provider) and will exit. You can **rebuild** the database to apply the new settings, see [Rebuild Database](../cli.md#rebuild-database). ## Getting Started diff --git a/docs/processing.md b/docs/configuration/processing.md similarity index 99% rename from docs/processing.md rename to docs/configuration/processing.md index 7dcb1518..89847b37 100644 --- a/docs/processing.md +++ b/docs/configuration/processing.md @@ -92,7 +92,7 @@ conversion_options: - Offloads processing to docling-serve API - Better for heavy workloads and production -- Requires docling-serve instance (see [Remote processing setup](remote-processing.md)) +- Requires docling-serve instance (see [Remote processing setup](../remote-processing.md)) To use remote processing: diff --git a/docs/providers.md b/docs/configuration/providers.md similarity index 100% rename from docs/providers.md rename to docs/configuration/providers.md diff --git a/docs/qa-research.md b/docs/configuration/qa-research.md similarity index 97% rename from docs/qa-research.md rename to docs/configuration/qa-research.md index 0209c908..4d3f2e0a 100644 --- a/docs/qa-research.md +++ b/docs/configuration/qa-research.md @@ -68,4 +68,4 @@ The server exposes: - `GET /health` - Health check endpoint - `POST /v1/agent/stream` - Research graph streaming endpoint (Server-Sent Events) -See [Server Mode](server.md) for more details. +See [Server Mode](../server.md) for more details. diff --git a/docs/storage.md b/docs/configuration/storage.md similarity index 100% rename from docs/storage.md rename to docs/configuration/storage.md diff --git a/docs/index.md b/docs/index.md index f191372c..8f9403dc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -55,7 +55,7 @@ haiku-rag ask "Who is the author of haiku.rag?" - [Getting started](tutorial.md) - Tutorial - [Installation](installation.md) - Install haiku.rag with different providers -- [Configuration](config-index.md) - Environment variables and settings +- [Configuration](configuration/index.md) - Environment variables and settings - [CLI](cli.md) - Command line interface usage - [Server](server.md) - File monitoring and server mode - [MCP](mcp.md) - Model Context Protocol integration diff --git a/docs/installation.md b/docs/installation.md index f5052801..9d08444a 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -43,7 +43,7 @@ The slim package has minimal dependencies and lets you install only what you nee - **OpenAI** (GPT models for QA and embeddings) - **Anthropic** (Claude models for QA) -See [Configuration](config-index.md) for configuring providers including advanced options like vLLM. +See [Configuration](configuration/index.md) for configuring providers including advanced options like vLLM. ## Requirements @@ -71,7 +71,7 @@ When using `haiku.rag-slim`, you can skip installing the `docling` extra and ins - Offloading heavy document processing to a dedicated service - Production deployments with separate processing infrastructure -See [Remote processing](remote-processing.md) for setup instructions and [Document Processing](processing.md) for configuration options. +See [Remote processing](remote-processing.md) for setup instructions and [Document Processing](configuration/processing.md) for configuration options. ## Docker diff --git a/docs/python.md b/docs/python.md index 01d8e72d..823a690d 100644 --- a/docs/python.md +++ b/docs/python.md @@ -321,6 +321,6 @@ answer = await client.ask( The QA agent will search your documents for relevant information and use the configured LLM to generate a comprehensive answer. With `cite=True`, responses include citations showing which documents were used as sources. Citations prefer the document title when present, otherwise they use the URI. -The QA provider and model are configured in `haiku.rag.yaml` or can be passed directly to the client (see [Configuration](config-index.md)). +The QA provider and model are configured in `haiku.rag.yaml` or can be passed directly to the client (see [Configuration](configuration/index.md)). See also: [Agents](agents.md) for details on the QA agent and the multi‑agent research workflow. diff --git a/docs/remote-processing.md b/docs/remote-processing.md index 2e7c281d..823b692e 100644 --- a/docs/remote-processing.md +++ b/docs/remote-processing.md @@ -44,7 +44,7 @@ docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=1 quay.io/docling-project/doc ### Configuration -Configure haiku.rag to use docling-serve. See the [Document Processing](processing.md) guide for all available options. +Configure haiku.rag to use docling-serve. See the [Document Processing](configuration/processing.md) guide for all available options. ```yaml # haiku.rag.yaml diff --git a/docs/server.md b/docs/server.md index bb344c7b..c4bab7d9 100644 --- a/docs/server.md +++ b/docs/server.md @@ -122,7 +122,7 @@ agui: cors_credentials: true ``` -See [Configuration](qa-research.md#ag-ui-server-configuration) for all available options. +See [Configuration](configuration/qa-research.md#ag-ui-server-configuration) for all available options. ### Using the Streaming Endpoints diff --git a/docs/tutorial.md b/docs/tutorial.md index a11369c4..56bd9404 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -51,7 +51,7 @@ export OPENAI_API_KEY="" For the list of available OpenAI models and their vector dimensions, see the [OpenAI documentation](https://platform.openai.com/docs/guides/embeddings). -See [Configuration](config-index.md) for all available options. +See [Configuration](configuration/index.md) for all available options. ## Adding the first documents @@ -216,4 +216,4 @@ The following people are presenting talks at PyCon Finland 2025: ## Configuration -See [Configuration page](./config-index.md) for complete documentation on YAML configuration and all available options. +See [Configuration page](./configuration/index.md) for complete documentation on YAML configuration and all available options. diff --git a/haiku_rag_slim/haiku/rag/config/loader.py b/haiku_rag_slim/haiku/rag/config/loader.py index 0b5cba9f..5c7a3074 100644 --- a/haiku_rag_slim/haiku/rag/config/loader.py +++ b/haiku_rag_slim/haiku/rag/config/loader.py @@ -48,47 +48,8 @@ def load_yaml_config(path: Path) -> dict: def generate_default_config() -> dict: - """Generate a default YAML config structure with documentation.""" - return { - "environment": "production", - "storage": { - "data_dir": "", - "vacuum_retention_seconds": 86400, - }, - "monitor": { - "directories": [], - "ignore_patterns": [], - "include_patterns": [], - }, - "lancedb": {"uri": "", "api_key": "", "region": ""}, - "embeddings": { - "provider": "ollama", - "model": "qwen3-embedding:4b", - "vector_dim": 2560, - }, - "reranking": {"provider": "", "model": ""}, - "qa": {"provider": "ollama", "model": "gpt-oss"}, - "research": {"provider": "", "model": ""}, - "processing": { - "chunk_size": 256, - "context_chunk_radius": 0, - "markdown_preprocessor": "", - }, - "providers": { - "ollama": {"base_url": "http://localhost:11434"}, - "vllm": { - "embeddings_base_url": "", - "rerank_base_url": "", - "qa_base_url": "", - "research_base_url": "", - }, - }, - "agui": { - "host": "0.0.0.0", - "port": 8000, - "cors_origins": ["*"], - "cors_credentials": True, - "cors_methods": ["GET", "POST", "OPTIONS"], - "cors_headers": ["*"], - }, - } + """Generate a default YAML config structure from AppConfig defaults.""" + from haiku.rag.config.models import AppConfig + + default_config = AppConfig() + return default_config.model_dump(mode="json", exclude_none=False) diff --git a/mkdocs.yml b/mkdocs.yml index 06cec6e0..72aa2c9e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -60,11 +60,11 @@ nav: - Getting started: tutorial.md - Installation: installation.md - Configuration: - - config-index.md - - Providers: providers.md - - QA and Research: qa-research.md - - Document Processing: processing.md - - Storage: storage.md + - configuration/index.md + - Providers: configuration/providers.md + - QA and Research: configuration/qa-research.md + - Document Processing: configuration/processing.md + - Storage: configuration/storage.md - CLI: cli.md - Python: python.md - Agents: agents.md diff --git a/tests/test_config_loader.py b/tests/test_config.py similarity index 68% rename from tests/test_config_loader.py rename to tests/test_config.py index 468e1a52..9c3daa2e 100644 --- a/tests/test_config_loader.py +++ b/tests/test_config.py @@ -1,5 +1,7 @@ import pytest +import yaml +from haiku.rag.config import AppConfig from haiku.rag.config.loader import ( find_config_file, generate_default_config, @@ -98,19 +100,6 @@ def test_find_config_file_cli_path_not_exists(tmp_path): find_config_file(config_file) -def test_generate_default_config(): - """Test generating default config structure.""" - config = generate_default_config() - - assert config["environment"] == "production" - assert "storage" in config - assert "embeddings" in config - assert "qa" in config - assert "providers" in config - assert config["embeddings"]["provider"] == "ollama" - assert config["embeddings"]["vector_dim"] == 2560 - - def test_config_precedence_cwd_over_user(tmp_path, monkeypatch): """Test that cwd config takes precedence over user config.""" # Create separate directories for cwd and user config @@ -163,3 +152,57 @@ def test_config_precedence_env_var_over_cwd(tmp_path, monkeypatch): config = load_yaml_config(found) assert config["environment"] == "from-env-var" + + +def test_generate_default_config_completeness(): + """Test that generated config has all fields from AppConfig and validates.""" + config_data = generate_default_config() + + # Validate against AppConfig model + config = AppConfig.model_validate(config_data) + + # Get all fields from AppConfig + expected_fields = set(AppConfig.model_fields.keys()) + actual_fields = set(config_data.keys()) + + # Verify all expected fields are present + assert expected_fields == actual_fields, ( + f"Missing fields: {expected_fields - actual_fields}, " + f"Extra fields: {actual_fields - expected_fields}" + ) + + # Verify nested structures are dicts/lists, not model instances + for field_name, field_value in config_data.items(): + assert not hasattr(field_value, "model_dump"), ( + f"Field {field_name} should be dict/primitive, not Pydantic model" + ) + + # Verify config validates successfully + assert config.environment == "production" + assert config.embeddings.provider == "ollama" + assert config.qa.model.provider == "ollama" + assert config.research.model.provider == "ollama" + assert config.reranking.model is None + + +def test_init_config_creates_valid_yaml(tmp_path): + """Test that generated config can be written to YAML and loaded back.""" + config_file = tmp_path / "test-config.yaml" + + # Generate and write config + config_data = generate_default_config() + + with open(config_file, "w") as f: + f.write("# haiku.rag configuration file\n") + f.write( + "# See https://ggozad.github.io/haiku.rag/configuration/ for details\n\n" + ) + yaml.dump(config_data, f, default_flow_style=False, sort_keys=False) + + # Load it back + with open(config_file) as f: + loaded_data = yaml.safe_load(f) + + # Validate it + config = AppConfig.model_validate(loaded_data) + assert config.environment == "production"