Update documentation

.gitignore

@@ -16,8 +16,9 @@ tests/data/
 .pytest_cache/
 .ruff_cache/
 
-# environment variables
+# environment variables and config files
 .env
+haiku.rag.yaml
 TODO.md
 PLAN.md
 DEVNOTES.md

README.md

@@ -4,6 +4,8 @@ Retrieval-Augmented Generation (RAG) library built on LanceDB.
 
 `haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work with LanceDB as a local vector database. It uses LanceDB for storing embeddings and performs semantic (vector) search as well as full-text search combined through native hybrid search with Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
 
+> **Note**: Configuration now uses YAML files instead of environment variables. If you're upgrading from an older version, run `haiku-rag init-config --from-env` to migrate your `.env` file to `haiku.rag.yaml`. See [Configuration](https://ggozad.github.io/haiku.rag/configuration/) for details.
+
 ## Features
 
 - **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure
@@ -58,10 +60,11 @@ haiku-rag research \
 haiku-rag rebuild
 
 # Start server with file monitoring
-export MONITOR_DIRECTORIES="/path/to/docs"
+haiku-rag serve --monitor
-haiku-rag serve
 ```
 
+To customize settings, create a `haiku.rag.yaml` config file (see [Configuration](https://ggozad.github.io/haiku.rag/configuration/)).
+
 ## Python Usage
 
 ```python
@@ -172,7 +175,7 @@ See the [examples directory](examples/) for working examples:
 Full documentation at: https://ggozad.github.io/haiku.rag/
 
 - [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup
-- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - Environment variables
+- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - YAML configuration
 - [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
 - [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs
 - [Agents](https://ggozad.github.io/haiku.rag/agents/) - QA agent and multi-agent research

docs/configuration.md

@@ -1,20 +1,122 @@
 # Configuration
 
-Configuration is done through the use of environment variables.
+Configuration is done through YAML configuration files.
 
 !!! note
     If you create a db with certain settings and later change them, `haiku.rag` will detect incompatibilities (for example, if you change embedding provider) and will exit. You can **rebuild** the database to apply the new settings, see [Rebuild Database](./cli.md#rebuild-database).
 
+## Getting Started
+
+Generate a configuration file with defaults:
+
+```bash
+haiku-rag init-config
+```
+
+This creates a `haiku.rag.yaml` file in your current directory with all available settings.
+
+To migrate from environment variables (`.env` file):
+
+```bash
+haiku-rag init-config --from-env
+```
+
+## Configuration File Locations
+
+`haiku.rag` searches for configuration files in this order:
+
+1. Path specified via `--config` flag: `haiku-rag --config /path/to/config.yaml <command>`
+2. `./haiku.rag.yaml` (current directory)
+3. `~/.config/haiku.rag/config.yaml` (user config directory)
+
+## Minimal Configuration
+
+A minimal configuration file with defaults:
+
+```yaml
+# haiku.rag.yaml
+environment: production
+
+embeddings:
+  provider: ollama
+  model: qwen3-embedding
+  vector_dim: 4096
+
+qa:
+  provider: ollama
+  model: gpt-oss
+```
+
+## Complete Configuration Example
+
+```yaml
+# haiku.rag.yaml
+environment: production
+
+storage:
+  data_dir: ""  # Empty = use default platform location
+  monitor_directories:
+    - /path/to/documents
+    - /another/path
+  disable_autocreate: false
+  vacuum_retention_seconds: 60
+
+lancedb:
+  uri: ""  # Empty for local, or db://, s3://, az://, gs://
+  api_key: ""
+  region: ""
+
+embeddings:
+  provider: ollama
+  model: qwen3-embedding
+  vector_dim: 4096
+
+reranking:
+  provider: ""  # Empty to disable, or mxbai, cohere, vllm
+  model: ""
+
+qa:
+  provider: ollama
+  model: gpt-oss
+
+research:
+  provider: ""  # Empty to use qa settings
+  model: ""
+
+processing:
+  chunk_size: 256
+  context_chunk_radius: 0
+  markdown_preprocessor: ""
+
+providers:
+  ollama:
+    base_url: http://localhost:11434
+
+  vllm:
+    embeddings_base_url: ""
+    rerank_base_url: ""
+    qa_base_url: ""
+    research_base_url: ""
+
+  api_keys:
+    voyage: ""
+    openai: ""
+    anthropic: ""
+    cohere: ""
+
+a2a:
+  max_contexts: 1000
+```
+
 ## File Monitoring
 
 Set directories to monitor for automatic indexing:
 
-```bash
+```yaml
-# Monitor single directory
+storage:
-MONITOR_DIRECTORIES="/path/to/documents"
+  monitor_directories:
-
+    - /path/to/documents
-# Monitor multiple directories
+    - /another_path/to/documents
-MONITOR_DIRECTORIES="/path/to/documents,/another_path/to/documents"
 ```
 
 ## Embedding Providers
@@ -23,44 +125,60 @@ If you use Ollama, you can use any pulled model that supports embeddings.
 
 ### Ollama (Default)
 
-```bash
+```yaml
-EMBEDDINGS_PROVIDER="ollama"
+embeddings:
-EMBEDDINGS_MODEL="mxbai-embed-large"
+  provider: ollama
-EMBEDDINGS_VECTOR_DIM=1024
+  model: mxbai-embed-large
+  vector_dim: 1024
 ```
 
 ### VoyageAI
-If you want to use VoyageAI embeddings you will need to install `haiku.rag` with the VoyageAI extras,
+
+If you want to use VoyageAI embeddings you will need to install `haiku.rag` with the VoyageAI extras:
 
 ```bash
 uv pip install haiku.rag[voyageai]
 ```
 
-```bash
+```yaml
-EMBEDDINGS_PROVIDER="voyageai"
+embeddings:
-EMBEDDINGS_MODEL="voyage-3.5"
+  provider: voyageai
-EMBEDDINGS_VECTOR_DIM=1024
+  model: voyage-3.5
-VOYAGE_API_KEY="your-api-key"
+  vector_dim: 1024
+
+providers:
+  api_keys:
+    voyage: your-api-key
 ```
 
 ### OpenAI
-OpenAI embeddings are included in the default installation. Simply set environment variables:
 
-```bash
+OpenAI embeddings are included in the default installation:
-EMBEDDINGS_PROVIDER="openai"
+
-EMBEDDINGS_MODEL="text-embedding-3-small"  # or text-embedding-3-large
+```yaml
-EMBEDDINGS_VECTOR_DIM=1536
+embeddings:
-OPENAI_API_KEY="your-api-key"
+  provider: openai
+  model: text-embedding-3-small  # or text-embedding-3-large
+  vector_dim: 1536
+
+providers:
+  api_keys:
+    openai: your-api-key
 ```
 
 ### vLLM
+
 For high-performance local inference, you can use vLLM to serve embedding models with OpenAI-compatible APIs:
 
-```bash
+```yaml
-EMBEDDINGS_PROVIDER="vllm"
+embeddings:
-EMBEDDINGS_MODEL="mixedbread-ai/mxbai-embed-large-v1"  # Any embedding model supported by vLLM
+  provider: vllm
-EMBEDDINGS_VECTOR_DIM=512  # Dimension depends on the model
+  model: mixedbread-ai/mxbai-embed-large-v1
-VLLM_EMBEDDINGS_BASE_URL="http://localhost:8000"  # vLLM server URL
+  vector_dim: 512
+
+providers:
+  vllm:
+    embeddings_base_url: http://localhost:8000
 ```
 
 **Note:** You need to run a vLLM server separately with an embedding model loaded.
@@ -71,60 +189,79 @@ Configure which LLM provider to use for question answering. Any provider and mod
 
 ### Ollama (Default)
 
-```bash
+```yaml
-QA_PROVIDER="ollama"
+qa:
-QA_MODEL="gpt-oss"
+  provider: ollama
-OLLAMA_BASE_URL="http://localhost:11434"
+  model: gpt-oss
+
+providers:
+  ollama:
+    base_url: http://localhost:11434
 ```
 
 ### OpenAI
 
-OpenAI QA is included in the default installation. Simply configure:
+OpenAI QA is included in the default installation:
 
-```bash
+```yaml
-QA_PROVIDER="openai"
+qa:
-QA_MODEL="gpt-4o-mini"  # or gpt-4, gpt-3.5-turbo, etc.
+  provider: openai
-OPENAI_API_KEY="your-api-key"
+  model: gpt-4o-mini  # or gpt-4, gpt-3.5-turbo, etc.
+
+providers:
+  api_keys:
+    openai: your-api-key
 ```
 
 ### Anthropic
 
-Anthropic QA is included in the default installation. Simply configure:
+Anthropic QA is included in the default installation:
 
-```bash
+```yaml
-QA_PROVIDER="anthropic"
+qa:
-QA_MODEL="claude-3-5-haiku-20241022"  # or claude-3-5-sonnet-20241022, etc.
+  provider: anthropic
-ANTHROPIC_API_KEY="your-api-key"
+  model: claude-3-5-haiku-20241022  # or claude-3-5-sonnet-20241022, etc.
+
+providers:
+  api_keys:
+    anthropic: your-api-key
 ```
 
 ### vLLM
 
-For high-performance local inference, you can use vLLM to serve models with OpenAI-compatible APIs:
+For high-performance local inference:
 
-```bash
+```yaml
-QA_PROVIDER="vllm"
+qa:
-QA_MODEL="Qwen/Qwen3-4B"  # Any model with tool support in vLLM
+  provider: vllm
-VLLM_QA_BASE_URL="http://localhost:8002"  # vLLM server URL
+  model: Qwen/Qwen3-4B  # Any model with tool support in vLLM
+
+providers:
+  vllm:
+    qa_base_url: http://localhost:8002
 ```
 
 **Note:** You need to run a vLLM server separately with a model that supports tool calling loaded. Consult the specific model's documentation for proper vLLM serving configuration.
 
 ### Other Providers
 
-Any provider supported by Pydantic AI can be used. Examples include:
+Any provider supported by Pydantic AI can be used. Examples:
 
-```bash
+```yaml
 # Google Gemini
-QA_PROVIDER="gemini"
+qa:
-QA_MODEL="gemini-1.5-flash"
+  provider: gemini
+  model: gemini-1.5-flash
 
 # Groq
-QA_PROVIDER="groq"
+qa:
-QA_MODEL="llama-3.3-70b-versatile"
+  provider: groq
+  model: llama-3.3-70b-versatile
 
 # Mistral
-QA_PROVIDER="mistral"
+qa:
-QA_MODEL="mistral-small-latest"
+  provider: mistral
+  model: mistral-small-latest
 ```
 
 See the [Pydantic AI documentation](https://ai.pydantic.dev/models/) for the complete list of supported providers and models.
@@ -133,7 +270,7 @@ See the [Pydantic AI documentation](https://ai.pydantic.dev/models/) for the com
 
 Reranking improves search quality by re-ordering the initial search results using specialized models. When enabled, the system retrieves more candidates (3x the requested limit) and then reranks them to return the most relevant results.
 
-Reranking is **disabled by default** (`RERANK_PROVIDER=""`) for faster searches. You can enable it by configuring one of the providers below.
+Reranking is **disabled by default** (`provider: ""`) for faster searches. You can enable it by configuring one of the providers below.
 
 ### MixedBread AI
 
@@ -145,29 +282,38 @@ uv pip install haiku.rag[mxbai]
 
 Then configure:
 
-```bash
+```yaml
-RERANK_PROVIDER="mxbai"
+reranking:
-RERANK_MODEL="mixedbread-ai/mxbai-rerank-base-v2"
+  provider: mxbai
+  model: mixedbread-ai/mxbai-rerank-base-v2
 ```
 
 ### Cohere
 
-Cohere reranking is included in the default installation. Simply configure:
+Cohere reranking is included in the default installation:
 
-```bash
+```yaml
-RERANK_PROVIDER="cohere"
+reranking:
-RERANK_MODEL="rerank-v3.5"
+  provider: cohere
-COHERE_API_KEY="your-api-key"
+  model: rerank-v3.5
+
+providers:
+  api_keys:
+    cohere: your-api-key
 ```
 
 ### vLLM
 
 For high-performance local reranking using dedicated reranking models:
 
-```bash
+```yaml
-RERANK_PROVIDER="vllm"
+reranking:
-RERANK_MODEL="mixedbread-ai/mxbai-rerank-base-v2"  # Any reranking model supported by vLLM
+  provider: vllm
-VLLM_RERANK_BASE_URL="http://localhost:8001"  # vLLM server URL
+  model: mixedbread-ai/mxbai-rerank-base-v2
+
+providers:
+  vllm:
+    rerank_base_url: http://localhost:8001
 ```
 
 **Note:** vLLM reranking uses the `/rerank` API endpoint. You need to run a vLLM server separately with a reranking model loaded. Consult the specific model's documentation for proper vLLM serving configuration.
@@ -178,78 +324,91 @@ VLLM_RERANK_BASE_URL="http://localhost:8001"  # vLLM server URL
 
 By default, `haiku.rag` uses a local LanceDB database:
 
-```bash
+```yaml
-# Default data directory (where local LanceDB is stored)
+storage:
-DEFAULT_DATA_DIR="/path/to/data"
+  data_dir: /path/to/data  # Empty = use default platform location
 ```
 
-For remote storage, use the `LANCEDB_URI` setting with various backends:
+For remote storage, use the `lancedb` settings with various backends:
 
-```bash
+```yaml
 # LanceDB Cloud
-LANCEDB_URI="db://your-database-name"
+lancedb:
-LANCEDB_API_KEY="your-api-key"
+  uri: db://your-database-name
-LANCEDB_REGION="us-west-2"  # optional
+  api_key: your-api-key
+  region: us-west-2  # optional
 
 # Amazon S3
-LANCEDB_URI="s3://my-bucket/my-table"
+lancedb:
+  uri: s3://my-bucket/my-table
 # Use AWS credentials or IAM roles
 
 # Azure Blob Storage
-LANCEDB_URI="az://my-container/my-table"
+lancedb:
+  uri: az://my-container/my-table
 # Use Azure credentials
 
 # Google Cloud Storage
-LANCEDB_URI="gs://my-bucket/my-table"
+lancedb:
+  uri: gs://my-bucket/my-table
 # Use GCP credentials
 
 # HDFS
-LANCEDB_URI="hdfs://namenode:port/path/to/table"
+lancedb:
+  uri: hdfs://namenode:port/path/to/table
 ```
 
-Authentication is handled through standard cloud provider credentials (AWS CLI, Azure CLI, gcloud, etc.) or by setting `LANCEDB_API_KEY` for LanceDB Cloud.
+Authentication is handled through standard cloud provider credentials (AWS CLI, Azure CLI, gcloud, etc.) or by setting `api_key` for LanceDB Cloud.
 
 **Note:** Table optimization is automatically handled by LanceDB Cloud (`db://` URIs) and is disabled for better performance. For object storage backends (S3, Azure, GCS), optimization is still performed locally.
 
 #### Disable database auto-creation
 
-By default, haiku.rag creates the local LanceDB directory and required tables on first use. To prevent accidental database creation and fail fast if a database hasn’t been set up yet, set:
+By default, haiku.rag creates the local LanceDB directory and required tables on first use. To prevent accidental database creation and fail fast if a database hasn't been set up yet:
 
-```bash
+```yaml
-DISABLE_DB_AUTOCREATE=true
+storage:
+  disable_autocreate: true
 ```
 
 When enabled, for local paths, haiku.rag errors if the LanceDB directory does not exist, and it will not create parent directories.
 
 ### Document Processing
 
-```bash
+```yaml
-# Chunk size for document processing
+processing:
-CHUNK_SIZE=256
+  # Chunk size for document processing
+  chunk_size: 256
 
-# Number of adjacent chunks to include before/after retrieved chunks for context
+  # Number of adjacent chunks to include before/after retrieved chunks for context
-# 0 = no expansion (default), 1 = include 1 chunk before and after, etc.
+  # 0 = no expansion (default), 1 = include 1 chunk before and after, etc.
-# When expanded chunks overlap or are adjacent, they are automatically merged
+  # When expanded chunks overlap or are adjacent, they are automatically merged
-# into single chunks with continuous content to eliminate duplication
+  # into single chunks with continuous content to eliminate duplication
-CONTEXT_CHUNK_RADIUS=0
+  context_chunk_radius: 0
 
-# Vacuum retention threshold (seconds) for automatic cleanup
+  # Optional dotted path or file path to a callable that preprocesses
-# When documents are added/updated, old table versions older than this are removed
+  # markdown content before chunking
-# Default: 60 seconds (safe for concurrent connections)
+  markdown_preprocessor: ""
-# Set to 0 for aggressive cleanup (removes all old versions immediately)
+
-VACUUM_RETENTION_SECONDS=60
+storage:
+  # Vacuum retention threshold (seconds) for automatic cleanup
+  # When documents are added/updated, old table versions older than this are removed
+  # Default: 60 seconds (safe for concurrent connections)
+  # Set to 0 for aggressive cleanup (removes all old versions immediately)
+  vacuum_retention_seconds: 60
 ```
 
 #### Markdown Preprocessor
 
 Optionally preprocess Markdown before chunking by pointing to a callable that receives and returns Markdown text. This is useful for normalizing content, stripping boilerplate, or applying custom transformations before chunk boundaries are computed.
 
-```bash
+```yaml
-# A callable path in one of these formats:
+processing:
-# - package.module:func
+  # A callable path in one of these formats:
-# - package.module.func
+  # - package.module:func
-# - /abs/or/relative/path/to/file.py:func
+  # - package.module.func
-MARKDOWN_PREPROCESSOR="my_pkg.preprocess:clean_md"
+  # - /abs/or/relative/path/to/file.py:func
+  markdown_preprocessor: my_pkg.preprocess:clean_md
 ```
 
 !!! note
@@ -271,3 +430,16 @@ def clean_md(text: str) -> str:
         out.append(line)
     return "\n".join(out)
 ```
+
+## Migration from Environment Variables
+
+!!! warning "Deprecation Notice"
+    Environment variable configuration via `.env` files is deprecated and will be removed in future versions. Please migrate to YAML configuration.
+
+To migrate your existing `.env` file to YAML:
+
+```bash
+haiku-rag init-config --from-env
+```
+
+This will read your current environment variables and generate a `haiku.rag.yaml` file with those settings.

examples/docker/haiku.rag.yaml.example

@@ -0,0 +1,25 @@
+# haiku.rag configuration for Docker deployment
+# See https://ggozad.github.io/haiku.rag/configuration/ for details
+
+environment: production
+
+storage:
+  data_dir: /data
+  monitor_directories:
+    - /docs
+
+embeddings:
+  provider: ollama
+  model: nomic-embed-text
+  vector_dim: 768
+
+qa:
+  provider: ollama
+  model: qwen3
+
+providers:
+  ollama:
+    base_url: http://host.docker.internal:11434
+
+# For other providers (OpenAI, Anthropic, VoyageAI, etc.),
+# see: https://ggozad.github.io/haiku.rag/configuration/