# Changelog ## [Unreleased] ### Changed - **Download Models Progress**: `haiku-rag download-models` now shows real-time progress with Rich progress bars for Ollama model downloads - **Refactored Download Models**: Moved core download logic to `HaikuRAG.download_models()` async generator that yields `DownloadProgress` events, separating business logic from UI ## [0.20.0] - 2025-11-28 ### Added - **DoclingDocument Storage**: Full DoclingDocument JSON is now stored with each document, enabling rich context and visual grounding - Documents store the complete DoclingDocument structure (JSON) and schema version - Chunks store metadata with JSON pointer references (`doc_item_refs`), semantic labels, section headings, and page numbers - New `ChunkMetadata` model for structured chunk provenance: `doc_item_refs`, `headings`, `labels`, `page_numbers` - `Document.get_docling_document()` method to parse stored DoclingDocument - `ChunkMetadata.resolve_doc_items()` to resolve JSON pointer refs to actual DocItem objects - `ChunkMetadata.resolve_bounding_boxes()` for visual grounding with page coordinates - LRU cache (100 documents) for parsed DoclingDocument objects to avoid repeated JSON parsing - **Enhanced Search Results**: `search()` and `expand_context()` now return full provenance information - `SearchResult` includes `page_numbers`, `headings`, `labels`, and `doc_item_refs` - QA and research agents use provenance for better citations (page numbers, section headings) - **Inspector Visual Grounding**: New visual grounding modal in the database inspector - View page images with highlighted bounding boxes for chunks - Keyboard navigation between pages (←/→ arrows) - Access from both main detail view and search results - Requires `textual-image` dependency - **Visual Grounding CLI**: New `haiku-rag visualize ` command - Displays page images with highlighted bounding boxes for a chunk - Requires terminal with image support (iTerm2, Kitty, etc.) - **New `import_document()` Method**: Import pre-processed documents with custom chunks - Use when document conversion, chunking, and embedding were done externally - `chunks` required; `content` optional if `docling_document_json` is provided - When `docling_document_json` is provided without `content`, content is extracted from the DoclingDocument - Validates docling JSON parses correctly if provided - **`update_document_fields()` DoclingDocument Support**: Added `docling_document_json` and `docling_version` parameters - When `docling_document_json` is provided without `chunks`, content is extracted and document is rechunked - When `docling_document_json` is provided with `chunks`, both are stored (chunks used as-is) - `content` and `docling_document_json` are mutually exclusive to avoid ambiguity - **New `convert()` Method**: Convert files, URLs, or text to DoclingDocument - `client.convert(Path(...))` - convert local file - `client.convert("https://...")` - download and convert URL - `client.convert("text content")` - convert plain text - Supports `file://` URIs - **New `chunk()` Method**: Chunk a DoclingDocument into Chunk objects - `client.chunk(docling_doc)` - returns `list[Chunk]` without embeddings - **New `contextualize()` and `embed_chunks()` Utilities**: Standalone embedding utilities in `haiku.rag.embeddings` - `contextualize(chunks)` - prepend section headings to chunk content for better semantic search - `embed_chunks(chunks)` - generate embeddings for chunks, returns new Chunk objects with embeddings set ### Changed - **BREAKING: `create_document()` API**: Removed `chunks` parameter - `create_document()` now always processes content (converts, chunks, embeds) - Use new `import_document()` for pre-processed documents with custom chunks - **BREAKING: Chunker Interface**: `DocumentChunker.chunk()` now returns `list[Chunk]` instead of `list[str]` - Chunks include structured metadata (doc_item_refs, labels, headings, page_numbers) in the `metadata` dict - All chunker implementations updated: `DoclingLocalChunker`, `DoclingServeChunker` - **Page Image Generation**: `generate_page_images=True` is now always enabled for local docling converter - Required for visual grounding features - Removed `generate_page_images` config option (docling-serve already generates page images by default) - **Chunk Text Storage**: Chunks now store raw text without heading contextualization - Section headings are prepended only at embedding time for better semantic search - Stored chunk content stays clean without duplicate heading prefixes - Headings remain available in `ChunkMetadata` for display and citations - Local and serve chunkers now produce identical output - **QA Prompts**: Updated to use page numbers and section headings in citations when available - **Citation Models**: Introduced `RawSearchAnswer` for LLM output, `SearchAnswer` extends it with resolved citations - Cleaner separation: LLM outputs chunk IDs, citations resolved programmatically - `Citation` fields are now required (no defaults) for type safety ### Migration This release requires a database rebuild to populate the new DoclingDocument fields: ```bash haiku-rag rebuild ``` Existing documents without DoclingDocument data will work but won't have provenance information. The `rebuild` command re-processes all documents to populate the new fields. ## [0.19.6] - 2025-12-03 ### Changed - **BREAKING: Explicit Database Creation**: Databases must now be explicitly created before use - New `haiku-rag init` command creates a new empty database - Python API: `HaikuRAG(path, create=True)` to create database programmatically - Operations on non-existent databases raise `FileNotFoundError` - **BREAKING: Embeddings Configuration**: Restructured to nested `EmbeddingModelConfig` - Config path changed from `embeddings.{provider, model, vector_dim}` to `embeddings.model.{provider, name, vector_dim}` - Automatic migration upgrades existing databases to new format - **Database Migrations**: Always run when opening an existing database ## [0.19.5] - 2025-12-01 ### Changed - **Rebuild Performance**: Optimized `rebuild --embed-only` to use batch updates via LanceDB's `merge_insert` instead of individual chunk updates, and skip chunks with unchanged embeddings ## [0.19.4] - 2025-11-28 ### Added - **Rebuild Modes**: New options for `rebuild` command to control what gets rebuilt - `--embed-only`: Only regenerate embeddings, keeping existing chunks (fastest option when changing embedding model) - `--rechunk`: Re-chunk from existing document content without accessing source files - Default (no flag): Full rebuild with source file re-conversion - Python API: `rebuild_database(mode=RebuildMode.EMBED_ONLY | RECHUNK | FULL)` ## [0.19.3] - 2025-11-27 ### Changed - **Async Chunker**: `DoclingServeChunker` now uses `httpx.AsyncClient` instead of sync `requests` ### Fixed - **OCR Options**: Fixed `DoclingLocalConverter` using base `OcrOptions` class which docling's OCR factory doesn't recognize. Now uses `OcrAutoOptions` for automatic OCR engine selection. - **Dependencies**: Added `opencv-python-headless` to the `docling` optional dependency for table structure detection. ## [0.19.2] - 2025-11-27 ### Changed - **Async Converters**: Made document converters fully async - `BaseConverter.convert_file()` and `convert_text()` are now async methods - `DoclingLocalConverter` wraps blocking Docling operations with `asyncio.to_thread()` - `DoclingServeConverter` now uses `httpx.AsyncClient` instead of sync `requests` - **Async Model Prefetch**: `prefetch_models()` is now async - Uses `httpx.AsyncClient` for Ollama model pulls - Wraps blocking Docling and HuggingFace downloads with `asyncio.to_thread()` ## [0.19.1] - 2025-11-26 ### Added - **LM Studio Provider**: Added support for LM Studio as a provider for embeddings and QA/research models - Configure with `provider: lm_studio` in embeddings, QA, or research model settings - Supports thinking control for reasoning models (gpt-oss, etc.) - Default base URL: `http://localhost:1234` ### Fixed - **Configuration**: Fixed `init-config` command generating invalid configuration files (#165) - Refactored `generate_default_config()` to use Pydantic model serialization instead of manual dict construction - Updated `qa`, `research`, and `reranking` sections to use new `ModelConfig` structure ## [0.19.0] - 2025-11-25 ### Added - **Model Customization**: Added support for per-model configuration settings - New `enable_thinking` parameter to control reasoning behavior (true/false/None) - Support for `temperature` and `max_tokens` settings on QA and research models - All settings apply to any provider that supports them - **Database Inspector**: New `inspect` CLI command launches interactive TUI for browsing documents and chunks & searching - **Evaluations**: Added `evaluations` CLI script for running benchmarks (replaces `python -m evaluations.benchmark`) - **Evaluations**: Added `--db` option to override evaluation database path - Default database location moved to haiku.rag data directory: - macOS: `~/Library/Application Support/haiku.rag/evaluations/dbs/` - Linux: `~/.local/share/haiku.rag/evaluations/dbs/` - Windows: `C:/Users//AppData/Roaming/haiku.rag/evaluations/dbs/` - Previously stored in `evaluations/data/` within the repository - **Evaluations**: Added comprehensive experiment metadata tracking for better reproducibility - Records dataset name, test case count, and all model configurations - Tracks embedder settings: provider, model, and vector dimensions - Tracks QA model: provider and model name - Tracks judge model: provider and model name for LLM evaluation - Tracks processing parameters: `chunk_size` and `context_chunk_radius` - Tracks retrieval configuration: `retrieval_limit` for number of chunks retrieved - Tracks reranking configuration: `rerank_provider` and `rerank_model` - Enables comparison of evaluation runs with different configurations in Logfire - **Evaluations**: Refactored retrieval evaluation to use pydantic-ai experiment framework - New `evaluators` module with `MRREvaluator` (Mean Reciprocal Rank) and `MAPEvaluator` (Mean Average Precision) - Retrieval benchmarks now use `Dataset.evaluate()` with full Logfire experiment tracking - Dataset specifications now declare their retrieval evaluator (MRR for RepliQA, MAP for Wix) - Replaced Recall@K and Success@K with industry-standard MRR and MAP metrics - Unified evaluation framework for both retrieval and QA benchmarks - **AG-UI Events**: Enhanced ActivitySnapshot events with richer structured data - Added `stepName` field to identify which graph node emitted each activity - Added structured fields to activity content while preserving backward-compatible `message` field: - **Planning**: `sub_questions` - list of sub-question strings - **Searching**: `query` - the search query, `confidence` - answer confidence (on success), `error` - error message (on failure) - **Analyzing** (research): `insights` - list of insight objects, `gaps` - list of gap objects, `resolved_gaps` - list of resolved gap strings - **Evaluating** (research): `confidence` - confidence score, `is_sufficient` - sufficiency flag - **Evaluating** (deep QA): `is_sufficient` - sufficiency flag, `iterations` - iteration count ### Changed - **Evaluations**: Renamed `--qa-limit` CLI parameter to `--limit`, now applies to both retrieval and QA benchmarks - **Evaluations**: Retrieval evaluator selection moved from runtime logic to dataset configuration ## [0.18.0] - 2025-11-21 ### Added - **Manual Vector Indexing**: New `create-index` CLI command for explicit vector index creation - Creates IVF_PQ indexes - Requires minimum 256 chunks (LanceDB training data requirement) - New `search.vector_index_metric` config option: `cosine` (default), `l2`, or `dot` - New `search.vector_refine_factor` config option (default: 30) for accuracy/speed tradeoff - Indexes not created automatically during ingestion to avoid performance degradation - Manual rebuilding required after adding significant new data - **Enhanced Info Command**: `haiku-rag info` now shows storage sizes and vector index statistics - Displays storage size for documents and chunks tables in human-readable format - Shows vector index status (exists/not created) - Shows indexed and unindexed chunk counts for monitoring index staleness ### Changed - **BREAKING: Default Embedding Model**: Changed default embedding model from `qwen3-embedding` to `qwen3-embedding:4b` with vector dimension 2560 (previously 4096) - New installations will use the smaller, more efficient 4B parameter model by default - **Action required**: Existing databases created with the old default will be incompatible. Users must either: - Explicitly set `embeddings.model: "qwen3-embedding"` and `embeddings.vector_dim: 4096` in their config to maintain compatibility with existing databases - Or run `haiku-rag rebuild` to re-embed all documents with the new default - This change provides better performance for most use cases while reducing resource requirements - **Evaluations**: Improved evaluation dataset naming and simplified evaluator configuration - `EvalDataset` now accepts dataset name for better organization in Logfire - Added `--name` CLI parameter to override evaluation run names - Removed `IsInstance` evaluator, using only `LLMJudge` for QA evaluation - **Search Accuracy**: Applied `refine_factor` to vector and hybrid searches for improved accuracy - Retrieves `refine_factor * limit` candidates and re-ranks in memory - Higher values increase accuracy but slow down queries ### Fixed - **AG-UI Activity Events**: Activity events now correctly use structured dict content instead of strings - **Graph Configuration**: Graph builder functions now properly accept and use non-global config (#149) - `build_research_graph()` and `build_deep_qa_graph()` now pass config to all agents and model creation - `get_model()` utility function accepts `config` parameter (defaults to global Config) - Allows creating multiple graphs with different configurations in the same application ## [0.17.2] - 2025-11-19 ### Added - **Document Update API**: New `update_document_fields()` method for partial document updates - Update individual fields (content, metadata, title, chunks) without fetching full document - Support for custom chunks or auto-generation from content ### Changed - **Chunk Creation**: `ChunkRepository.create()` now accepts both single chunks and lists for batch insertion - Batch insertion reduces LanceDB version creation when adding multiple chunks with custom chunks - Batch embedding generation for improved performance with multiple chunks - Updated core dependencies ## [0.17.1] - 2025-11-18 ### Added - **Conversion Options**: Fine-grained control over document conversion for both local and remote converters - New `conversion_options` config section in `ProcessingConfig` - OCR settings: `do_ocr`, `force_ocr`, `ocr_lang` for controlling OCR behavior - Table extraction: `do_table_structure`, `table_mode` (fast/accurate), `table_cell_matching` - Image settings: `images_scale` to control image resolution - Options work identically with both `docling-local` and `docling-serve` converters ### Changed - Increase reranking candidate retrieval multiplier from 3x to 10x for improved result quality - **Docker Images**: Main `haiku.rag` image no longer automatically built and published - **Conversion Options**: Removed the legacy `pdf_backend` setting; docling now chooses the optimal backend automatically ## [0.17.0] - 2025-11-17 ### Added - **Remote Processing**: Support for docling-serve as remote document processing and chunking service - New `converter` config option: `docling-local` (default) or `docling-serve` - New `chunker` config option: `docling-local` (default) or `docling-serve` - New `providers.docling_serve` config section with `base_url`, `api_key`, and `timeout` - Comprehensive error handling for connection, timeout, and authentication issues - **Chunking Strategies**: Support for both hybrid and hierarchical chunking - New `chunker_type` config option: `hybrid` (default) or `hierarchical` - Hybrid chunking: Structure-aware splitting that respects document boundaries - Hierarchical chunking: Preserves document hierarchy for nested documents - **Table Serialization Control**: Configurable table representation in chunks - New `chunking_use_markdown_tables` config option (default: `false`) - `false`: Tables serialized as narrative text ("Value A, Column 2 = Value B") - `true`: Tables preserved as markdown format with structure - **Chunking Configuration**: Additional chunking control options - New `chunking_merge_peers` config option (default: `true`) to merge undersized successive chunks - **Docker Images**: Two Docker images for different deployment scenarios - `haiku.rag`: Full image with all dependencies for self-contained deployments - `haiku.rag-slim`: Minimal image designed for use with external docling-serve - Multi-platform support (linux/amd64, linux/arm64) - Docker Compose examples with docling-serve integration - Automated CI/CD workflows for both images - Build script (`scripts/build-docker-images.sh`) for local multi-platform builds ### Changed - **BREAKING: Chunking Tokenizer**: Switched from tiktoken to HuggingFace tokenizers for consistency with docling-serve - Default tokenizer changed from tiktoken "gpt-4o" to "Qwen/Qwen3-Embedding-0.6B" - New `chunking_tokenizer` config option in `ProcessingConfig` for customization - `download-models` CLI command now also downloads the configured HuggingFace tokenizer - **Docker Examples**: Updated examples to demonstrate remote processing - `examples/docker` now uses slim image with docling-serve - `examples/ag-ui-research` backend uses slim image with docling-serve - Configuration examples include remote processing setup ## [0.16.1] - 2025-11-14 ### Changed - **Evaluations**: Refactored QA benchmark to run entire dataset as single evaluation for better Logfire experiment tracking - **Evaluations**: Added `.env` file loading support via `python-dotenv` dependency ## [0.16.0] - 2025-11-13 ### Added - **AG-UI Protocol Support**: Full AG-UI (Agent-UI) protocol implementation for graph execution with event streaming - New `AGUIEmitter` class for emitting AG-UI events from graphs - Support for all AG-UI event types: lifecycle events (`RUN_STARTED`, `RUN_FINISHED`, `RUN_ERROR`), step events (`STEP_STARTED`, `STEP_FINISHED`), state updates (`STATE_SNAPSHOT`, `STATE_DELTA`), activity narration (`ACTIVITY_SNAPSHOT`), and text messages (`TEXT_MESSAGE_CHUNK`) - `AGUIConsoleRenderer` for rendering AG-UI event streams to terminal with Rich formatting - `stream_graph()` utility function for executing graphs with AG-UI event emission - State diff computation for efficient state synchronization - **Delta State Updates**: AG-UI emitter now supports incremental state updates via JSON Patch operations (`STATE_DELTA` events) to reduce bandwidth, configurable via `use_deltas` parameter (enabled by default) - **AG-UI Server**: Starlette-based HTTP server for serving graphs via AG-UI protocol - Server-Sent Events (SSE) streaming endpoint at `/v1/agent/stream` - Health check endpoint at `/health` - Full CORS support configurable via `agui` config section - `create_agui_server()` function for programmatic server creation - **Deep QA AG-UI Support**: Deep QA graph now fully supports AG-UI event streaming - Integration with `AGUIEmitter` for progress tracking - Step-by-step execution visibility via AG-UI events - **CLI AG-UI Flag**: New `--agui` flag for `serve` command to start AG-UI server - **Graph Module**: New unified `haiku.rag.graph` module containing all graph-related functionality - **Common Graph Nodes**: New factory functions (`create_plan_node`, `create_search_node`) in `haiku.rag.graph.common.nodes` for reusable graph components - **AG-UI Research Example**: New full-stack example (`examples/ag-ui-research`) demonstrating agent+graph architecture with CopilotKit frontend - Pydantic AI agent with research tool that invokes the research graph - Custom AG-UI streaming endpoint with anyio memory streams - React/Next.js frontend with split-pane UI showing live research state - Real-time progress tracking of questions, answers, insights, and gaps - Docker Compose setup for easy local development ### Changed - **Vacuum Retention**: Default `vacuum_retention_seconds` increased from 60 seconds to 86400 seconds (1 day) for better version retention in typical workflows - **BREAKING**: Major refactoring of graph-related code into unified `haiku.rag.graph` module structure: - `haiku.rag.research` → `haiku.rag.graph.research` - `haiku.rag.qa.deep` → `haiku.rag.graph.deep_qa` - `haiku.rag.agui` → `haiku.rag.graph.agui` - `haiku.rag.graph_common` → `haiku.rag.graph.common` - **BREAKING**: Research and Deep QA graphs now use AG-UI event protocol instead of direct console logging - Removed `console` and `stream` parameters from graph dependencies - All progress updates now emit through `AGUIEmitter` - **BREAKING**: `ResearchState` converted from dataclass to Pydantic `BaseModel` for JSON serialization and AG-UI compatibility - Research and Deep QA graphs now emit detailed execution events for better observability - CLI research command now uses AG-UI event rendering for `--verbose` output - Improved graph execution visibility with step-by-step progress tracking - Updated all documentation to reflect new import paths and AG-UI usage - Updated examples (ag-ui-research, a2a-server) to use new import paths ### Fixed - **Document Creation**: Optimized `create_document` to skip unnecessary DoclingDocument conversion when chunks are pre-provided - **FileReader**: Error messages now include both original exception details and file path for easier debugging - **Database Auto-creation**: Read operations (search, list, get, ask, research) no longer auto-create empty databases. Write operations (add, add-src, delete, rebuild) still create the database as needed. This prevents the confusing scenario where a search query creates an empty database. Fixes issue #137. ### Removed - **BREAKING**: Removed `disable_autocreate` config option - the behavior is now automatic based on operation type - **BREAKING**: Removed legacy `ResearchStream` and `ResearchStreamEvent` classes (replaced by AG-UI event protocol) ## [0.15.0] - 2025-11-07 ### Added - **File Monitor**: Orphan deletion feature - automatically removes documents from database when source files are deleted (enabled via `monitor.delete_orphans` config option, default: false) ### Changed - **Configuration**: All CLI commands now properly support `--config` parameter for specifying custom configuration files - Configuration loading consolidated across CLI, app, and client with consistent resolution order - `HaikuRAGApp` and MCP server now accept `config` parameter for programmatic configuration - Updated CLI documentation to clarify global vs per-command options - **BREAKING**: Standardized configuration filename to `haiku.rag.yaml` in user directories (was incorrectly using `config.yaml`). Users with existing `config.yaml` in their user directory will need to rename it to `haiku.rag.yaml` ### Fixed - **File Monitor**: Fixed incorrect "Updated document" logging for unchanged files - monitor now properly skips files when MD5 hash hasn't changed ### Removed - **BREAKING**: A2A (Agent-to-Agent) protocol support has been moved to a separate self-contained package in `examples/a2a-server/`. The A2A server is no longer part of the main haiku.rag package. Users who need A2A functionality can install and run it from the examples directory with `cd examples/a2a-server && uv sync`. - **BREAKING**: Removed deprecated `.env`-based configuration system. The `haiku-rag init-config --from-env` command and `load_config_from_env()` function have been removed. All configuration must now be done via YAML files. Environment variables for API keys (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) and service URLs (e.g., `OLLAMA_BASE_URL`) are still supported and can be set via `.env` files. ## [0.14.1] - 2025-11-06 ### Added - Migrated research and deep QA agents to use Pydantic Graph beta API for better graph execution - Automatic semaphore-based concurrency control for parallel sub-question processing - `max_concurrency` parameter for controlling parallel execution in research and deep QA (default: 1) ### Changed - **BREAKING**: Research and Deep QA graphs now use `pydantic_graph.beta` instead of the class-based graph implementation - Refactored graph common patterns into `graph_common` module - Sub-questions now process using `.map()` for true parallel execution - Improved graph structure with cleaner node definitions and flow control - Pinned critical dependencies: `docling-core`, `lancedb`, `docling` ## [0.14.0] - 2024-11-05 ### Added - New `haiku.rag-slim` package with minimal dependencies for users who want to install only what they need - Evaluations package (`haiku.rag-evals`) for internal benchmarking and testing - Improved search filtering performance by using pandas DataFrames for joins instead of SQL WHERE IN clauses ### Changed - **BREAKING**: Restructured project into UV workspace with three packages: - `haiku.rag-slim` - Core package with minimal dependencies - `haiku.rag` - Full package with all extras (recommended for most users) - `haiku.rag-evals` - Internal benchmarking and evaluation tools - Migrated from `pydantic-ai` to `pydantic-ai-slim` with extras system - Docling is now an optional dependency (install with `haiku.rag-slim[docling]`) - Package metadata checks now use `haiku.rag-slim` (always present) instead of `haiku.rag` - Docker image optimized: removed evaluations package, reducing installed packages from 307 to 259 - Improved vector search performance through optimized score normalization ### Fixed - ImportError now properly raised when optional docling dependency is missing ## [0.13.3] - 2024-11-04 ### Added - Support for Zero Entropy reranker - Filter parameter to `search()` for filtering documents before search - Filter parameter to CLI `search` command - Filter parameter to CLI `list` command for filtering document listings - Config option to pass custom configuration files to evaluation commands - Document filtering now respects configured include/exclude patterns when using `add-src` with directories - Max retries to insight_agent when producing structured output ### Fixed - CLI now loads `.env` files at startup - Info command no longer attempts to use deprecated `.env` settings - Documentation typos ## [0.13.2] - 2024-11-04 ### Added - Gitignore-style pattern filtering for file monitoring using pathspec - Include/exclude pattern documentation for FileMonitor ### Changed - Moved monitor configuration to its own section in config - Improved configuration documentation - Updated dependencies ## [0.13.1] - 2024-11-03 ### Added - Initial version tracking [Unreleased]: https://github.com/ggozad/haiku.rag/compare/0.14.0...HEAD [0.14.0]: https://github.com/ggozad/haiku.rag/compare/0.13.3...0.14.0 [0.13.3]: https://github.com/ggozad/haiku.rag/compare/0.13.2...0.13.3 [0.13.2]: https://github.com/ggozad/haiku.rag/compare/0.13.1...0.13.2 [0.13.1]: https://github.com/ggozad/haiku.rag/releases/tag/0.13.1