diff --git a/CHANGELOG.md b/CHANGELOG.md index a52e4ee3..87c02a81 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,15 +1,16 @@ # Changelog ## [Unreleased] -## [0.16.1] - 2025-11-14 - -### Changed - - **BREAKING: Chunking Tokenizer**: Switched from tiktoken to HuggingFace tokenizers for consistency with docling-serve - Default tokenizer changed from tiktoken "gpt-4o" to "Qwen/Qwen3-Embedding-0.6B" - New `chunking_tokenizer` config option in `ProcessingConfig` for customization - Removed `tiktoken` dependency - `download-models` CLI command now also downloads the configured HuggingFace tokenizer + +## [0.16.1] - 2025-11-14 + +### Changed + - **Evaluations**: Refactored QA benchmark to run entire dataset as single evaluation for better Logfire experiment tracking - **Evaluations**: Added `.env` file loading support via `python-dotenv` dependency diff --git a/README.md b/README.md index 99637f64..4325c308 100644 --- a/README.md +++ b/README.md @@ -9,14 +9,14 @@ Retrieval-Augmented Generation (RAG) library built on LanceDB. - **Local LanceDB**: No external servers required, supports also LanceDB cloud storage, S3, Google Cloud & Azure - **Multiple embedding providers**: Ollama, VoyageAI, OpenAI, vLLM - **Multiple QA providers**: Any provider/model supported by Pydantic AI -- **Research graph (multi‑agent)**: Plan → Search → Evaluate → Synthesize with agentic AI - **Native hybrid search**: Vector + full-text search with native LanceDB RRF reranking - **Reranking**: Default search result reranking with MixedBread AI, Cohere, Zero Entropy, or vLLM - **Question answering**: Built-in QA agents on your documents +- **Research graph (multi‑agent)**: Plan → Search → Evaluate → Synthesize with agentic AI - **File monitoring**: Auto-index files when run as server -- **40+ file formats**: PDF, DOCX, HTML, Markdown, code files, URLs -- **MCP server**: Expose as tools for AI assistants - **CLI & Python API**: Use from command line or Python +- **MCP server**: Expose as tools for AI assistants +- **Flexible document processing**: Local (docling) or remote (docling-serve) processing ## Installation diff --git a/docs/configuration.md b/docs/configuration.md index cc34c9e6..26961fb7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -98,6 +98,12 @@ processing: chunk_size: 256 context_chunk_radius: 0 markdown_preprocessor: "" + converter: docling-local # docling-local or docling-serve + chunker: docling-local # docling-local or docling-serve + chunker_type: hybrid # hybrid or hierarchical + chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B" + chunking_merge_peers: true + chunking_use_markdown_tables: false providers: ollama: @@ -108,6 +114,11 @@ providers: rerank_base_url: "" qa_base_url: "" research_base_url: "" + + docling_serve: + base_url: http://localhost:5001 + api_key: "" + timeout: 300 ``` ## Programmatic Configuration @@ -199,11 +210,86 @@ monitor: ``` Patterns follow [gitignore syntax](https://git-scm.com/docs/gitignore#_pattern_format): + - `*` matches anything except `/` - `**` matches zero or more directories - `?` matches any single character - `[abc]` matches any character in the set +## Document Processing + +Configure how documents are converted and chunked: + +```yaml +processing: + # Chunking configuration + chunk_size: 256 # Maximum tokens per chunk + context_chunk_radius: 0 # Context radius for chunk expansion + markdown_preprocessor: "" # Optional preprocessor script + + # Converter selection + converter: docling-local # docling-local or docling-serve + + # Chunker selection and configuration + chunker: docling-local # docling-local or docling-serve + chunker_type: hybrid # hybrid or hierarchical + chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B" # HuggingFace model for tokenization + chunking_merge_peers: true # Merge undersized successive chunks + chunking_use_markdown_tables: false # Use markdown tables vs narrative format +``` + +### Local vs Remote Processing + +**Local processing** (default): + +- Uses `docling` library locally +- No external dependencies +- Good for development and small workloads + +**Remote processing** (docling-serve): + +- Offloads processing to docling-serve API +- Better for heavy workloads and production +- Requires docling-serve instance (see [Remote processing setup](remote-processing.md)) + +To use remote processing: + +```yaml +processing: + converter: docling-serve + chunker: docling-serve + +providers: + docling_serve: + base_url: http://localhost:5001 + api_key: "your-api-key" # Optional + timeout: 300 # Request timeout in seconds +``` + +### Chunking Strategies + +**Hybrid chunking** (default): +- Structure-aware chunking +- Respects document boundaries +- Best for most use cases + +**Hierarchical chunking**: +- Creates hierarchical chunk structure +- Preserves document hierarchy +- Useful for complex documents + +### Table Serialization + +Control how tables are represented in chunks: + +```yaml +processing: + chunking_use_markdown_tables: false # Default: narrative format +``` + +- `false`: Tables as narrative text ("Value A, Column 2 = Value B") +- `true`: Tables as markdown (preserves table structure) + ## Embedding Providers If you use Ollama, you can use any pulled model that supports embeddings. diff --git a/docs/index.md b/docs/index.md index ac35679e..b14b2adb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -11,6 +11,7 @@ - **Question Answering**: Built-in QA agents using Ollama, OpenAI, or Anthropic - **File monitoring**: Automatically index files when run as a server - **Extended file format support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, code files and more. Or add a URL! +- **Flexible document processing**: Local processing with docling or remote with [docling-serve](remote-processing.md) - **MCP server**: Exposes functionality as MCP tools - **CLI commands**: Access all functionality from your terminal - Add sources from text, files, or URLs, optionally with a human‑readable title @@ -60,6 +61,7 @@ haiku-rag ask "Who is the author of haiku.rag?" - [MCP](mcp.md) - Model Context Protocol integration - [Python](python.md) - Python API reference - [Agents](agents.md) - QA agent and multi-agent research +- [Remote processing](remote-processing.md) - Remote document processing with docling-serve ## License diff --git a/docs/installation.md b/docs/installation.md index 5dc0e782..7e8f4a4d 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -58,7 +58,20 @@ You can prefetch all required runtime models before first use: haiku-rag download-models ``` -This will download Docling models and pull any Ollama models referenced by your current configuration. +This will download: +- Docling models for document processing +- HuggingFace tokenizer models for chunking +- Any Ollama models referenced by your current configuration + +## Remote Processing (Optional) + +When using `haiku.rag-slim`, you can skip installing the `docling` extra and instead use [docling-serve](https://github.com/docling-project/docling-serve) for remote document processing. This is useful for: + +- Keeping dependencies minimal +- Offloading heavy document processing to a dedicated service +- Production deployments with separate processing infrastructure + +See [Remote processing](remote-processing.md) for setup instructions and [Document Processing](configuration.md#document-processing) for configuration options. ## Docker diff --git a/docs/remote-processing.md b/docs/remote-processing.md new file mode 100644 index 00000000..42d6be71 --- /dev/null +++ b/docs/remote-processing.md @@ -0,0 +1,131 @@ +# Remote Processing + +`haiku.rag` can use [docling-serve](https://github.com/docling-project/docling-serve) for remote document processing and chunking, offloading resource-intensive operations to a dedicated service. + +## Overview + +docling-serve is a REST API service that provides: + +- Document conversion (PDF, DOCX, PPTX, images, etc.) +- Intelligent chunking with structure preservation +- OCR capabilities for scanned documents +- Table and figure extraction + +## When to Use docling-serve + +**Use local processing (default) when:** + +- Working with small to medium document volumes +- Running on development machines +- Want zero external dependencies +- Processing simple document formats + +**Use docling-serve when:** + +- Processing large volumes of documents +- Working with complex PDFs requiring OCR +- Running in production environments +- Want to separate compute-intensive tasks +- Need to scale document processing independently + +## Setup + +### Running docling-serve + +See the [official docling-serve repository](https://github.com/docling-project/docling-serve) for installation options. The quickest way is using Docker: + +```bash +docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=1 quay.io/docling-project/docling-serve +``` + +### Configuration + +Configure haiku.rag to use docling-serve. See the [Document Processing section in Configuration](configuration.md#document-processing) for all available options. + +```yaml +# haiku.rag.yaml +processing: + converter: docling-serve # Use remote conversion + chunker: docling-serve # Use remote chunking + +providers: + docling_serve: + base_url: http://localhost:5001 + api_key: "" # Optional API key for authentication + timeout: 300 # Request timeout in seconds +``` + +## Features + +### Remote Document Conversion + +When `converter: docling-serve` is configured, documents are sent to the docling-serve API for conversion: + +```python +from haiku.rag.client import HaikuRAG + +async with HaikuRAG() as client: + # PDF is processed by docling-serve + doc = await client.create_document_from_file("complex.pdf") +``` + +### Remote Chunking + +When `chunker: docling-serve` is configured, chunking is performed remotely: + +```yaml +processing: + chunker: docling-serve + chunker_type: hybrid # or hierarchical + chunk_size: 256 + chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B" + chunking_merge_peers: true + chunking_use_markdown_tables: false +``` + +## Advanced Configuration + +### Custom Tokenizers + +You can use any HuggingFace tokenizer model: + +```yaml +processing: + chunking_tokenizer: "bert-base-uncased" # Or any HF model +``` + +### Chunking Strategies + +**Hybrid Chunking** (default): + +- Best for most documents +- Preserves semantic boundaries +- Structure-aware splitting + +**Hierarchical Chunking**: + +- Maintains document hierarchy +- Better for deeply nested documents +- Preserves parent-child relationships + +```yaml +processing: + chunker_type: hierarchical +``` + +### Table Handling + +Control how tables are represented: + +```yaml +processing: + chunking_use_markdown_tables: true # Preserve table structure +``` + +- `false` (default): Tables as narrative text +- `true`: Tables as markdown format + +## Resources + +- [docling-serve GitHub](https://github.com/docling-project/docling-serve) +- [docling-serve Documentation](https://github.com/docling-project/docling-serve#readme) diff --git a/mkdocs.yml b/mkdocs.yml index 4a686f98..1604399e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -64,6 +64,7 @@ nav: - Python: python.md - Agents: agents.md - Server: server.md + - Remote processing: remote-processing.md - MCP: mcp.md - Benchmarks: benchmarks.md markdown_extensions: