diff --git a/CHANGELOG.md b/CHANGELOG.md index 87c02a81..6c9b0827 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,29 @@ # Changelog ## [Unreleased] +### Added + +- **Remote Processing**: Support for docling-serve as remote document processing and chunking service + - New `converter` config option: `docling-local` (default) or `docling-serve` + - New `chunker` config option: `docling-local` (default) or `docling-serve` + - New `providers.docling_serve` config section with `base_url`, `api_key`, and `timeout` + - Comprehensive error handling for connection, timeout, and authentication issues +- **Chunking Strategies**: Support for both hybrid and hierarchical chunking + - New `chunker_type` config option: `hybrid` (default) or `hierarchical` + - Hybrid chunking: Structure-aware splitting that respects document boundaries + - Hierarchical chunking: Preserves document hierarchy for nested documents +- **Table Serialization Control**: Configurable table representation in chunks + - New `chunking_use_markdown_tables` config option (default: `false`) + - `false`: Tables serialized as narrative text ("Value A, Column 2 = Value B") + - `true`: Tables preserved as markdown format with structure +- **Chunking Configuration**: Additional chunking control options + - New `chunking_merge_peers` config option (default: `true`) to merge undersized successive chunks + +### Changed + - **BREAKING: Chunking Tokenizer**: Switched from tiktoken to HuggingFace tokenizers for consistency with docling-serve - Default tokenizer changed from tiktoken "gpt-4o" to "Qwen/Qwen3-Embedding-0.6B" - New `chunking_tokenizer` config option in `ProcessingConfig` for customization - - Removed `tiktoken` dependency - `download-models` CLI command now also downloads the configured HuggingFace tokenizer ## [0.16.1] - 2025-11-14