danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update changelog

Browse source
This commit is contained in:
Yiorgis Gozadinos 2025-11-17 13:11:11 +02:00
parent 973522ffe2
commit da4179678b
No known key found for this signature in database
1 changed files with 20 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
CHANGELOG.md
View file

@ -1,10 +1,29 @@
# Changelog
## [Unreleased]
### Added
- **Remote Processing**: Support for docling-serve as remote document processing and chunking service
- New `converter` config option: `docling-local` (default) or `docling-serve`
- New `chunker` config option: `docling-local` (default) or `docling-serve`
- New `providers.docling_serve` config section with `base_url`, `api_key`, and `timeout`
- Comprehensive error handling for connection, timeout, and authentication issues
- **Chunking Strategies**: Support for both hybrid and hierarchical chunking
- New `chunker_type` config option: `hybrid` (default) or `hierarchical`
- Hybrid chunking: Structure-aware splitting that respects document boundaries
- Hierarchical chunking: Preserves document hierarchy for nested documents
- **Table Serialization Control**: Configurable table representation in chunks
- New `chunking_use_markdown_tables` config option (default: `false`)
- `false`: Tables serialized as narrative text ("Value A, Column 2 = Value B")
- `true`: Tables preserved as markdown format with structure
- **Chunking Configuration**: Additional chunking control options
- New `chunking_merge_peers` config option (default: `true`) to merge undersized successive chunks
### Changed
- **BREAKING: Chunking Tokenizer**: Switched from tiktoken to HuggingFace tokenizers for consistency with docling-serve
- Default tokenizer changed from tiktoken "gpt-4o" to "Qwen/Qwen3-Embedding-0.6B"
- New `chunking_tokenizer` config option in `ProcessingConfig` for customization
- Removed `tiktoken` dependency
- `download-models` CLI command now also downloads the configured HuggingFace tokenizer
## [0.16.1] - 2025-11-14