danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Use single-threaded zstd compressor

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-06-24 10:08:25 +03:00
parent 2ca12c0096
commit 62d2326933
No known key found for this signature in database
2 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
CHANGELOG.md
View file

@ -1,6 +1,10 @@
# Changelog # Changelog
## [Unreleased] ## [Unreleased]
### Fixed
- zstd compression uses a fresh `zstandard` compressor/decompressor per call instead of shared module-level singletons, fixing process segfaults when ingester workers compress documents concurrently (Python < 3.14).
## [0.61.0] - 2026-06-23 ## [0.61.0] - 2026-06-23
### Added ### Added

2
haiku_rag_slim/haiku/rag/store/compression.py
View file

@ -17,7 +17,7 @@ except ImportError:
# _prepare_document_from_docling), so construct a fresh instance per call # _prepare_document_from_docling), so construct a fresh instance per call
# rather than sharing a module-level singleton. # rather than sharing a module-level singleton.
def _zstd_compress(data: bytes) -> bytes: def _zstd_compress(data: bytes) -> bytes:
return ZstdCompressor(threads=-1).compress(data) return ZstdCompressor().compress(data)
def _zstd_decompress(data: bytes) -> bytes: def _zstd_decompress(data: bytes) -> bytes:
content_size = get_frame_parameters(data).content_size content_size = get_frame_parameters(data).content_size