Commit graph

35 commits

Author SHA1 Message Date
Yiorgis Gozadinos
72ef18e39d
Reject unknown and out-of-range configuration values
Every section inherited plain BaseModel, so unknown keys were dropped
silently: providers.docling_serve.timeout was documented for months while
being ignored, and a typo in any setting took the default. Sections now
derive from ConfigModel, which forbids extras, so a stale or misspelled key
fails with its path. This already found search.context_radius in a live app
config and providers.vllm in soliplex's example.

converter, chunker and chunker_type are Literals. Sizes, limits,
dimensions, token budgets, attempt counts and breaker thresholds must be
positive; retention, delays, intervals and cooldowns non-negative;
similarity_threshold within 0-1; port within 0-65535. port 0 keeps its
OS-assigned meaning and worker_count allows 0 for an API-and-reaper-only
process.

get_reranker caught ImportError and returned None, so a configured reranker
whose extra was missing silently disappeared. It now propagates.
raise_missing_extra names the install command and re-raises when the failure
came from inside an installed package, so a broken transitive import is not
reported as a missing one. zeroentropy imported bare and now guards like the
others.

The haiku.rag package declares the jina extra. jina-local already worked
there through cross-encoder's transitive transformers and torch; the
resolved package set is unchanged, but the support is now promised rather
than inherited.

Provider fields stay unconstrained: get_model ends in a pass-through to
pydantic-ai for any provider it supports, so a Literal there would reject
valid configurations.
2026-08-19 15:32:51 +03:00
Yiorgis Gozadinos
e8f00fcff4
Make get_config the only configuration lookup
haiku.rag.config exported two configuration instances: the lazy _config
behind get_config/set_config, and Config, loaded at import time. Nothing
linked them, and eleven signatures captured Config as a default argument,
so set_config could not reach the factories, the client, the store or the
MCP server. reranking/base.py went further and snapshotted the configured
reranker name into a class attribute at import.

Config is removed. Internal defaults are config: AppConfig | None = None,
resolved through get_config() per call. RerankerBase._model is None and
CohereReranker takes its model name as an argument, like every other
reranker.

The suite patched attributes on Config while production read the instance
get_config() returns, a different object, so those patches were no-ops
waiting to happen. They now go through get_config().
2026-08-19 14:43:40 +03:00
Yiorgis Gozadinos
f96a428ef1
Fix defects found reviewing the coverage work
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.

Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.

Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.

Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
2026-07-27 10:44:32 +03:00
Yiorgis Gozadinos
1e8e5e9f6f
Cover MCP, ingester and converter error paths
Add tests for the MCP tools' degradation contracts, malformed WebDAV
multistatus bodies, dry-run poller sweeps including the circuit-open and
discover-failure paths, FS source scheme and symlink handling, docling-serve
zip parsing, and the remaining embedding and reranker helpers. Parametrize
_strip_etag.

Drop the misplaced pragma on the analyze handler, which sat on the return and
left the except uncovered. Add one on the FS symlink OSError guard, which
resolve(strict=False) absorbs for every real link.
2026-07-26 19:14:26 +03:00
Yiorgis Gozadinos
d9cc9a4eea
Narrow AutoTokenizer type for transformers 5 2026-07-23 12:01:28 +03:00
Chris McDonough
633517bf04 Widen chunk() type to accept None, add test for None guard 2026-06-22 14:24:12 -04:00
Chris McDonough
21a8f52893 Use patch.object to satisfy type checker 2026-06-22 14:09:01 -04:00
Chris McDonough
99a200c9d2 Restore None guard, add thread-safety test for _chunk_sync 2026-06-22 14:05:56 -04:00
Chris McDonough
25b0a155b0 Remove None guard and test — type system guarantees document is not None 2026-06-22 14:04:58 -04:00
Chris McDonough
256fd66b46 Add test for chunker.chunk(None) to cover guard clause 2026-06-22 13:58:37 -04:00
Yiorgis Gozadinos
c6514c9df4
compare off-loop work against actual event-loop thread, fix ty 2026-06-22 10:17:02 +03:00
bryan davis
fe954a090e
move cpu bound actions off of main loop 2026-06-18 16:35:48 -05:00
Yiorgis Gozadinos
339d231a4e
Remove repliqa from test setup 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
b491f767f3
Let httpx errors flow through DoclingServeClient 2026-05-26 17:38:49 +03:00
Yiorgis Gozadinos
9ed24ad53e
swap doclaynet.pdf to the full paper, add real-PDF split/merge test 2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
0f4c4af495
Cache the chunking tokenizer to avoid HF Hub 429 2026-05-13 13:32:24 +03:00
Yiorgis Gozadinos
2038d43435
collapse pictures enum to picture_description.enabled boolean 2026-05-05 11:16:08 +03:00
Yiorgis Gozadinos
c9fdad15a9
bump docling-core to >=2.74.1 so that local docling-local and docling-serve chunkers run the same MarkdownTableSerializer 2026-05-03 18:40:44 +03:00
Yiorgis Gozadinos
a3a1ad9811
detect per-document failure status in docling-serve chunker 2026-03-26 14:02:11 +02:00
Yiorgis Gozadinos
b22ac33dde
Pass OCR options from conversion_options to docling-serve chunker 2026-02-03 11:34:46 +02:00
Yiorgis Gozadinos
f861664656
Extract shared DoclingServeClient for async workflow & use it for VLM support, chunking, converting 2026-01-07 11:09:23 +02:00
Yiorgis Gozadinos
0da8ba97c4
Docling-serve cassettes 2025-12-26 18:59:46 +02:00
Yiorgis Gozadinos
50b7fb4461
Chunkers now set chunk.order directly 2025-12-18 11:35:48 +02:00
Yiorgis Gozadinos
ab57c55eaa
Drop ChunkWithMetadata, use good old Chunk 2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
343272d6fb
Chunkers return chunks with metadata, i.e. (refs, labels, headings, page_numbers) 2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
51104ddb5a
Use httpx for docling-serve 2025-11-27 15:06:13 +02:00
Yiorgis Gozadinos
14bdb7cb27
async versions for convert_file() convert_text() 2025-11-26 17:46:25 +02:00
Yiorgis Gozadinos
81b7c7d9d1
Tests for docling-serve chunker 2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
75cbdb47a8
Add chunker configuration options
- Add `chunker_type` config field to choose between "hybrid" (default) and "hierarchical" chunking strategies
  - Add `chunking_merge_peers` config field for HybridChunker (default: true)
  - Add `chunking_use_markdown_tables` config field to control table serialization format (default: false, matching docling's default)
2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
de5bb117fe
Chunker abstraction 2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
541552215e
Switch to using HuggingFace tokenizers 2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
7ba1c2376e
Introduce converters for supporting more than local docling document conversion. Transform existing FileReader and utils to "docling-local" converter 2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
ad4b655a31
Operate on DoclingDocument instead of Document 2025-08-08 17:24:34 +02:00
Yiorgis Gozadinos
91f6122ef7
Basic chunking using docling 2025-08-01 16:27:44 +02:00
Yiorgis Gozadinos
6c948a4afd
Chucker & tests 2025-06-15 21:08:06 +02:00