Commit graph

33 commits

Author SHA1 Message Date
Yiorgis Gozadinos
f96a428ef1
Fix defects found reviewing the coverage work
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.

Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.

Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.

Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
2026-07-27 10:44:32 +03:00
Yiorgis Gozadinos
1e8e5e9f6f
Cover MCP, ingester and converter error paths
Add tests for the MCP tools' degradation contracts, malformed WebDAV
multistatus bodies, dry-run poller sweeps including the circuit-open and
discover-failure paths, FS source scheme and symlink handling, docling-serve
zip parsing, and the remaining embedding and reranker helpers. Parametrize
_strip_etag.

Drop the misplaced pragma on the analyze handler, which sat on the return and
left the except uncovered. Add one on the FS symlink OSError guard, which
resolve(strict=False) absorbs for every real link.
2026-07-26 19:14:26 +03:00
Yiorgis Gozadinos
d9cc9a4eea
Narrow AutoTokenizer type for transformers 5 2026-07-23 12:01:28 +03:00
Chris McDonough
633517bf04 Widen chunk() type to accept None, add test for None guard 2026-06-22 14:24:12 -04:00
Chris McDonough
21a8f52893 Use patch.object to satisfy type checker 2026-06-22 14:09:01 -04:00
Chris McDonough
99a200c9d2 Restore None guard, add thread-safety test for _chunk_sync 2026-06-22 14:05:56 -04:00
Chris McDonough
25b0a155b0 Remove None guard and test — type system guarantees document is not None 2026-06-22 14:04:58 -04:00
Chris McDonough
256fd66b46 Add test for chunker.chunk(None) to cover guard clause 2026-06-22 13:58:37 -04:00
Yiorgis Gozadinos
c6514c9df4
compare off-loop work against actual event-loop thread, fix ty 2026-06-22 10:17:02 +03:00
bryan davis
fe954a090e
move cpu bound actions off of main loop 2026-06-18 16:35:48 -05:00
Yiorgis Gozadinos
339d231a4e
Remove repliqa from test setup 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
b491f767f3
Let httpx errors flow through DoclingServeClient 2026-05-26 17:38:49 +03:00
Yiorgis Gozadinos
9ed24ad53e
swap doclaynet.pdf to the full paper, add real-PDF split/merge test 2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
0f4c4af495
Cache the chunking tokenizer to avoid HF Hub 429 2026-05-13 13:32:24 +03:00
Yiorgis Gozadinos
2038d43435
collapse pictures enum to picture_description.enabled boolean 2026-05-05 11:16:08 +03:00
Yiorgis Gozadinos
c9fdad15a9
bump docling-core to >=2.74.1 so that local docling-local and docling-serve chunkers run the same MarkdownTableSerializer 2026-05-03 18:40:44 +03:00
Yiorgis Gozadinos
a3a1ad9811
detect per-document failure status in docling-serve chunker 2026-03-26 14:02:11 +02:00
Yiorgis Gozadinos
b22ac33dde
Pass OCR options from conversion_options to docling-serve chunker 2026-02-03 11:34:46 +02:00
Yiorgis Gozadinos
f861664656
Extract shared DoclingServeClient for async workflow & use it for VLM support, chunking, converting 2026-01-07 11:09:23 +02:00
Yiorgis Gozadinos
0da8ba97c4
Docling-serve cassettes 2025-12-26 18:59:46 +02:00
Yiorgis Gozadinos
50b7fb4461
Chunkers now set chunk.order directly 2025-12-18 11:35:48 +02:00
Yiorgis Gozadinos
ab57c55eaa
Drop ChunkWithMetadata, use good old Chunk 2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
343272d6fb
Chunkers return chunks with metadata, i.e. (refs, labels, headings, page_numbers) 2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
51104ddb5a
Use httpx for docling-serve 2025-11-27 15:06:13 +02:00
Yiorgis Gozadinos
14bdb7cb27
async versions for convert_file() convert_text() 2025-11-26 17:46:25 +02:00
Yiorgis Gozadinos
81b7c7d9d1
Tests for docling-serve chunker 2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
75cbdb47a8
Add chunker configuration options
- Add `chunker_type` config field to choose between "hybrid" (default) and "hierarchical" chunking strategies
  - Add `chunking_merge_peers` config field for HybridChunker (default: true)
  - Add `chunking_use_markdown_tables` config field to control table serialization format (default: false, matching docling's default)
2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
de5bb117fe
Chunker abstraction 2025-11-17 13:05:06 +02:00
Yiorgis Gozadinos
541552215e
Switch to using HuggingFace tokenizers 2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
7ba1c2376e
Introduce converters for supporting more than local docling document conversion. Transform existing FileReader and utils to "docling-local" converter 2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
ad4b655a31
Operate on DoclingDocument instead of Document 2025-08-08 17:24:34 +02:00
Yiorgis Gozadinos
91f6122ef7
Basic chunking using docling 2025-08-01 16:27:44 +02:00
Yiorgis Gozadinos
6c948a4afd
Chucker & tests 2025-06-15 21:08:06 +02:00