Commit graph

2108 commits

Author SHA1 Message Date
Yiorgis Gozadinos
ee38b3db42
Merge pull request #501 from ggozad/fix/docling-text-format-sniffing
Prevent magic-byte sniffing from misrouting text conversion in docling-local
2026-07-15 19:41:21 +03:00
Yiorgis Gozadinos
d2f5ecfc58
Prevent magic-byte sniffing from misrouting text conversion in docling-local
Docling guesses a DocumentStream's format from its first bytes before
considering the extension, so markdown/HTML content starting with a binary
magic signature ("BM" = BMP, "ID3" = MP3) was routed to an image or audio
backend and fell back to plain-text conversion. Prefix the encoded text
with a newline so the sniff finds nothing and the extension decides.
2026-07-15 19:20:54 +03:00
Yiorgis Gozadinos
5dd310a9b7
vb 2026-07-14 16:06:50 +03:00
Yiorgis Gozadinos
c1ec13f081
Lower search.max_context_chars default to 5000 2026-07-14 11:58:42 +03:00
Yiorgis Gozadinos
b9434045b9
Merge pull request #498 from ggozad/fix/remove-obsolete-mxbai
Remove the mxbai reranking provider
2026-07-14 11:37:26 +03:00
Yiorgis Gozadinos
144900d385
Remove the mxbai reranking provider 2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
5e2a928013
Allow transformers 5.x in the mxbai extra 2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
8f085101fc
Merge pull request #497 from ggozad/fix/context-expansion
Context expansion: never drop a retrieved result when merging and clipping
2026-07-14 10:58:07 +03:00
Yiorgis Gozadinos
82db15cf3c
Fill missing item-table pages from fully-visible inputs in context expansion 2026-07-14 10:32:45 +03:00
Yiorgis Gozadinos
ee5b7b7313
Split merged context-expansion groups that cannot afford every constituent 2026-07-14 10:15:28 +03:00
Yiorgis Gozadinos
4863abc513
Update cl 2026-07-10 15:25:33 +03:00
Yiorgis Gozadinos
5c41f8a0ed
vb 2026-07-10 13:38:24 +03:00
Yiorgis Gozadinos
1287059e76
Merge pull request #493 from ggozad/logfire-debug-skills
Add Logfire debugging skills
2026-07-10 13:36:50 +03:00
Yiorgis Gozadinos
271fdf9b5a
Add Logfire debugging skills and worker-breaker event 2026-07-10 13:23:17 +03:00
Yiorgis Gozadinos
3396394468
Merge pull request #492 from ggozad/feat/telemetry-improvements
Improve logfire telemetry
2026-07-10 11:50:40 +03:00
Yiorgis Gozadinos
a82673a900
Route eval scripts through the shared telemetry configure 2026-07-10 11:38:33 +03:00
Yiorgis Gozadinos
9b3b21b1a4
Document ingestion observability; drop stale logfire install extra 2026-07-10 11:22:23 +03:00
Yiorgis Gozadinos
73ba764922
Emit a docling_serve.request span per instance attempt 2026-07-10 11:16:27 +03:00
Yiorgis Gozadinos
da79f92af1
Honor OTEL_SERVICE_NAME and set service name/version for all processes 2026-07-10 11:11:46 +03:00
Yiorgis Gozadinos
da0748916b
vb 2026-07-09 15:58:58 +03:00
Yiorgis Gozadinos
c8497d7cd1
Merge pull request #489 from ggozad/fix/excessive-citations
Scope citations to the matched content and make visual grounding faithful
2026-07-09 15:58:15 +03:00
Yiorgis Gozadinos
e7d5a0440d
Tune two-tone visualization colors 2026-07-09 15:37:25 +03:00
Yiorgis Gozadinos
18c0f6c5e8
Add --no-expand to the visualize command for chunk-only grounding 2026-07-09 12:49:42 +03:00
Yiorgis Gozadinos
995d081aa1
Visualize the exact context the model saw via Citation.doc_item_refs
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
2026-07-09 11:51:37 +03:00
Yiorgis Gozadinos
c424f51056
Anchor and clip merged citations on the highest-scoring chunk
A merged search result took its chunk_id from whichever constituent
sorted earliest in the document, while its score was the max across the
group — so the citation's identity could point at a different, less
relevant chunk. Anchor chunk_id and the content/refs fallbacks on the
max-score constituent, clip the budget window around that same chunk so
its evidence is never trimmed away, and narrow page_numbers, doc_item_refs,
and attached image bytes to the items that survive the clip.
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
6d86237dd6
Draw matched content stronger than expanded context in visualizations 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
0bcf34363a
Carry merged chunk ids on SearchResult and Citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
b14152a45b
Keep picture and table hits section-bounded during context expansion 2026-07-09 11:05:40 +03:00
Yiorgis Gozadinos
777ce193f8
Merge pull request #491 from ggozad/fix/image-embeddings-limit
Filter picture chunks at ingest and pool embedding/reranking HTTP clients
2026-07-09 10:55:01 +03:00
Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking 2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
43a580afb3
Dedupe picture chunks and skip small pictures at chunking 2026-07-09 10:24:32 +03:00
Yiorgis Gozadinos
0c8ff570d6
vb 2026-07-08 17:04:18 +03:00
Yiorgis Gozadinos
234ece9639
Merge pull request #488 from ggozad/fix/analyst-id-in-filter
Rename document_meta identity column document_id to id
2026-07-08 16:50:47 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id 2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
d1d06729dc
Merge pull request #487 from ggozad/fix/duplicate-ingestion
Prevent duplicate documents from concurrent same-URI ingestion
2026-07-08 11:50:27 +03:00
Yiorgis Gozadinos
ec787435c8
Bind app example backend to loopback and document its lack of auth 2026-07-08 11:37:03 +03:00
Yiorgis Gozadinos
23bbca5998
Re-check URI under the write lock to prevent duplicate ingestion 2026-07-08 11:26:45 +03:00
Yiorgis Gozadinos
87548770a3
Merge pull request #486 from ggozad/feat/breakers-per-instance
docling-serve client: fail over to another instance with per-instance circuit breaking
2026-07-08 11:24:49 +03:00
Yiorgis Gozadinos
3c7699f156
Fail docling-serve requests over to another instance with per-instance breaking
When a docling-serve instance crashes or returns 5xx, DoclingServeClient now
retries the request on a different instance (up to max_attempts) and trips a
per-instance circuit breaker so subsequent jobs skip a dead instance until its
cooldown elapses. Reuses the shared CircuitBreaker; adds max_attempts and a
nested circuit_breaker to providers.docling_serve.

Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:44 +03:00
Yiorgis Gozadinos
d781335868
Move CircuitBreaker to a shared module
Relocate CircuitBreaker from ingester/pollers to haiku/rag/circuit_breaker
so non-ingester callers (docling-serve provider) can reuse it without
depending on the ingester package.

Co-Authored-By: bryan davis <bryan@monkeytronics.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:03:30 +03:00
Yiorgis Gozadinos
d71695833d
Merge pull request #485 from ggozad/fix/update-uri
Allow updating a document's uri via update_document
2026-07-08 10:41:24 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document 2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
d84147511a
Drop broken type=gha cache from Docker slim publish 2026-07-03 20:32:51 +03:00
Yiorgis Gozadinos
9a9bea86d3
vb 2026-07-03 13:35:10 +03:00
Yiorgis Gozadinos
8f3cf70eb2
Merge pull request #482 from ggozad/fix/attach-picture-from-caption
Attach figure bytes when a result matches its caption
2026-07-03 13:32:17 +03:00
Yiorgis Gozadinos
d2593304c5
Attach figure bytes when a result matches its caption 2026-07-03 13:22:02 +03:00
Yiorgis Gozadinos
07098d55e4
Merge pull request #481 from ggozad/chore/clean-up
Clean up reranker/converter/embedder layers and cut test runtime
2026-06-29 15:26:07 +03:00
Yiorgis Gozadinos
7f50d3698f
Validate reranker config before importing the provider package 2026-06-29 15:15:26 +03:00
Yiorgis Gozadinos
ac64aa9c17
Build the skills rag_db fixture once per session 2026-06-29 15:15:26 +03:00