Yiorgis Gozadinos
8e8c4433bd
Extract PDF /EmbeddedFiles attachments as child documents
2026-05-28 15:51:32 +03:00
Yiorgis Gozadinos
26bc71d6d8
Cascade delete_document to children via metadata.parent_uri
2026-05-28 15:36:00 +03:00
Yiorgis Gozadinos
5400085147
Auto-prune dead jobs when a sibling DELETE succeeds
2026-05-27 17:17:01 +03:00
Yiorgis Gozadinos
1e6dc34871
Skip FS DELETE enqueue when the file is already back
2026-05-27 15:18:46 +03:00
Yiorgis Gozadinos
15d13cab04
Probe docling-serve only when converter or chunker uses it
2026-05-27 15:18:25 +03:00
Yiorgis Gozadinos
b0d0ac588d
Split snapshot APIs and resolve_fetcher by intent
2026-05-27 14:39:54 +03:00
Yiorgis Gozadinos
2c63ceda0c
Backfill ingester test gaps and drop unneeded retry clamp
2026-05-27 14:24:22 +03:00
Yiorgis Gozadinos
cfbd0b09d6
Tighten HTTP config-removal handling
2026-05-27 14:07:45 +03:00
Yiorgis Gozadinos
7466539b4f
Emit DELETE for HTTP URLs removed from config
2026-05-27 13:39:18 +03:00
Yiorgis Gozadinos
d7fdd61fed
Resolve worker source by source_id, not just supports(uri)
2026-05-27 13:36:03 +03:00
Yiorgis Gozadinos
922d1d567d
Prevent DELETE/UPSERT race for the same URI
2026-05-27 13:30:46 +03:00
Yiorgis Gozadinos
1c710433c7
Collapse worker_count and max_concurrent into worker_count
2026-05-27 12:48:14 +03:00
Yiorgis Gozadinos
87af5e5139
Boot-reap stale claims at WorkerPool start
2026-05-27 11:46:37 +03:00
Yiorgis Gozadinos
1b36452629
Surface provider reachability and per-job failure context on dashboard
2026-05-27 11:38:48 +03:00
Yiorgis Gozadinos
439307d5af
Add pool-wide circuit breaker to the worker pool
2026-05-27 10:52:48 +03:00
Yiorgis Gozadinos
b491f767f3
Let httpx errors flow through DoclingServeClient
2026-05-26 17:38:49 +03:00
Yiorgis Gozadinos
fdb73fc41f
Drop defensive branches
2026-05-26 16:56:32 +03:00
Yiorgis Gozadinos
29b3ccdbda
Drain orphan cancel-cleanup releases before closing the queue
2026-05-26 16:14:17 +03:00
Yiorgis Gozadinos
3c608046e2
Bound ?limit and ?offset on /jobs and /dlq
2026-05-26 16:05:47 +03:00
Yiorgis Gozadinos
761956eb70
Extend reaper-resurrection guard to reschedule and release_if_claimed
2026-05-26 16:04:24 +03:00
Yiorgis Gozadinos
a822b754b7
Guard mark_succeeded/mark_dead against reaper resurrection
2026-05-26 14:07:44 +03:00
Yiorgis Gozadinos
9cb3ce40ad
Shield cancel-cleanup release in worker pool
2026-05-26 13:57:35 +03:00
Yiorgis Gozadinos
78fc5d0e05
Discover within-root symlinks in FSSource
2026-05-26 13:54:33 +03:00
Yiorgis Gozadinos
e5e0df6ade
Unlink PDF slice tempfiles when slice write fails
2026-05-26 13:46:24 +03:00
Yiorgis Gozadinos
98efd73177
Serialize JobRepo and SyncStateRepo on one shared lock
2026-05-26 13:11:01 +03:00
Yiorgis Gozadinos
5f31a6b12f
Fix round-robin docling-serve
2026-05-26 13:07:21 +03:00
Yiorgis Gozadinos
cf6caf14fe
Tighten retry/reap_stale guards; fix SourceSummary.type leak
2026-05-26 12:38:55 +03:00
Yiorgis Gozadinos
f89cc998eb
add sources_provider to WorkerPool. workers now resolve extra info through these sources
2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
410279cd6c
Preserve sync_state revision when poller bumps last_seen_at
2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
ef9cacf981
Harden FS source against symlink escape; surface pool/poller liveness in /health, additional auth tests
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
4aee18dcbe
Operator dashboard at GET /; tighten Logfire span shape
...
Self-contained HTML status page served from the ingester's FastAPI app.
Polls /health, /sources, /stats, /jobs?status={claimed,dead,succeeded}
every 3s from the browser and renders queue chips, sources with
last-poll/skip-reason/circuit state, active jobs with cancel, recent
failures with retry, and recently-completed feed with op badges so
DELETE rows are visually distinct from UPSERTs. Zero external deps —
single static HTML, no CDN, no fonts, no images. Works offline.
To support the dashboard:
- New /stats endpoint exposing rolling throughput (5m/30m/1h), worker
occupancy, oldest-queued age, and per-source DLQ + queue-depth
breakdowns. Each field is a single SQL aggregation against the queue.
- JobRepo gains count_succeeded_since, oldest_queued_age_seconds,
counts_by_source.
- SourceSummary gains last_skip_reason. BasePoller now records the
reason the most recent sweep attempt was skipped ("pending_work" /
"circuit_open"), cleared on the next successful poll. Closes the
gap where operators couldn't tell from /sources alone why a source
wasn't picking up new work.
Auth: dashboard route is unauthenticated (markup only). The JS attaches
the bearer to its own JSON fetches; on 401 it prompts once and stashes
the token in localStorage.
Two Logfire fixes that landed alongside:
- Drop logfire.instrument_fastapi() and the [fastapi] extra. The control
plane is polled frequently (dashboard + docker healthcheck), so every
endpoint became a span and drowned the useful traces. logfire itself
stays — pulled in transitively via pydantic-ai-slim[logfire] — so
ingester.poller.* / ingester.job / document.* spans keep emitting.
- Wrap FSPoller._handle_watch_change in an ingester.poller.watch_event
span and pass _enqueue_extra. Without this, the watchfiles callback
ran with no active context, the _otel carrier in job.extra was empty,
and the worker's ingester.job span surfaced as an orphan trace root
instead of nesting under the FS event that caused it.
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
5affe70eae
Cover unparseable-metadata path
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
9ca06df745
Bound pdf_split memory + release lock between slices
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
c2f681dd78
constant-time auth, migration short-circuit
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
da3cfe1a58
Release claim on jobs when cancelled
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
710276ffd8
record cassettes against new fixtures, fix stale assertions
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
ed36cc2230
typed UnsupportedSourceError replaces pipeline string-marker matching
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
57e89426ea
regex-based _strip_etag with weak-marker test
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
9ed24ad53e
swap doclaynet.pdf to the full paper, add real-PDF split/merge test
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
6bcc2f6357
PDF split-convert-merge for memory-bound large PDFs
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
ec94426556
HTTPSource.head() returns ETag for the revision short-circuit
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
055fd23d5d
round-robin docling-serve across multiple base_urls
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
5ccbadde0a
Handle shutdown more gracefully, by stopping pollers and cancelling jobs after timeout. Skip periodic poll if a source has pending jobs
2026-05-26 11:44:03 +03:00
Yiorgis Gozadinos
65c309e12b
Handle WebDAV as source
2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
13cbadeb6f
canonical metadata keys: source_revision + content_type, bump to 0.50.0
2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
6d15f1f247
fix URL-decode file:// paths before stat/exists checks
2026-05-26 11:42:42 +03:00
Yiorgis Gozadinos
7ea61a7b10
drop the old monitor, rename serve→mcp, add e2e tests
2026-05-26 11:42:41 +03:00
Yiorgis Gozadinos
de3b3fa1c9
HTTP control plane
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
75c3896588
additional config, pollers, serve
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
bdcc1caa49
worker pool, rety policy, pipeline
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
4b3641244c
SQLite queue + haiku-ingester CLI skeleton
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
717fc3320b
route create_document_from_source through source adapters
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
b9637cd625
add HTTP and S3 source adapters + resolve_fetcher
2026-05-26 11:41:53 +03:00
Yiorgis Gozadinos
d0a730ef60
ingester: add source-adapter scaffolding + FS source
2026-05-26 11:41:53 +03:00
Yiorgis Gozadinos
8e9e84132e
drop list_documents/get_document from default RAG skill
2026-05-26 11:35:43 +03:00
Yiorgis Gozadinos
260b8e8b09
Merge pull request #382 from ggozad/chore/updates
...
Prepare for pydantic-ai 2.0, update haiku.skills
2026-05-21 14:30:55 +03:00
Yiorgis Gozadinos
188d35023f
rotate chat TUI conversation id per launch / clear-chat
2026-05-21 14:21:51 +03:00
Yiorgis Gozadinos
80e3ae0280
clean up deprecation warnings: docling annotations + haiku.skills 0.17.1
2026-05-21 13:45:40 +03:00
Yiorgis Gozadinos
d14bf88096
fix migration failures on pre-v0.48.0 document_items schemas
2026-05-21 10:53:21 +03:00
Yiorgis Gozadinos
b48ac2fdd0
sandbox + cite: guard sort precondition; require rag in cite fallback
2026-05-20 13:02:25 +03:00
Yiorgis Gozadinos
8c57a3ca99
Drop the multi-agent research workflow
2026-05-20 12:46:48 +03:00
Yiorgis Gozadinos
a317a951d9
CLI citations: compact panel, inline figures, doc/chunk IDs in footer
2026-05-20 12:00:24 +03:00
Yiorgis Gozadinos
3858ab905a
toc.json nodes carry chunk_ids; fix cite to accept DB-resolvable chunk_ids
2026-05-20 11:37:35 +03:00
Yiorgis Gozadinos
04eeec77d2
per-doc lazy items/toc cache; index document_items for fast per-doc lookup
2026-05-20 10:08:25 +03:00
Yiorgis Gozadinos
e3314e36fd
search: only attach picture bytes from pre-expansion chunks
2026-05-19 15:03:22 +03:00
Yiorgis Gozadinos
e8c61ad0c6
items.jsonl exposes chunk_ids per row; drops position and tree_depth
2026-05-19 14:20:11 +03:00
Yiorgis Gozadinos
6f95e2bc27
Delete the standalone QA and analysis agents
2026-05-19 11:39:20 +03:00
Yiorgis Gozadinos
947a26b391
client.analyze routes through the rag-analysis skill; drop documents= and AnalysisResult.program. Re-record all cassettes that are relevant
2026-05-19 10:46:32 +03:00
Yiorgis Gozadinos
d96d2eeb0f
client.ask routes through the rag skill; always show citations in CLI
2026-05-19 10:19:39 +03:00
Yiorgis Gozadinos
57d077e2c3
cite raises ModelRetry on unresolved chunk_ids
2026-05-18 16:49:44 +03:00
Yiorgis Gozadinos
84e91097a8
Stop hitting the embedder in test_sandbox_toc fixtures
2026-05-18 16:49:44 +03:00
Yiorgis Gozadinos
3b6bef3bfb
Rename get_captions_for_chunk() to get_text_for_refs()
2026-05-18 16:49:44 +03:00
Yiorgis Gozadinos
74dea65f21
analysis.model inherits qa.model when unset; per-skill vision gate
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
514b0c51f8
surface figure captions in search results, lower search.limit to 5
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
e51841b924
Drop list_documents from analysis skill
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
f13461cdca
route figures through cite, drop show_image
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
387c4f12a9
multimodal sandbox: show_image + picture_refs, drop llm
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
2831030f9b
expose toc.json + heading_level/tree_depth in analysis VFS
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
1360303edf
persist docling heading hierarchy on document_items
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
5de4d50622
coverage
2026-05-14 15:54:43 +03:00
Yiorgis Gozadinos
57f273e000
Minor fixes, CI should build
2026-05-14 15:43:36 +03:00
Yiorgis Gozadinos
519afe6709
Add cross-encoder reranking provider
2026-05-14 15:43:35 +03:00
Yiorgis Gozadinos
dbb87e091f
Ensure schema parity for _StagingChunkRecord
2026-05-14 15:11:18 +03:00
Yiorgis Gozadinos
849eca94d0
Make rebuild --embed-only idempotent across crashes
2026-05-14 15:02:12 +03:00
Yiorgis Gozadinos
a035b0f9e4
Stream rebuild --embed-only through a staging table to bound memory
2026-05-14 15:02:12 +03:00
Yiorgis Gozadinos
704bea6f4a
Stabilize test_client_search ranking assertion
2026-05-14 14:37:27 +03:00
Yiorgis Gozadinos
a98ddc14b8
Replace picture_description.enabled with processing.pictures enum
2026-05-13 16:22:44 +03:00
Yiorgis Gozadinos
bdab201c42
Merge pull request #371 from ggozad/fix/image-results
...
Verify picture bytes before attaching to multimodal tool returns
2026-05-13 16:19:09 +03:00
Yiorgis Gozadinos
a6e5664c78
Verify picture bytes before attaching to multimodal tool returns
2026-05-13 16:01:39 +03:00
Yiorgis Gozadinos
072edad62f
Merge pull request #368 from ggozad/feat/relax-compat-check
...
Relax embedding compatibility check to vector_dim only
2026-05-13 15:43:37 +03:00
Yiorgis Gozadinos
701e9d3632
Relax embedding compat check to vector_dim only
2026-05-13 14:44:18 +03:00
Yiorgis Gozadinos
8b135c4d4d
Expose embedding batch size as config
2026-05-13 14:09:47 +03:00
Yiorgis Gozadinos
c6cd847299
Add ModelConfig.extra_body for raw provider pass-through
2026-05-13 13:45:09 +03:00
Yiorgis Gozadinos
0f4c4af495
Cache the chunking tokenizer to avoid HF Hub 429
2026-05-13 13:32:24 +03:00
Yiorgis Gozadinos
df21286cc9
add tests for mixed <img> sources and IMAGE-format wiring
2026-05-12 15:37:50 +03:00
Yiorgis Gozadinos
afd4665517
Thread source_uri through URL ingest
2026-05-12 15:13:02 +03:00
Yiorgis Gozadinos
a25d9699ee
Wire per-format options across docling-local
2026-05-12 15:02:59 +03:00
Yiorgis Gozadinos
9f8c9fe3b1
cover s3 uri override, unsupported extension, watcher error path
2026-05-11 11:29:56 +03:00
Yiorgis Gozadinos
6631e339ee
record VCR cassettes for s3 source tests
2026-05-11 11:29:56 +03:00
Yiorgis Gozadinos
ef82e6eb08
swap aioboto3 for obstore in the S3 client path
2026-05-11 11:29:42 +03:00
