Yiorgis Gozadinos
5a23e4eda6
Surface failed discovery sweeps in run-batch
2026-06-01 10:27:31 +03:00
Yiorgis Gozadinos
6f2a40c676
cover run-batch, serve, and _stop_pool with tests
2026-05-29 17:43:58 +03:00
Yiorgis Gozadinos
5ba7838b71
Add haiku-ingester run-batch, remove run-once
2026-05-29 17:00:59 +03:00
Yiorgis Gozadinos
62be23d130
vb
2026-05-29 11:47:12 +03:00
Yiorgis Gozadinos
e7c7df2915
Always use the Store-owned embedder
2026-05-29 11:36:53 +03:00
Yiorgis Gozadinos
37a78a4e9b
Cache reranker on the client instead of rebuilding per search
2026-05-29 10:33:50 +03:00
Yiorgis Gozadinos
402957d3e5
Fix CI flakes for PDF attachment extraction
2026-05-28 17:51:02 +03:00
Yiorgis Gozadinos
8e8c4433bd
Extract PDF /EmbeddedFiles attachments as child documents
2026-05-28 15:51:32 +03:00
Yiorgis Gozadinos
26bc71d6d8
Cascade delete_document to children via metadata.parent_uri
2026-05-28 15:36:00 +03:00
Yiorgis Gozadinos
5400085147
Auto-prune dead jobs when a sibling DELETE succeeds
2026-05-27 17:17:01 +03:00
Yiorgis Gozadinos
4d1b89de8f
Custom hover tooltips for truncated dashboard cells
2026-05-27 15:42:17 +03:00
Yiorgis Gozadinos
1e6dc34871
Skip FS DELETE enqueue when the file is already back
2026-05-27 15:18:46 +03:00
Yiorgis Gozadinos
15d13cab04
Probe docling-serve only when converter or chunker uses it
2026-05-27 15:18:25 +03:00
Yiorgis Gozadinos
b0d0ac588d
Split snapshot APIs and resolve_fetcher by intent
2026-05-27 14:39:54 +03:00
Yiorgis Gozadinos
2c63ceda0c
Backfill ingester test gaps and drop unneeded retry clamp
2026-05-27 14:24:22 +03:00
Yiorgis Gozadinos
cfbd0b09d6
Tighten HTTP config-removal handling
2026-05-27 14:07:45 +03:00
Yiorgis Gozadinos
7466539b4f
Emit DELETE for HTTP URLs removed from config
2026-05-27 13:39:18 +03:00
Yiorgis Gozadinos
d7fdd61fed
Resolve worker source by source_id, not just supports(uri)
2026-05-27 13:36:03 +03:00
Yiorgis Gozadinos
922d1d567d
Prevent DELETE/UPSERT race for the same URI
2026-05-27 13:30:46 +03:00
Yiorgis Gozadinos
1c710433c7
Collapse worker_count and max_concurrent into worker_count
2026-05-27 12:48:14 +03:00
Yiorgis Gozadinos
6f74fe67b3
Cap claimed-row count at max_concurren
2026-05-27 12:10:42 +03:00
Yiorgis Gozadinos
87af5e5139
Boot-reap stale claims at WorkerPool start
2026-05-27 11:46:37 +03:00
Yiorgis Gozadinos
1b36452629
Surface provider reachability and per-job failure context on dashboard
2026-05-27 11:38:48 +03:00
Yiorgis Gozadinos
439307d5af
Add pool-wide circuit breaker to the worker pool
2026-05-27 10:52:48 +03:00
Yiorgis Gozadinos
b491f767f3
Let httpx errors flow through DoclingServeClient
2026-05-26 17:38:49 +03:00
Yiorgis Gozadinos
fdb73fc41f
Drop defensive branches
2026-05-26 16:56:32 +03:00
Yiorgis Gozadinos
291b565b4c
Stream dashboard index.html via FileResponse
2026-05-26 16:31:41 +03:00
Yiorgis Gozadinos
29b3ccdbda
Drain orphan cancel-cleanup releases before closing the queue
2026-05-26 16:14:17 +03:00
Yiorgis Gozadinos
3c608046e2
Bound ?limit and ?offset on /jobs and /dlq
2026-05-26 16:05:47 +03:00
Yiorgis Gozadinos
761956eb70
Extend reaper-resurrection guard to reschedule and release_if_claimed
2026-05-26 16:04:24 +03:00
Yiorgis Gozadinos
6799a4a6d4
Document worker-pool settings
2026-05-26 14:13:16 +03:00
Yiorgis Gozadinos
a822b754b7
Guard mark_succeeded/mark_dead against reaper resurrection
2026-05-26 14:07:44 +03:00
Yiorgis Gozadinos
9cb3ce40ad
Shield cancel-cleanup release in worker pool
2026-05-26 13:57:35 +03:00
Yiorgis Gozadinos
78fc5d0e05
Discover within-root symlinks in FSSource
2026-05-26 13:54:33 +03:00
Yiorgis Gozadinos
e5e0df6ade
Unlink PDF slice tempfiles when slice write fails
2026-05-26 13:46:24 +03:00
Yiorgis Gozadinos
98efd73177
Serialize JobRepo and SyncStateRepo on one shared lock
2026-05-26 13:11:01 +03:00
Yiorgis Gozadinos
5f31a6b12f
Fix round-robin docling-serve
2026-05-26 13:07:21 +03:00
Yiorgis Gozadinos
cf6caf14fe
Tighten retry/reap_stale guards; fix SourceSummary.type leak
2026-05-26 12:38:55 +03:00
Yiorgis Gozadinos
f89cc998eb
add sources_provider to WorkerPool. workers now resolve extra info through these sources
2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
410279cd6c
Preserve sync_state revision when poller bumps last_seen_at
2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
083ff401d5
vacuum db after migration
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
73f8349a00
Use haiku.rag as telemtry scope
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
d900651b92
Render "queue busy" badge on sources skipped due to pending_work
...
Replaces the confusing em-dash + raw `pending_work` text when no successful sweep has run yet.
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
ef9cacf981
Harden FS source against symlink escape; surface pool/poller liveness in /health, additional auth tests
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
4aee18dcbe
Operator dashboard at GET /; tighten Logfire span shape
...
Self-contained HTML status page served from the ingester's FastAPI app.
Polls /health, /sources, /stats, /jobs?status={claimed,dead,succeeded}
every 3s from the browser and renders queue chips, sources with
last-poll/skip-reason/circuit state, active jobs with cancel, recent
failures with retry, and recently-completed feed with op badges so
DELETE rows are visually distinct from UPSERTs. Zero external deps —
single static HTML, no CDN, no fonts, no images. Works offline.
To support the dashboard:
- New /stats endpoint exposing rolling throughput (5m/30m/1h), worker
occupancy, oldest-queued age, and per-source DLQ + queue-depth
breakdowns. Each field is a single SQL aggregation against the queue.
- JobRepo gains count_succeeded_since, oldest_queued_age_seconds,
counts_by_source.
- SourceSummary gains last_skip_reason. BasePoller now records the
reason the most recent sweep attempt was skipped ("pending_work" /
"circuit_open"), cleared on the next successful poll. Closes the
gap where operators couldn't tell from /sources alone why a source
wasn't picking up new work.
Auth: dashboard route is unauthenticated (markup only). The JS attaches
the bearer to its own JSON fetches; on 401 it prompts once and stashes
the token in localStorage.
Two Logfire fixes that landed alongside:
- Drop logfire.instrument_fastapi() and the [fastapi] extra. The control
plane is polled frequently (dashboard + docker healthcheck), so every
endpoint became a span and drowned the useful traces. logfire itself
stays — pulled in transitively via pydantic-ai-slim[logfire] — so
ingester.poller.* / ingester.job / document.* spans keep emitting.
- Wrap FSPoller._handle_watch_change in an ingester.poller.watch_event
span and pass _enqueue_extra. Without this, the watchfiles callback
ran with no active context, the _otel carrier in job.extra was empty,
and the worker's ingester.job span surfaced as an orphan trace root
instead of nesting under the FS event that caused it.
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
5affe70eae
Cover unparseable-metadata path
2026-05-26 11:44:47 +03:00
Yiorgis Gozadinos
9ca06df745
Bound pdf_split memory + release lock between slices
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
c2f681dd78
constant-time auth, migration short-circuit
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
da3cfe1a58
Release claim on jobs when cancelled
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
14beba6786
Lock pypdfium2 under concurrent workers; per-slice convert spans
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
ba623b0862
mark event-loop glue + defensive guards as no-cover
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
ed36cc2230
typed UnsupportedSourceError replaces pipeline string-marker matching
2026-05-26 11:44:46 +03:00
Yiorgis Gozadinos
57e89426ea
regex-based _strip_etag with weak-marker test
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
6bcc2f6357
PDF split-convert-merge for memory-bound large PDFs
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
f4468b65ee
Add --host, --port to ingester cli, update docker compose example
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
ec94426556
HTTPSource.head() returns ETag for the revision short-circuit
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
055fd23d5d
round-robin docling-serve across multiple base_urls
2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
5ccbadde0a
Handle shutdown more gracefully, by stopping pollers and cancelling jobs after timeout. Skip periodic poll if a source has pending jobs
2026-05-26 11:44:03 +03:00
Yiorgis Gozadinos
65c309e12b
Handle WebDAV as source
2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
13cbadeb6f
canonical metadata keys: source_revision + content_type, bump to 0.50.0
2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
fa672c298a
logfire spans for the ingester, nested poller/job/document traces
2026-05-26 11:42:42 +03:00
Yiorgis Gozadinos
6d15f1f247
fix URL-decode file:// paths before stat/exists checks
2026-05-26 11:42:42 +03:00
Yiorgis Gozadinos
7ea61a7b10
drop the old monitor, rename serve→mcp, add e2e tests
2026-05-26 11:42:41 +03:00
Yiorgis Gozadinos
1ca3c25a83
logging, sqlite cursors
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
de3b3fa1c9
HTTP control plane
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
75c3896588
additional config, pollers, serve
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
bdcc1caa49
worker pool, rety policy, pipeline
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
4b3641244c
SQLite queue + haiku-ingester CLI skeleton
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
717fc3320b
route create_document_from_source through source adapters
2026-05-26 11:41:54 +03:00
Yiorgis Gozadinos
b9637cd625
add HTTP and S3 source adapters + resolve_fetcher
2026-05-26 11:41:53 +03:00
Yiorgis Gozadinos
d0a730ef60
ingester: add source-adapter scaffolding + FS source
2026-05-26 11:41:53 +03:00
Yiorgis Gozadinos
8e9e84132e
drop list_documents/get_document from default RAG skill
2026-05-26 11:35:43 +03:00
Yiorgis Gozadinos
14d9951016
vb
2026-05-21 14:32:13 +03:00
Yiorgis Gozadinos
260b8e8b09
Merge pull request #382 from ggozad/chore/updates
...
Prepare for pydantic-ai 2.0, update haiku.skills
2026-05-21 14:30:55 +03:00
Yiorgis Gozadinos
188d35023f
rotate chat TUI conversation id per launch / clear-chat
2026-05-21 14:21:51 +03:00
Yiorgis Gozadinos
80e3ae0280
clean up deprecation warnings: docling annotations + haiku.skills 0.17.1
2026-05-21 13:45:40 +03:00
Yiorgis Gozadinos
392c74039b
bump pydantic-ai-slim to 1.100 and haiku.skills to 0.17
...
Migrate off two APIs slated for removal in pydantic-ai 2.0:
- Agent(tool_retries=, output_retries=) -> Agent(retries={"tools": ...,
"output": ...}) in the LLM-as-judge evaluator.
- Evaluator.evaluation_name class attribute -> overriding
get_default_evaluation_name() on the citation MRR / MAP evaluators.
haiku.skills 0.17.0 already migrated its internal AGUIAdapter,
MCPToolset and ProcessEventStream usage; no further changes needed on
our side beyond the pin bumps.
2026-05-21 13:20:54 +03:00
Yiorgis Gozadinos
d14bf88096
fix migration failures on pre-v0.48.0 document_items schemas
2026-05-21 10:53:21 +03:00
Yiorgis Gozadinos
6690953480
fix mxbai reranker crash in chat TUI
2026-05-20 17:18:57 +03:00
Yiorgis Gozadinos
b48ac2fdd0
sandbox + cite: guard sort precondition; require rag in cite fallback
2026-05-20 13:02:25 +03:00
Yiorgis Gozadinos
f9392cee46
evaluations: judge model moves to config.evaluations.judge
2026-05-20 12:56:22 +03:00
Yiorgis Gozadinos
8c57a3ca99
Drop the multi-agent research workflow
2026-05-20 12:46:48 +03:00
Yiorgis Gozadinos
a317a951d9
CLI citations: compact panel, inline figures, doc/chunk IDs in footer
2026-05-20 12:00:24 +03:00
Yiorgis Gozadinos
3858ab905a
toc.json nodes carry chunk_ids; fix cite to accept DB-resolvable chunk_ids
2026-05-20 11:37:35 +03:00
Yiorgis Gozadinos
04eeec77d2
per-doc lazy items/toc cache; index document_items for fast per-doc lookup
2026-05-20 10:08:25 +03:00
Yiorgis Gozadinos
e3314e36fd
search: only attach picture bytes from pre-expansion chunks
2026-05-19 15:03:22 +03:00
Yiorgis Gozadinos
e8c61ad0c6
items.jsonl exposes chunk_ids per row; drops position and tree_depth
2026-05-19 14:20:11 +03:00
Yiorgis Gozadinos
6f95e2bc27
Delete the standalone QA and analysis agents
2026-05-19 11:39:20 +03:00
Yiorgis Gozadinos
947a26b391
client.analyze routes through the rag-analysis skill; drop documents= and AnalysisResult.program. Re-record all cassettes that are relevant
2026-05-19 10:46:32 +03:00
Yiorgis Gozadinos
d96d2eeb0f
client.ask routes through the rag skill; always show citations in CLI
2026-05-19 10:19:39 +03:00
Yiorgis Gozadinos
57d077e2c3
cite raises ModelRetry on unresolved chunk_ids
2026-05-18 16:49:44 +03:00
Yiorgis Gozadinos
87c551c892
Cover 0.48.0 migration
2026-05-18 16:49:44 +03:00
Yiorgis Gozadinos
3b6bef3bfb
Rename get_captions_for_chunk() to get_text_for_refs()
2026-05-18 16:49:44 +03:00
Yiorgis Gozadinos
74dea65f21
analysis.model inherits qa.model when unset; per-skill vision gate
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
52e93d08eb
Explain why no images in sandbox search(). We might want to add show_image() back in the future.
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
c562c397f9
move analysis skill toward search-first; document VFS doc_id vs uri
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
514b0c51f8
surface figure captions in search results, lower search.limit to 5
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
7d99be471f
Add tree-sitter-json on tui
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
e51841b924
Drop list_documents from analysis skill
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
68388032be
consolidate #/pictures/ prefix, tighten CHANGELOG, log migration exc_info
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
f13461cdca
route figures through cite, drop show_image
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
387c4f12a9
multimodal sandbox: show_image + picture_refs, drop llm
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
2831030f9b
expose toc.json + heading_level/tree_depth in analysis VFS
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
1360303edf
persist docling heading hierarchy on document_items
2026-05-18 16:48:34 +03:00
Yiorgis Gozadinos
f208730a5b
Merge pull request #374 from tseaver/feat-cli-search_type
...
feat: add '--search-type' option to CLI 'search'
2026-05-18 16:39:28 +03:00
Tres Seaver
8026693450
chore: ruff format
2026-05-18 09:31:38 -04:00
Tres Seaver
ad0161e6da
chore: isort
2026-05-18 09:31:12 -04:00
Tres Seaver
f9ce3bc401
refactor: move 'SearchType' to 'store.models.chunk'
2026-05-18 09:27:34 -04:00
Tres Seaver
d7e96f2658
fix: typo
2026-05-18 09:17:53 -04:00
Tres Seaver
4feac9683a
fix: default 'search_type' to None
2026-05-18 09:17:09 -04:00
Yiorgis Gozadinos
04415a9d09
bump pydantic-monty, refresh deps, cap transformers<5
2026-05-18 16:02:55 +03:00
Yiorgis Gozadinos
1df6530d03
Bump pydantic-ai, prepare for 2.*
2026-05-18 15:14:29 +03:00
Yiorgis Gozadinos
b1e8bdc839
Bump docling
2026-05-18 15:14:29 +03:00
Tres Seaver
b331dd0b76
fix: default 'search_type' to 'None' in 'client.search.search'
...
Apply 'hybrid' default only for text queries.
2026-05-18 06:52:03 -04:00
Tres Seaver
5079205d3d
fix: use 'SearchType' for 'ChunkRepository.search'
2026-05-18 06:50:49 -04:00
Tres Seaver
7953de46b9
fix: declare 'SearchType' for 'search_type' arg
...
fix: allow 'search_type' only for text searches
2026-05-18 06:35:51 -04:00
Tres Seaver
b2ffc50168
WIP: add '--search-type' option to CLI 'search'
2026-05-18 06:35:50 -04:00
Yiorgis Gozadinos
97a7eb246c
vb
2026-05-14 16:03:33 +03:00
Yiorgis Gozadinos
5de4d50622
coverage
2026-05-14 15:54:43 +03:00
Yiorgis Gozadinos
57f273e000
Minor fixes, CI should build
2026-05-14 15:43:36 +03:00
Yiorgis Gozadinos
519afe6709
Add cross-encoder reranking provider
2026-05-14 15:43:35 +03:00
Yiorgis Gozadinos
dbb87e091f
Ensure schema parity for _StagingChunkRecord
2026-05-14 15:11:18 +03:00
Yiorgis Gozadinos
849eca94d0
Make rebuild --embed-only idempotent across crashes
2026-05-14 15:02:12 +03:00
Yiorgis Gozadinos
a035b0f9e4
Stream rebuild --embed-only through a staging table to bound memory
2026-05-14 15:02:12 +03:00
Yiorgis Gozadinos
862a209137
vb
2026-05-13 16:53:35 +03:00
Yiorgis Gozadinos
a98ddc14b8
Replace picture_description.enabled with processing.pictures enum
2026-05-13 16:22:44 +03:00
Yiorgis Gozadinos
bdab201c42
Merge pull request #371 from ggozad/fix/image-results
...
Verify picture bytes before attaching to multimodal tool returns
2026-05-13 16:19:09 +03:00
Yiorgis Gozadinos
a6e5664c78
Verify picture bytes before attaching to multimodal tool returns
2026-05-13 16:01:39 +03:00
Yiorgis Gozadinos
072edad62f
Merge pull request #368 from ggozad/feat/relax-compat-check
...
Relax embedding compatibility check to vector_dim only
2026-05-13 15:43:37 +03:00
Yiorgis Gozadinos
701e9d3632
Relax embedding compat check to vector_dim only
2026-05-13 14:44:18 +03:00
Yiorgis Gozadinos
8b135c4d4d
Expose embedding batch size as config
2026-05-13 14:09:47 +03:00
Yiorgis Gozadinos
c6cd847299
Add ModelConfig.extra_body for raw provider pass-through
2026-05-13 13:45:09 +03:00
Yiorgis Gozadinos
0f4c4af495
Cache the chunking tokenizer to avoid HF Hub 429
2026-05-13 13:32:24 +03:00
Yiorgis Gozadinos
d512f9fbd1
Disable Typer locals in CLI tracebacks
2026-05-13 13:24:05 +03:00
Yiorgis Gozadinos
bf1cd68f64
Preserve cause chain on docling wrapper exceptions
2026-05-13 12:18:53 +03:00
Yiorgis Gozadinos
674b80e8c9
Support streaming for markdown
2026-05-12 16:01:42 +03:00
Yiorgis Gozadinos
afd4665517
Thread source_uri through URL ingest
2026-05-12 15:13:02 +03:00
Yiorgis Gozadinos
a25d9699ee
Wire per-format options across docling-local
2026-05-12 15:02:59 +03:00
Yiorgis Gozadinos
ef82e6eb08
swap aioboto3 for obstore in the S3 client path
2026-05-11 11:29:42 +03:00
Yiorgis Gozadinos
37099988a6
poll S3 prefixes with S3Watcher and wire into serve
2026-05-11 11:28:40 +03:00
Yiorgis Gozadinos
d009da06d6
support s3:// document sources
2026-05-11 11:28:31 +03:00
Yiorgis Gozadinos
387032511e
Simplify
2026-05-08 12:11:25 +03:00
Yiorgis Gozadinos
788fac422e
Clean up migration
2026-05-08 11:52:57 +03:00
Yiorgis Gozadinos
e55e3d13c8
fix _patch_picture_descriptions silently dropping docling_pages
2026-05-06 12:57:41 +03:00
Yiorgis Gozadinos
75f75ec505
render attached pictures in inspector context modal under qa.model.vision
2026-05-06 12:00:12 +03:00
Yiorgis Gozadinos
7fac35d2af
rename embed_image_query to embed_image; run description check on text path
2026-05-05 16:15:47 +03:00
Yiorgis Gozadinos
1ccb5b5fad
auto-append /v1 to vllm base_url, matching ollama behavior
2026-05-05 16:08:14 +03:00
Yiorgis Gozadinos
4478b7ce2b
test v0.45.0 migration
2026-05-05 16:06:14 +03:00
Yiorgis Gozadinos
c96894a0f0
fix cross-document picture dedup in agent and skill search tools
2026-05-05 15:56:10 +03:00
Yiorgis Gozadinos
61a88edfd8
drop bogus deprecation warning for processing.pictures
2026-05-05 14:46:41 +03:00
Yiorgis Gozadinos
49274ab51b
attach picture bytes through skill search and document the schema
2026-05-05 14:33:57 +03:00
Yiorgis Gozadinos
ffc7b95375
yield per document during rebuild for live progress reporting
2026-05-05 14:04:37 +03:00
Yiorgis Gozadinos
a45d82f206
Update benchmark
2026-05-05 13:58:48 +03:00
Yiorgis Gozadinos
dca3188a48
hydrate documents lazily during rebuild
2026-05-05 12:47:45 +03:00
Yiorgis Gozadinos
ff82d36c2f
add rebuild --descriptions: run VLM over stored picture bytes only
2026-05-05 12:02:46 +03:00
Yiorgis Gozadinos
5fad0aa5c6
warn when picture descriptions silently fail to come back
2026-05-05 11:44:18 +03:00
Yiorgis Gozadinos
2038d43435
collapse pictures enum to picture_description.enabled boolean
2026-05-05 11:16:08 +03:00
Yiorgis Gozadinos
e88321767d
always store picture bytes; collapse converter to a single zip path
2026-05-05 10:40:40 +03:00
Yiorgis Gozadinos
ee41bc676f
remove defensive code
2026-05-04 15:12:27 +03:00
Yiorgis Gozadinos
e8d89aa035
dedup picture-only chunks at the search-result layer
2026-05-04 13:31:15 +03:00
Yiorgis Gozadinos
aa3e9406cf
auto-append /v1 to per-model Ollama base_url, fix flaky tests
2026-05-04 13:12:37 +03:00
Yiorgis Gozadinos
65d9c74224
expose image-as-query through MCP and the CLI.
2026-05-04 11:46:45 +03:00
Yiorgis Gozadinos
ff656504d3
make client.search() polymorphic on query type: str | bytes | PIL.Image.Image. Bytes/PIL queries embed via embed_image_query and run vector-only against the chunks table
2026-05-04 11:22:22 +03:00
Yiorgis Gozadinos
21f12cd770
add vision: bool flag on ModelConfig (default False). Gate the agent search tool's BinaryContent attachment on qa.model.vision so picture bytes are only sent to vision-capable QA models.
2026-05-04 10:47:21 +03:00
Yiorgis Gozadinos
6037f26673
drop mlx
2026-05-03 19:05:49 +03:00
Yiorgis Gozadinos
c9fdad15a9
bump docling-core to >=2.74.1 so that local docling-local and docling-serve chunkers run the same MarkdownTableSerializer
2026-05-03 18:40:44 +03:00
Yiorgis Gozadinos
c9227c649f
emit synthetic picture chunks at ingest under multimodal embedders.
...
processing.chunk() merges text chunks with one synthetic Chunk per PictureItem-with-bytes,
sorted by iterate_items() position so chunk.order is structural.
embed_chunks dispatches on a Chunk._picture_data PrivateAttr
(text through embed_documents, picture through embed_image_query)
2026-05-03 17:04:43 +03:00
Yiorgis Gozadinos
a36f920f5d
drop the batched embed_images method from multi-modal embedders
2026-05-03 16:21:47 +03:00
Yiorgis Gozadinos
37215f1a00
add multimodal embedder support for vllm and mlx
2026-05-03 10:57:56 +03:00
Yiorgis Gozadinos
75c8e83515
Allow uri override on create_document_from_source
2026-04-30 16:36:35 +03:00
Yiorgis Gozadinos
4c9ef81fc6
drop the existing-row-not-found guard from the 0.45.0 picture-bytes migration.
2026-04-30 15:48:03 +03:00
Yiorgis Gozadinos
b290bbe82d
rename document_items repository's _LIGHT_COLUMNS to _METADATA_COLUMNS
2026-04-30 15:43:03 +03:00
Yiorgis Gozadinos
875c635a77
have rebuild --rechunk re-chunk from the stored docling blob instead of re-converting from the markdown export
2026-04-30 15:30:24 +03:00
Yiorgis Gozadinos
7831057339
add processing.pictures enum (none|description|image), replacing the implicit pair of generate_picture_images+picture_description.enabled flags
2026-04-30 14:26:14 +03:00
Yiorgis Gozadinos
b01c649684
surface picture image bytes in SearchResult and emit multimodal ToolReturn from the agent search tool
2026-04-30 12:43:05 +03:00
Yiorgis Gozadinos
6a77ce92a9
extract picture bytes to document_items.picture_data at ingest, strip them from the docling_document blob, and add 0.45.0 migration to backfill existing
...
databases
2026-04-30 11:26:37 +03:00
Yiorgis Gozadinos
191cdc636b
retrieve picture image bytes via referenced+zip path, fix docling-serve not returning images even when set to do so. Remove xfail from relevant test
2026-04-30 11:23:29 +03:00
Yiorgis Gozadinos
dc16f74b58
add picture_data column to document_items
2026-04-30 11:23:29 +03:00
Yiorgis Gozadinos
29bc42514e
vb
2026-04-29 13:45:36 +03:00
Yiorgis Gozadinos
5f08a2a8ae
Adapt rag SKILL.md in order to improve citation rate
2026-04-28 14:51:28 +03:00
Yiorgis Gozadinos
7685a592c2
bump haiku.skills to 0.16.0
2026-04-28 12:03:14 +03:00
Yiorgis Gozadinos
6807f082ec
vb
2026-04-25 09:35:47 +03:00
Tres Seaver
0d70f4aec6
fix: pass absolute 'db_path' to 'lancedb.connect_async'
...
Closes #354 .
2026-04-24 17:59:01 -04:00
Yiorgis Gozadinos
81ba248483
vb
2026-04-24 17:47:57 +03:00
Yiorgis Gozadinos
821b7361e9
Push document-id filter into chunk search query
2026-04-24 16:07:07 +03:00
Yiorgis Gozadinos
31fc22ce9d
List tables consistently
2026-04-24 15:57:07 +03:00
Yiorgis Gozadinos
cd4224a727
drop private title/embedding forwarders from HaikuRAG
2026-04-24 15:52:50 +03:00
Yiorgis Gozadinos
b23bad32c3
Check table_names() before dropping chunks in recreate_embeddings_table
2026-04-24 15:32:06 +03:00
Yiorgis Gozadinos
729899299b
Log post-rebuild vacuum failures and clarify vacuum-await scope
2026-04-24 15:31:36 +03:00
Yiorgis Gozadinos
258ffa41fa
Improve test coverage
2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
aca3ccc29f
Close connections if __aenter__ fails mid-initialization
2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
3224d1f20c
Extract document orchestration into client/documents.py
2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
cb02a2c3d8
Extract rebuild machinery into client/rebuild.py
2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
528acb79a3
Extract LLM agents into client/agents.py
2026-04-24 14:43:58 +03:00
Yiorgis Gozadinos
12939f1444
Extract search, expand_context, visualize_chunk into client/search.py
2026-04-24 14:43:58 +03:00
Yiorgis Gozadinos
d4ecebf896
Extract processing primitives into client/processing.py
2026-04-24 14:42:53 +03:00
Yiorgis Gozadinos
b5e480bfa1
Extract title generation into client/titles.py
2026-04-24 14:42:53 +03:00
Yiorgis Gozadinos
e814459e1f
Extract download_models into client/downloads.py
2026-04-24 14:42:53 +03:00
Yiorgis Gozadinos
43f2774c14
Turn client.py into a client/ package
2026-04-24 14:42:53 +03:00
Yiorgis Gozadinos
d0ed9e213f
Pin FTS search column to content_fts
2026-04-24 14:42:53 +03:00
