Commit graph

126 commits

Author SHA1 Message Date
Yiorgis Gozadinos
8b45f81464
Load only content when listing documents with include_content 2026-07-31 10:12:19 +03:00
Yiorgis Gozadinos
5200b71e2b
Correct two justifications
Drop the coverage-gate CHANGELOG entry: it is dev infrastructure, invisible
to anyone upgrading. Broaden the check_source_accessible entry to the cases
it now covers.

The nameless-attachment guard reads an empty /F out of untrusted PDF bytes,
so it is boundary validation, not a formality; only the reason it goes
uncovered belongs in the pragma.
2026-07-27 12:18:15 +03:00
Yiorgis Gozadinos
f96a428ef1
Fix defects found reviewing the coverage work
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.

Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.

Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.

Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
2026-07-27 10:44:32 +03:00
Yiorgis Gozadinos
7ba78fdde3
Normalize pragma comments to the codebase's single-line form 2026-07-26 20:18:19 +03:00
Yiorgis Gozadinos
f6acb65e95
Reach and enforce 100% coverage
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.

Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.

Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
2026-07-26 20:11:02 +03:00
Yiorgis Gozadinos
dddaf0f84c
Remove unreachable code and fix two defects it surfaced
Delete repository methods with no callers (SettingsRepository CRUD,
ChunkRepository.update/delete/get_chunks_in_range), the DataFrame branch of
_process_search_results whose only caller always passes a query, and guards
that cannot be reached from their call sites: the rebuild mode=None default,
the staging drop already performed by _resolve_rebuild_recovery, the empty
batch skip, two context fast paths, the doctor prefix guard, the poller
_task attribute that is never assigned, and a docling caption fallback for
a field name no item class defines.

set_haiku_version built a recreated settings row from the process-global
Config rather than the store's own, so a store opened with a custom config
stamped global settings into the database.

check_source_accessible called urlparse outside its try block, so a stored
URI with a malformed IPv6 host raised ValueError instead of reporting the
source as inaccessible, aborting the whole rebuild sweep.
2026-07-26 18:45:48 +03:00
Yiorgis Gozadinos
4c5050d161
Accept images on ask/analyze 2026-07-25 10:23:55 +03:00
Yiorgis Gozadinos
77585ef26a
fix per-question capability limits and tool isolation 2026-07-24 15:26:18 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
543aba7547
Multimodal reranking: send picture chunks to vllm rerankers as images
reranking.multimodal (vllm provider only) attaches picture bytes to
synthetic picture chunks before rerank; VLLMReranker sends them as
content-parts documents (base64 data URI + description text) in the
same /v1/rerank request as plain text documents.
2026-07-24 12:29:20 +03:00
Yiorgis Gozadinos
1ed424b50d
Serialize metadata-only document updates with the write lock 2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
0e271eaf4b
Remove --before/--at time travel 2026-07-16 13:11:33 +03:00
Yiorgis Gozadinos
0813a1c980
Add tag CLI commands and history tag annotations 2026-07-16 13:11:27 +03:00
Yiorgis Gozadinos
0cbde6b7a2
Serialize tag operations, vacuum, and metadata refresh with writes 2026-07-16 12:32:57 +03:00
Yiorgis Gozadinos
144900d385
Remove the mxbai reranking provider 2026-07-14 11:09:55 +03:00
Yiorgis Gozadinos
e7d5a0440d
Tune two-tone visualization colors 2026-07-09 15:37:25 +03:00
Yiorgis Gozadinos
18c0f6c5e8
Add --no-expand to the visualize command for chunk-only grounding 2026-07-09 12:49:42 +03:00
Yiorgis Gozadinos
995d081aa1
Visualize the exact context the model saw via Citation.doc_item_refs
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
2026-07-09 11:51:37 +03:00
Yiorgis Gozadinos
6d86237dd6
Draw matched content stronger than expanded context in visualizations 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking 2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
43a580afb3
Dedupe picture chunks and skip small pictures at chunking 2026-07-09 10:24:32 +03:00
Yiorgis Gozadinos
a1cf405cae
Rename document_meta identity column document_id to id 2026-07-08 16:32:55 +03:00
Yiorgis Gozadinos
23bbca5998
Re-check URI under the write lock to prevent duplicate ingestion 2026-07-08 11:26:45 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document 2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
d2593304c5
Attach figure bytes when a result matches its caption 2026-07-03 13:22:02 +03:00
Yiorgis Gozadinos
2b2b475279
Collapse docling compression to a single function 2026-06-24 10:43:27 +03:00
Yiorgis Gozadinos
a1ec310bf4
Add VoyageAI multimodal embedder 2026-06-23 15:23:10 +03:00
Yiorgis Gozadinos
8cb52ca1ed
Merge pull request #459 from mcdonc/thread-rebuild-compress
Thread compress_docling_split in rebuild.py off the asyncio event loop
2026-06-23 09:23:53 +03:00
Yiorgis Gozadinos
775dc58b53
Drop redundant _apply_descriptions_sync unit test, annotate the helper 2026-06-23 09:14:36 +03:00
Yiorgis Gozadinos
0ee36a269d
Merge pull request #460 from mcdonc/thread-processing-iterate
Thread picture chunk merging off the asyncio event loop
2026-06-23 09:05:45 +03:00
Yiorgis Gozadinos
141e288b83
Tighten _merge_picture_chunks type annotations
Restore the typing the merge logic carried inline before extraction:
text_chunks: list[Chunk], existing_picture_data: dict[str, bytes] | None,
-> list[Chunk], and first_pos(c: Chunk) -> int.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 08:56:51 +03:00
Yiorgis Gozadinos
4436673f10
Tighten extract_items hoist annotation in _update_document_with_chunks 2026-06-23 08:47:55 +03:00
Chris McDonough
ccab012747 Thread picture chunk merging off the asyncio event loop
Extracts build_picture_chunks + iterate_items position mapping into a
sync helper and wraps it with asyncio.to_thread so image-heavy
documents don't block the event loop during chunking.

Fixes #456.
2026-06-22 14:01:35 -04:00
Chris McDonough
cee9484f35 Fix import sort order 2026-06-22 13:49:59 -04:00
Chris McDonough
d160080ff8 Move lazy imports to module scope 2026-06-22 13:48:09 -04:00
Chris McDonough
97748c2fee Fix import sorting lint error 2026-06-22 13:46:21 -04:00
Chris McDonough
22b4397148 Thread compress_docling_split in rebuild.py off the asyncio event loop
Extracts pydantic serialization and zstd compression into a sync
helper and wraps it with asyncio.to_thread so picture description
rebuilds don't block the event loop.

Fixes #457.
2026-06-22 13:42:40 -04:00
Chris McDonough
09e78ef46a Thread extract_items off the asyncio event loop
Moves CPU-bound docling document item extraction into worker threads
via asyncio.to_thread so the event loop stays responsive during
ingestion. The extract_items calls are hoisted out of the write lock
(they are pure computation with no DB I/O) and run with placeholder
document IDs that are patched after the DB create returns.

Fixes #453.
2026-06-22 13:19:38 -04:00
Yiorgis Gozadinos
d3a1011baf
write fetched bodies off the event loop 2026-06-22 10:37:48 +03:00
Yiorgis Gozadinos
183d595494
prepare stored Docling blobs off the event loop 2026-06-22 10:30:02 +03:00
bryan davis
faa97f8bc6
improve concurrency management for pdf attachments 2026-06-18 14:30:02 -05:00
Yiorgis Gozadinos
1e65d36766
Pass fetched FetchResult to ingester metadata providers 2026-06-16 16:37:26 +03:00
Yiorgis Gozadinos
70c9c2778f
Serialize all in-process pdfium access under a shared lock 2026-06-14 10:50:42 +03:00
Yiorgis Gozadinos
2b8b9477b7
Serialize and roll back the multi-table document delete 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
f2a5ac4246
Throttle background auto-vacuum to at most once per 5 minutes 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
5d5d87d44c
Throttle background auto-vacuum to at most once per 5 minutes 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
3366a6d383
Split meta document attributes into a document_meta table 2026-06-12 10:17:17 +03:00
Yiorgis Gozadinos
4bc52f0710
Serialize multi-table document writes and bound update version churn 2026-06-12 10:16:39 +03:00
Yiorgis Gozadinos
8ac670714d
Tidy PDF attachment extension fix 2026-06-12 09:44:11 +03:00