Commit graph

86 commits

Author SHA1 Message Date
Yiorgis Gozadinos
cc04f92f28
Batch embeddings across import_documents batches
_store_documents_with_chunks embedded each document's chunks in its own
embed_chunks call; chunks missing embeddings are now flattened across the
whole batch, embedded in one pass honoring embeddings.batch_size, and
assigned back positionally.
2026-08-17 10:52:11 +03:00
Yiorgis Gozadinos
ac9b2cbf81
Load document blobs only when asked 2026-08-06 13:04:51 +03:00
Yiorgis Gozadinos
f96a428ef1
Fix defects found reviewing the coverage work
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.

Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.

Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.

Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
2026-07-27 10:44:32 +03:00
Yiorgis Gozadinos
f6acb65e95
Reach and enforce 100% coverage
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.

Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.

Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
2026-07-26 20:11:02 +03:00
Yiorgis Gozadinos
7c76c3399e
Cover get_model provider branches and visualize_chunk edge paths
Parametrize the get_model tests whose only assertion was the returned model
type, extending the table to the reasoning-off, groq-parsed and Bedrock
o-series/qwen/unmapped branches. Fold format_citations_no_title into the
superset test that already covered its scenario.

Add tests for cosine_similarity on parallel vectors, multi-citation Rich
rendering, picture rendering with missing and undecodable bytes, a missing
docling distribution, and the visualize_chunk short-circuits for absent
documents, absent rasters and refless chunks.
2026-07-26 19:23:54 +03:00
Yiorgis Gozadinos
dddaf0f84c
Remove unreachable code and fix two defects it surfaced
Delete repository methods with no callers (SettingsRepository CRUD,
ChunkRepository.update/delete/get_chunks_in_range), the DataFrame branch of
_process_search_results whose only caller always passes a query, and guards
that cannot be reached from their call sites: the rebuild mode=None default,
the staging drop already performed by _resolve_rebuild_recovery, the empty
batch skip, two context fast paths, the doctor prefix guard, the poller
_task attribute that is never assigned, and a docling caption fallback for
a field name no item class defines.

set_haiku_version built a recreated settings row from the process-global
Config rather than the store's own, so a store opened with a custom config
stamped global settings into the database.

check_source_accessible called urlparse outside its try block, so a stored
URI with a malformed IPv6 host raised ValueError instead of reporting the
source as inaccessible, aborting the whole rebuild sweep.
2026-07-26 18:45:48 +03:00
Yiorgis Gozadinos
e2b273ee2a
Consolidate duplicated client-side tests
Parametrize sibling tests that differed only in a literal value, and fold
two strict-subset tests into the survivors that already covered their
scenario. Every case that ran before still runs; the union of assertions
is applied to each case, strengthening list_all, get_pages_data and
resolve_doc_items.

Replace four hand-rolled log-capture handlers with a shared
capture_logs() contextmanager in conftest.

13 fewer test functions, 348 fewer lines.
2026-07-26 13:30:59 +03:00
Yiorgis Gozadinos
9deb1f2bd4
replace haiku.skills with native Pydantic AI capabilities 2026-07-24 15:26:17 +03:00
Yiorgis Gozadinos
1ed424b50d
Serialize metadata-only document updates with the write lock 2026-07-16 13:11:35 +03:00
Yiorgis Gozadinos
0e271eaf4b
Remove --before/--at time travel 2026-07-16 13:11:33 +03:00
Yiorgis Gozadinos
0813a1c980
Add tag CLI commands and history tag annotations 2026-07-16 13:11:27 +03:00
Yiorgis Gozadinos
18c0f6c5e8
Add --no-expand to the visualize command for chunk-only grounding 2026-07-09 12:49:42 +03:00
Yiorgis Gozadinos
995d081aa1
Visualize the exact context the model saw via Citation.doc_item_refs
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
2026-07-09 11:51:37 +03:00
Yiorgis Gozadinos
6d86237dd6
Draw matched content stronger than expanded context in visualizations 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document 2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
d3a1011baf
write fetched bodies off the event loop 2026-06-22 10:37:48 +03:00
Yiorgis Gozadinos
2b8b9477b7
Serialize and roll back the multi-table document delete 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
3366a6d383
Split meta document attributes into a document_meta table 2026-06-12 10:17:17 +03:00
Yiorgis Gozadinos
4bc52f0710
Serialize multi-table document writes and bound update version churn 2026-06-12 10:16:39 +03:00
Yiorgis Gozadinos
a7ed405d75
Add HaikuRAG.import_documents for batch document import 2026-06-09 10:23:37 +03:00
Yiorgis Gozadinos
339d231a4e
Remove repliqa from test setup 2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
e7c7df2915
Always use the Store-owned embedder 2026-05-29 11:36:53 +03:00
Yiorgis Gozadinos
9ed24ad53e
swap doclaynet.pdf to the full paper, add real-PDF split/merge test 2026-05-26 11:44:45 +03:00
Yiorgis Gozadinos
13cbadeb6f
canonical metadata keys: source_revision + content_type, bump to 0.50.0 2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
7ea61a7b10
drop the old monitor, rename serve→mcp, add e2e tests 2026-05-26 11:42:41 +03:00
Yiorgis Gozadinos
704bea6f4a
Stabilize test_client_search ranking assertion 2026-05-14 14:37:27 +03:00
Yiorgis Gozadinos
75c8e83515
Allow uri override on create_document_from_source 2026-04-30 16:36:35 +03:00
Yiorgis Gozadinos
258ffa41fa
Improve test coverage 2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
3224d1f20c
Extract document orchestration into client/documents.py 2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
9067b89d2f
fix convert() misreading text content that starts with a URL 2026-04-22 14:47:23 +03:00
Yiorgis Gozadinos
e2bac1e887
deduplicate escape_sql_string into utils 2026-04-16 12:11:53 +03:00
Yiorgis Gozadinos
53445b8722
Add docling_pages column to DocumentRecord/Document for separate page image storage. 2026-04-08 14:04:32 +03:00
Yiorgis Gozadinos
f2998ad0b5
add visualize_chunk test for multi-page chunks 2026-04-03 20:20:42 +03:00
Yiorgis Gozadinos
dc83dde9ab
Make test_client_resolve_document not require embeddings 2026-02-06 12:07:20 +01:00
Yiorgis Gozadinos
122da834d7
Use resolve_document() 2026-02-06 12:07:19 +01:00
Yiorgis Gozadinos
4241a4b09e
Security fixes for RLM. Fix type() builtin, AST validation for dict key access, sql injection 2026-02-06 12:07:19 +01:00
Yiorgis Gozadinos
62d35d2907
Add ocr_engine field to ConversionOptions 2026-01-20 13:41:43 +02:00
Yiorgis Gozadinos
5e55c0df54
replace pyright with ty type checker 2026-01-19 15:50:14 +02:00
Yiorgis Gozadinos
ce326d80ff
Disable ocr in tests previously marked as integration 2026-01-12 16:19:25 +02:00
Yiorgis Gozadinos
52363639a6
Remove integration marker from tests that have vcr 2026-01-12 16:03:05 +02:00
Yiorgis Gozadinos
9c4fd08c47
Update client serialization for compressed docling documents 2026-01-11 09:23:56 +02:00
Yiorgis Gozadinos
1e9a10a235
Record cassettes for embedder, ignore huggingface while recording 2025-12-29 13:23:53 +02:00
Yiorgis Gozadinos
c3a3b4a5b4
Extended VCR cassette recording to embedder, client, research graph, and search filter tests 2025-12-26 18:53:17 +02:00
Yiorgis Gozadinos
a56e1ba67c
Simplify import_document, update_document 2025-12-08 18:19:12 +02:00
Yiorgis Gozadinos
06b4f6a224
Add format parameter for text-to-DoclingDocument conversion 2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
9a5f8e4368
Add type-aware context expansion for search results 2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
e97d235c98
Unify update_document, update_document_fields 2025-12-08 15:56:02 +02:00
Yiorgis Gozadinos
4fbb4609b6
HaikuRag.chunk() utility method 2025-12-08 15:55:30 +02:00
Yiorgis Gozadinos
ace915473c
HaikuRag.convert() utility method 2025-12-08 15:55:30 +02:00