Commit graph

27 commits

Author SHA1 Message Date
Yiorgis Gozadinos
dbb87e091f
Ensure schema parity for _StagingChunkRecord 2026-05-14 15:11:18 +03:00
Yiorgis Gozadinos
849eca94d0
Make rebuild --embed-only idempotent across crashes 2026-05-14 15:02:12 +03:00
Yiorgis Gozadinos
a035b0f9e4
Stream rebuild --embed-only through a staging table to bound memory 2026-05-14 15:02:12 +03:00
Yiorgis Gozadinos
a98ddc14b8
Replace picture_description.enabled with processing.pictures enum 2026-05-13 16:22:44 +03:00
Yiorgis Gozadinos
e55e3d13c8
fix _patch_picture_descriptions silently dropping docling_pages 2026-05-06 12:57:41 +03:00
Yiorgis Gozadinos
08696773fe
Fix vcr test 2026-05-05 14:38:03 +03:00
Yiorgis Gozadinos
46b81f3fa1
cover picture-description provider and rebuild edge cases 2026-05-05 14:20:37 +03:00
Yiorgis Gozadinos
a45d82f206
Update benchmark 2026-05-05 13:58:48 +03:00
Yiorgis Gozadinos
ff82d36c2f
add rebuild --descriptions: run VLM over stored picture bytes only 2026-05-05 12:02:46 +03:00
Yiorgis Gozadinos
0ffca7727c
Fix caplog-based assertion flaking under xdist ordering 2026-04-24 15:51:29 +03:00
Yiorgis Gozadinos
ddcc4518ad
Cover FULL rebuild source-failure branch 2026-04-24 15:32:53 +03:00
Yiorgis Gozadinos
258ffa41fa
Improve test coverage 2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
82fd10e0ee
Migrate LanceDB to native async API
Convert all LanceDB operations from sync calls wrapped in async
functions to the native async API (connect_async, AsyncConnection,
AsyncTable, AsyncQuery). Database I/O no longer blocks the event loop.

- Store and HaikuRAG use async context managers (async with). Store
  initialization is deferred to __aenter__; direct construction
  without async with is no longer supported.
- Index creation uses config objects (FTS, BTree, IvfPq) instead of
  string-based index_type parameter.
- Upgrade callbacks are async.
- HaikuRAG tracks background vacuum tasks and awaits them in __aexit__
  and before destructive rebuild operations to avoid races with
  concurrent table mutations.
- temp_db_path fixture uses pytest's tmp_path for reliable async
  cleanup.
2026-04-24 14:42:52 +03:00
Yiorgis Gozadinos
f05159859c
Additional fixes to remove type ignores 2026-01-19 16:00:18 +02:00
Yiorgis Gozadinos
5e55c0df54
replace pyright with ty type checker 2026-01-19 15:50:14 +02:00
Yiorgis Gozadinos
8bdfbe526c
Fix embed-only rebuild with changed vector dimensions
When a database was created with one embedding model and rebuild
--embed-only was run with a different model, it failed with a
vector dimension validation error.

- Store reads stored vector_dim when opening existing databases
- _rebuild_embed_only recreates chunks table to handle dimension changes
- Add test for rebuild with changed vector dimensions
2026-01-14 17:42:28 +02:00
Yiorgis Gozadinos
9c4fd08c47
Update client serialization for compressed docling documents 2026-01-11 09:23:56 +02:00
Yiorgis Gozadinos
1e9a10a235
Record cassettes for embedder, ignore huggingface while recording 2025-12-29 13:23:53 +02:00
Yiorgis Gozadinos
84aa46a10b
Update rebuild to handle DoclingDocument json 2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
33b254ae3d
Always upgrade database on access. Explicitly create db by running init or HaikuRAG(path, create=True) 2025-12-02 16:49:55 +02:00
Yiorgis Gozadinos
e548a51ff7
Optimized rebuild --embed-only to use batch updates via LanceDB merge_insert instead of individual chunk updates 2025-12-01 17:53:27 +02:00
Yiorgis Gozadinos
88a4edb537
--embed-only, --rechunk options in rebuild command 2025-11-28 10:11:52 +02:00
Yiorgis Gozadinos
05c382fcb5
When rebuilding if a document does not exist anymore, re-embed the chunks instead of failing 2025-09-30 16:57:02 +03:00
Yiorgis Gozadinos
b83596ff07
Basic moving to lancedb. 2025-09-01 14:56:14 +03:00
Yiorgis Gozadinos
a2e0f2cefd
Remove Claude nonsense, use context manager 2025-07-12 23:00:49 +03:00
Yiorgis Gozadinos
85443ee963
Turn rebuild into a generator, track progress in command line 2025-07-02 11:10:31 +03:00
Yiorgis Gozadinos
11d9f2701e
Rebuild database in client & cli 2025-07-02 10:55:06 +03:00