Commit graph

21 commits

Author SHA1 Message Date
Yiorgis Gozadinos
82fd10e0ee
Migrate LanceDB to native async API
Convert all LanceDB operations from sync calls wrapped in async
functions to the native async API (connect_async, AsyncConnection,
AsyncTable, AsyncQuery). Database I/O no longer blocks the event loop.

- Store and HaikuRAG use async context managers (async with). Store
  initialization is deferred to __aenter__; direct construction
  without async with is no longer supported.
- Index creation uses config objects (FTS, BTree, IvfPq) instead of
  string-based index_type parameter.
- Upgrade callbacks are async.
- HaikuRAG tracks background vacuum tasks and awaits them in __aexit__
  and before destructive rebuild operations to avoid races with
  concurrent table mutations.
- temp_db_path fixture uses pytest's tmp_path for reliable async
  cleanup.
2026-04-24 14:42:52 +03:00
Yiorgis Gozadinos
b6113bf8ab
replace fixed-radius expansion with section-bounded algorithm
Context expansion is now automatic and structure-aware. For structured
documents, expands within the section containing the match. For sections
that exceed the budget or are too small, expands item-by-item outward
skipping noise labels. Unstructured documents use budget-based outward
expansion. Results sorted by relevance score.
2026-04-16 12:11:53 +03:00
Yiorgis Gozadinos
3fe953eccb
Minor fixes 2026-04-08 14:04:32 +03:00
Yiorgis Gozadinos
53445b8722
Add docling_pages column to DocumentRecord/Document for separate page image storage. 2026-04-08 14:04:32 +03:00
Yiorgis Gozadinos
6fdb15e3b2
Add DocumentRepository.get_docling_data() for lazy loading docling in expand_context 2026-04-07 11:19:04 +03:00
Yiorgis Gozadinos
8c2a101a8a
Fix out-of-memory in list_documents by adding column projection 2026-02-06 14:11:36 +01:00
Yiorgis Gozadinos
afc5b02285
Update schema to store docling_document as compressed bytes
- Change DocumentRecord.docling_document_json (str) to docling_document (bytes)
- Use large_binary Arrow type (64-bit offsets) to avoid 2GB column limit
- Decompress in Document.get_docling_document() using gzip
- Update DocumentRepository field mappings
2026-01-10 15:17:27 +02:00
Yiorgis Gozadinos
5c4799164c
Escape when getting by URI 2025-12-12 10:56:37 +02:00
Yiorgis Gozadinos
4f0f214f26
Remove unecessary tests, add no coverage pragmas where appropriate 2025-12-08 15:56:02 +02:00
Yiorgis Gozadinos
d267b2c433
LRU cache for DoclingDocument 2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
a5d8af07e9
Resolve DoclingDocument 2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
33b254ae3d
Always upgrade database on access. Explicitly create db by running init or HaikuRAG(path, create=True) 2025-12-02 16:49:55 +02:00
Yiorgis Gozadinos
2c554a9350
Do not use private methods in tests 2025-11-25 14:19:28 +02:00
Yiorgis Gozadinos
7ba1c2376e
Introduce converters for supporting more than local docling document conversion. Transform existing FileReader and utils to "docling-local" converter 2025-11-17 13:05:05 +02:00
Yiorgis Gozadinos
908296e155
Do not create unecessary DoclingDocument when chunks are provided. Rename private _create_with_docling & _update_with_docling to _create_and_chunk & _update_and_rechunk 2025-11-13 13:16:15 +02:00
Yiorgis Gozadinos
1f22f74e1a
Add filtering to document listing in CLI/client 2025-11-04 16:02:58 +02:00
Yiorgis Gozadinos
d1e2470d94
Upgrades are back. Make chunk order a table column rather than a metadata property 2025-09-18 13:42:07 +03:00
Yiorgis Gozadinos
b83596ff07
Basic moving to lancedb. 2025-09-01 14:56:14 +03:00
Yiorgis Gozadinos
1ff17c2f50
Add uri to Documents 2025-06-17 10:41:44 +02:00
Yiorgis Gozadinos
5f9fb12644
Use db repositories 2025-06-16 17:31:53 +02:00
Yiorgis Gozadinos
5d1a854934
Basic store for chunks and documents 2025-06-15 22:34:54 +02:00