Commit graph

47 commits

Author SHA1 Message Date
Yiorgis Gozadinos
971ec0a5b0
Coerce a string db_path to Path at the Store and capability boundaries
The documented `HaikuRAG("knowledge.lancedb")` and `rag(db_path="my.lancedb")`
forms both raised `AttributeError: 'str' object has no attribute 'exists'`.
`Store.__init__` assigned its argument to a `Path`-annotated attribute without
coercing, and `resolve_db_path` returned a non-None argument unchanged. Every
runnable example wraps the path in `Path(...)`, which is why it survived.

Coerce in `Store.__init__` and `resolve_db_path`; widen the annotations on
`Store`, `HaikuRAG` and both `create_capability` factories to accept `str`.
2026-08-21 12:49:45 +03:00
Yiorgis Gozadinos
ab19f78507
Move source adapters out of the ingester package
haiku.rag.ingester.sources was never ingester-only: one-shot client
ingestion resolves adapters through it (create_document_from_source), and
convert() now fetches through HTTPSource, so the core client imported into
the ingester package to reach them.

Move the package to haiku.rag.sources and update every import. No shims:
haiku.rag.ingester.sources is gone.

The haiku.rag.sources plugin entry-point group is unchanged, so third-party
source packages need no edit — the group name now matches the module path it
always implied.

Source unit tests move to tests/sources/. test_source_plugins.py stays in
tests/ingester/: it drives a PeriodicPoller against the job repo, so it is
plugin wiring through ingester machinery rather than a source test.
2026-08-20 11:46:55 +03:00
Yiorgis Gozadinos
e8f00fcff4
Make get_config the only configuration lookup
haiku.rag.config exported two configuration instances: the lazy _config
behind get_config/set_config, and Config, loaded at import time. Nothing
linked them, and eleven signatures captured Config as a default argument,
so set_config could not reach the factories, the client, the store or the
MCP server. reranking/base.py went further and snapshotted the configured
reranker name into a class attribute at import.

Config is removed. Internal defaults are config: AppConfig | None = None,
resolved through get_config() per call. RerankerBase._model is None and
CohereReranker takes its model name as an argument, like every other
reranker.

The suite patched attributes on Config while production read the instance
get_config() returns, a different object, so those patches were no-ops
waiting to happen. They now go through get_config().
2026-08-19 14:43:40 +03:00
Yiorgis Gozadinos
2da294b850
Put multi-table writes behind one transaction boundary
Four call sites repeated lock, snapshot, try/except, restore. Each used
restore_table_versions, which restores in _tables() order — documents
first, contradicting RESTORE_TABLE_ORDER — and each caught Exception, so
a cancellation mid-write skipped rollback and left the earlier table
writes committed.

Store.write_transaction() holds the lock, snapshots under it, and rolls
back through _rollback_to_snapshot: RESTORE_TABLE_ORDER, shielded from
cancellation, absorbed cancellation re-delivered, rollback failure raised
with the original as cause. The two single-table update_meta sites keep
the bare lock.

The batch documents write moves inside the guarded body; it was outside
the try, so a failure there was never rolled back. Auto-vacuum is
scheduled after the transaction rather than inside it.

restore_table_versions is removed; those four sites were its only callers.
2026-08-19 14:05:40 +03:00
Yiorgis Gozadinos
8b45f81464
Load only content when listing documents with include_content 2026-07-31 10:12:19 +03:00
Yiorgis Gozadinos
4c5050d161
Accept images on ask/analyze 2026-07-25 10:23:55 +03:00
Yiorgis Gozadinos
0e271eaf4b
Remove --before/--at time travel 2026-07-16 13:11:33 +03:00
Yiorgis Gozadinos
0813a1c980
Add tag CLI commands and history tag annotations 2026-07-16 13:11:27 +03:00
Yiorgis Gozadinos
18c0f6c5e8
Add --no-expand to the visualize command for chunk-only grounding 2026-07-09 12:49:42 +03:00
Yiorgis Gozadinos
995d081aa1
Visualize the exact context the model saw via Citation.doc_item_refs
visualize_chunk re-expanded chunks from scratch to recover their refs,
which could not faithfully reproduce the original merge, scores, and
clip — so a visualization could highlight different pages than the
citation covered. Carry the cited items on Citation.doc_item_refs and
resolve bounding boxes from them directly; re-expansion remains only as
the fallback for callers with no stored context (CLI, inspector). Chat,
inspector, the app endpoint, and the frontend pass the refs through.
2026-07-09 11:51:37 +03:00
Yiorgis Gozadinos
494f774473
Visualize all constituent chunks of a merged citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
c02bcd5dc9
Pool HTTP clients for vLLM embedding and vLLM/Jina reranking 2026-07-09 10:32:44 +03:00
Yiorgis Gozadinos
c2c8c24951
Allow updating a document's uri via update_document 2026-07-08 10:32:14 +03:00
Yiorgis Gozadinos
1e65d36766
Pass fetched FetchResult to ingester metadata providers 2026-06-16 16:37:26 +03:00
Yiorgis Gozadinos
2b8b9477b7
Serialize and roll back the multi-table document delete 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
5d5d87d44c
Throttle background auto-vacuum to at most once per 5 minutes 2026-06-12 10:17:18 +03:00
Yiorgis Gozadinos
a7ed405d75
Add HaikuRAG.import_documents for batch document import 2026-06-09 10:23:37 +03:00
Yiorgis Gozadinos
0e3d53791f
Add rebuild --set-embedder to reconcile embedder identity 2026-06-05 10:42:15 +03:00
Yiorgis Gozadinos
e7c7df2915
Always use the Store-owned embedder 2026-05-29 11:36:53 +03:00
Yiorgis Gozadinos
37a78a4e9b
Cache reranker on the client instead of rebuilding per search 2026-05-29 10:33:50 +03:00
Yiorgis Gozadinos
26bc71d6d8
Cascade delete_document to children via metadata.parent_uri 2026-05-28 15:36:00 +03:00
Yiorgis Gozadinos
d7fdd61fed
Resolve worker source by source_id, not just supports(uri) 2026-05-27 13:36:03 +03:00
Yiorgis Gozadinos
f89cc998eb
add sources_provider to WorkerPool. workers now resolve extra info through these sources 2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
8c57a3ca99
Drop the multi-agent research workflow 2026-05-20 12:46:48 +03:00
Yiorgis Gozadinos
6f95e2bc27
Delete the standalone QA and analysis agents 2026-05-19 11:39:20 +03:00
Yiorgis Gozadinos
947a26b391
client.analyze routes through the rag-analysis skill; drop documents= and AnalysisResult.program. Re-record all cassettes that are relevant 2026-05-19 10:46:32 +03:00
Yiorgis Gozadinos
d96d2eeb0f
client.ask routes through the rag skill; always show citations in CLI 2026-05-19 10:19:39 +03:00
Tres Seaver
f9ce3bc401
refactor: move 'SearchType' to 'store.models.chunk' 2026-05-18 09:27:34 -04:00
Tres Seaver
7953de46b9
fix: declare 'SearchType' for 'search_type' arg
fix: allow 'search_type' only for text searches
2026-05-18 06:35:51 -04:00
Yiorgis Gozadinos
afd4665517
Thread source_uri through URL ingest 2026-05-12 15:13:02 +03:00
Yiorgis Gozadinos
d009da06d6
support s3:// document sources 2026-05-11 11:28:31 +03:00
Yiorgis Gozadinos
a45d82f206
Update benchmark 2026-05-05 13:58:48 +03:00
Yiorgis Gozadinos
ff82d36c2f
add rebuild --descriptions: run VLM over stored picture bytes only 2026-05-05 12:02:46 +03:00
Yiorgis Gozadinos
ff656504d3
make client.search() polymorphic on query type: str | bytes | PIL.Image.Image. Bytes/PIL queries embed via embed_image_query and run vector-only against the chunks table 2026-05-04 11:22:22 +03:00
Yiorgis Gozadinos
c9227c649f
emit synthetic picture chunks at ingest under multimodal embedders.
processing.chunk() merges text chunks with one synthetic Chunk per PictureItem-with-bytes,
sorted by iterate_items() position so chunk.order is structural.
embed_chunks dispatches on a Chunk._picture_data PrivateAttr
(text through embed_documents, picture through embed_image_query)
2026-05-03 17:04:43 +03:00
Yiorgis Gozadinos
75c8e83515
Allow uri override on create_document_from_source 2026-04-30 16:36:35 +03:00
Yiorgis Gozadinos
b01c649684
surface picture image bytes in SearchResult and emit multimodal ToolReturn from the agent search tool 2026-04-30 12:43:05 +03:00
Yiorgis Gozadinos
cd4224a727
drop private title/embedding forwarders from HaikuRAG 2026-04-24 15:52:50 +03:00
Yiorgis Gozadinos
aca3ccc29f
Close connections if __aenter__ fails mid-initialization 2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
3224d1f20c
Extract document orchestration into client/documents.py 2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
cb02a2c3d8
Extract rebuild machinery into client/rebuild.py 2026-04-24 14:43:59 +03:00
Yiorgis Gozadinos
528acb79a3
Extract LLM agents into client/agents.py 2026-04-24 14:43:58 +03:00
Yiorgis Gozadinos
12939f1444
Extract search, expand_context, visualize_chunk into client/search.py 2026-04-24 14:43:58 +03:00
Yiorgis Gozadinos
d4ecebf896
Extract processing primitives into client/processing.py 2026-04-24 14:42:53 +03:00
Yiorgis Gozadinos
b5e480bfa1
Extract title generation into client/titles.py 2026-04-24 14:42:53 +03:00
Yiorgis Gozadinos
e814459e1f
Extract download_models into client/downloads.py 2026-04-24 14:42:53 +03:00
Yiorgis Gozadinos
43f2774c14
Turn client.py into a client/ package 2026-04-24 14:42:53 +03:00
Renamed from haiku_rag_slim/haiku/rag/client.py (Browse further)