Commit graph

8 commits

Author SHA1 Message Date
Yiorgis Gozadinos
ab19f78507
Move source adapters out of the ingester package
haiku.rag.ingester.sources was never ingester-only: one-shot client
ingestion resolves adapters through it (create_document_from_source), and
convert() now fetches through HTTPSource, so the core client imported into
the ingester package to reach them.

Move the package to haiku.rag.sources and update every import. No shims:
haiku.rag.ingester.sources is gone.

The haiku.rag.sources plugin entry-point group is unchanged, so third-party
source packages need no edit — the group name now matches the module path it
always implied.

Source unit tests move to tests/sources/. test_source_plugins.py stays in
tests/ingester/: it drives a PeriodicPoller against the job repo, so it is
plugin wiring through ingester machinery rather than a source test.
2026-08-20 11:46:55 +03:00
Yiorgis Gozadinos
44089e5b1f
Support a dburi for the ingester queue (SQLite + Postgres)
Migrate the ingester queue storage from raw aiosqlite to SQLAlchemy Core
async. The backend is chosen by ingester.queue.dburi: a SQLAlchemy async
URL points the queue at a database server, and SQLite remains the default
when unset. The Postgres path claims jobs with FOR UPDATE SKIP LOCKED so
multiple ingester processes can share one queue; SQLite caps the pool to a
single connection to keep the select-then-update claim atomic.
2026-06-03 14:34:09 +03:00
Yiorgis Gozadinos
1c710433c7
Collapse worker_count and max_concurrent into worker_count 2026-05-27 12:48:14 +03:00
Yiorgis Gozadinos
98efd73177
Serialize JobRepo and SyncStateRepo on one shared lock 2026-05-26 13:11:01 +03:00
Yiorgis Gozadinos
f89cc998eb
add sources_provider to WorkerPool. workers now resolve extra info through these sources 2026-05-26 11:45:35 +03:00
Yiorgis Gozadinos
13cbadeb6f
canonical metadata keys: source_revision + content_type, bump to 0.50.0 2026-05-26 11:43:30 +03:00
Yiorgis Gozadinos
6d15f1f247
fix URL-decode file:// paths before stat/exists checks 2026-05-26 11:42:42 +03:00
Yiorgis Gozadinos
7ea61a7b10
drop the old monitor, rename serve→mcp, add e2e tests 2026-05-26 11:42:41 +03:00