haiku.rag/tests/ingester
Yiorgis Gozadinos d1691d3942
Share document preparation and HTTP acquisition
Five call sites repeated the same post-conversion preparation: store the
Docling representation and resolve a title when none was supplied.
_prepare_and_title now owns that sequence. update_document continues to
call _prepare_document_from_docling directly because an explicit update
must preserve an existing empty title.

create_document, both content-replacement branches of update_document,
and source ingestion embedded eagerly before passing chunks to a
persistence funnel that checked them again. The funnels now own
embedding, including the checks required by import_document and
import_documents for caller-supplied chunks.

Move the document.embed span into ensure_chunks_embedded after its early
return. Every path that performs embedding is now instrumented, while
operations whose chunks are already embedded emit no span.

convert() previously used its own HTTP client and temporary-file path.
Route URL conversion through HTTPSource, matching source ingestion, and
move _write_fetch_body to processing.py so both paths share temporary
file handling without an import cycle.

Add walk_files for filesystem enumeration and use it from both
FSSource.discover and one-shot directory ingestion. Symlink escape
filtering now has one implementation.
2026-08-20 10:50:33 +03:00
..
__init__.py ingester: add source-adapter scaffolding + FS source 2026-05-26 11:41:53 +03:00
conftest.py Support a dburi for the ingester queue (SQLite + Postgres) 2026-06-03 14:34:09 +03:00
test_api.py Make ingester job retry idempotent against a live sibling 2026-06-16 16:05:05 +03:00
test_circuit_breaker.py Move CircuitBreaker to a shared module 2026-07-08 11:03:30 +03:00
test_cli.py Move ingester telemetry setup behind argument parsing 2026-08-13 11:37:36 +03:00
test_config.py Renew job leases from the worker pool; lease-based reaping 2026-06-25 13:26:09 +03:00
test_filter.py Improve test coverage for cli, filter, registry, and migrations 2026-06-01 18:00:47 +03:00
test_fs_source.py Share document preparation and HTTP acquisition 2026-08-20 10:50:33 +03:00
test_http_source.py Cover MCP, ingester and converter error paths 2026-07-26 19:14:26 +03:00
test_metadata.py Pass fetched FetchResult to ingester metadata providers 2026-06-16 16:37:26 +03:00
test_pipeline.py Classify obstore config errors as permanent 2026-08-06 12:53:45 +03:00
test_pollers.py Cover MCP, ingester and converter error paths 2026-07-26 19:14:26 +03:00
test_queue.py Add last_heartbeat_at lease column to the ingester queue 2026-06-25 13:25:37 +03:00
test_queue_postgres.py Add last_heartbeat_at lease column to the ingester queue 2026-06-25 13:25:37 +03:00
test_resolve_fetcher.py Improve test coverage for cli, filter, registry, and migrations 2026-06-01 18:00:47 +03:00
test_revision_round_trip.py Bump lancedb to 0.34.0 2026-07-16 12:32:56 +03:00
test_run_batch.py Show progress for run-batch drains 2026-06-22 12:51:38 +03:00
test_s3_source.py Add configurable max_file_size to reject oversized files before ingestion 2026-06-01 18:40:46 +03:00
test_serve_integration.py Support a dburi for the ingester queue (SQLite + Postgres) 2026-06-03 14:34:09 +03:00
test_source_plugins.py Add custom ingester sources via entry points 2026-06-16 10:54:24 +03:00
test_sources_base.py Fix shutdown-order bug: close source clients after workers stop 2026-06-01 09:55:39 -04:00
test_webdav_source.py Fix defects found reviewing the coverage work 2026-07-27 10:44:32 +03:00
test_workers.py Add Logfire debugging skills and worker-breaker event 2026-07-10 13:23:17 +03:00