urlparse().path keeps the leading slash in front of a Windows drive, so
file:///C:/docs/a.pdf read as \C:\docs\a.pdf and the ingester reported
"File does not exist" for every file it discovered. url2pathname is the
stdlib conversion that strips it, per platform.
Four sites each decided both "is this local" and "what path is this":
FSSource._uri_to_path and supports, resolve_adhoc_fetcher,
create_document_from_source and check_source_accessible, and convert.
is_local_uri and uri_to_path in haiku.rag.uri own those two decisions now,
which closes two more cases of the same root cause. A bare C:\docs\a.pdf
parses with scheme "c", so add-src raised "No source adapter for URI scheme
'c'" and convert silently treated the path as raw text. And convert and
check_source_accessible never percent-decoded at all, so a file named
a[b] c.md read as missing on Linux and macOS too.
A file URI's host is reattached after conversion rather than passed to
url2pathname, which as of 3.14 rejects a non-local authority off Windows.
file:////server/share is the empty-authority spelling of a UNC path, its
host being the first path segment, so that host is normalised into the
authority before conversion. Output is identical on 3.12, 3.13 and 3.14.
The ad-hoc FS fetcher roots at the path's own anchor rather than "/", which
on Windows is only the current drive.
test_uri.py runs on ubuntu, macos and windows across 3.13 and 3.14 without
the project installed: --noconftest because the repo conftest imports
dependencies that job does not need, and -o addopts= to drop the
repository's -n auto. The Windows legs are what cover the drive conversion.
Fixes#574.
Sixty-three comments said what the next statement already said: # Connect to
LanceDB above connect_lancedb, # Path object above isinstance(source, Path),
# Get page numbers from provenance above the prov loop, # Clear and populate
results above list_view.clear(). They cost a read and carry nothing.
The line is whether a comment restates one statement or labels a phase. Phase
labels stay: the migrations keep # Create staging table with new schema and
# Copy from staging to final table in batches, each heading ten lines of a
long procedure. So do comments carrying a fact the code cannot: the
merge_insert update-only note on document_meta, why the poller builds sources
eagerly, why create_document_from_source returns a list for directories, that
indexes need training data, the field-group markers in the config models, and
the file:// URL-encoding note in create_document_from_source.
capabilities/ is untouched. Its docstrings sit next to prompt surface, and
changing them needs an eval to back it.
The cassette-recording docs were wrong three ways. They named
tests/test_qa.py::test_qa_anthropic, which no longer exists; they targeted
whole modules, so a rewrite would re-record cassettes for services the
recorder is not running; and they used COHERE_API_KEY where the SDK reads
CO_API_KEY. docs/development.md now names exact tests with -n0, and the keyed
example is test_cohere_reranker, which owns the one cassette recording
api.cohere.com.
haiku.rag.ingester.sources was never ingester-only: one-shot client
ingestion resolves adapters through it (create_document_from_source), and
convert() now fetches through HTTPSource, so the core client imported into
the ingester package to reach them.
Move the package to haiku.rag.sources and update every import. No shims:
haiku.rag.ingester.sources is gone.
The haiku.rag.sources plugin entry-point group is unchanged, so third-party
source packages need no edit — the group name now matches the module path it
always implied.
Source unit tests move to tests/sources/. test_source_plugins.py stays in
tests/ingester/: it drives a PeriodicPoller against the job repo, so it is
plugin wiring through ingester machinery rather than a source test.
Five call sites repeated the same post-conversion preparation: store the
Docling representation and resolve a title when none was supplied.
_prepare_and_title now owns that sequence. update_document continues to
call _prepare_document_from_docling directly because an explicit update
must preserve an existing empty title.
create_document, both content-replacement branches of update_document,
and source ingestion embedded eagerly before passing chunks to a
persistence funnel that checked them again. The funnels now own
embedding, including the checks required by import_document and
import_documents for caller-supplied chunks.
Move the document.embed span into ensure_chunks_embedded after its early
return. Every path that performs embedding is now instrumented, while
operations whose chunks are already embedded emit no span.
convert() previously used its own HTTP client and temporary-file path.
Route URL conversion through HTTPSource, matching source ingestion, and
move _write_fetch_body to processing.py so both paths share temporary
file handling without an import cycle.
Add walk_files for filesystem enumeration and use it from both
FSSource.discover and one-shot directory ingestion. Symlink escape
filtering now has one implementation.
Extracts build_picture_chunks + iterate_items position mapping into a
sync helper and wraps it with asyncio.to_thread so image-heavy
documents don't block the event loop during chunking.
Fixes#456.
processing.chunk() merges text chunks with one synthetic Chunk per PictureItem-with-bytes,
sorted by iterate_items() position so chunk.order is structural.
embed_chunks dispatches on a Chunk._picture_data PrivateAttr
(text through embed_documents, picture through embed_image_query)
