FULL rebuild deleted a document before re-ingesting it from its URI, and
the handler around that logged and continued. A 404, a timeout or any
conversion error therefore removed the document permanently.
Deleting after a successful create is not an alternative:
create_document_from_source resolves the same URI to the existing
document and updates it in place, so a trailing delete would remove the
freshly rebuilt row.
Refresh in place instead. create_document_from_source takes an internal
force flag that skips the revision and MD5 short-circuits, so an
unchanged source is still re-converted, re-chunked and re-embedded into
the existing document, and the document id survives a rebuild.
A failed refresh now falls through to the stored-content path rather
than skipping the document: FULL recreates the chunks table before the
loop, so skipping left the document present but unsearchable until the
next rebuild.
The pending-batch flush moves out of the try. A failed flush is a lost
write and should abort the rebuild, not be logged and skipped.
check_source_accessible narrowed its handler to ValueError, but Path.exists
re-raises errno values outside its ignored set (EACCES, ENAMETOOLONG). Those
were swallowed before and now escaped into the rebuild sweep the guard exists
to protect. Catch OSError too.
Restore the arity guard in _common_path_prefix: without it an empty list
raises from min() and a single label yields a prefix covering the whole path.
Two tests would have hung rather than failed on regression (the vacuum skip
and the protected-wait cancellation); both are now bounded. The import
vacuum test raced against the done-callback that discards the task, and now
spies on the call instead, with a negative control.
Replace assertions that could not fail: blank-query search against an empty
corpus, a batch flush counted against an empty table, a picture description
asserting its own input state, and an FS scheme check with nothing on disk to
resolve. The get_model matrix asserted only the returned type across 26
cases and now pins the per-provider settings. The three batching tests now
count flushes, which revealed embed-only writes through chunks_table.add
rather than _flush_rebuild_batch.
Add tests for the vacuum-failure warning, documents deleted mid-rebuild,
chunkless documents, batch flushes in the embed-only, descriptions and full
paths, a missing docling blob under rechunk, missing and recoverable picture
bytes, and the source-missing fallback. Direct-call tests cover the staging
helpers and the idempotent phase-1 marker.
processing.chunk() merges text chunks with one synthetic Chunk per PictureItem-with-bytes,
sorted by iterate_items() position so chunk.order is structural.
embed_chunks dispatches on a Chunk._picture_data PrivateAttr
(text through embed_documents, picture through embed_image_query)
When a database was created with one embedding model and rebuild
--embed-only was run with a different model, it failed with a
vector dimension validation error.
- Store reads stored vector_dim when opening existing databases
- _rebuild_embed_only recreates chunks table to handle dimension changes
- Add test for rebuild with changed vector dimensions
