Commit graph

23 commits

Author SHA1 Message Date
Yiorgis Gozadinos
fdb5710491
Ask and analyze across several databases
Chunk 2 gave search a configured set to fan out over. ask and analyze
covered one database still: the RAG capability had no way to be told which
databases a question spanned, and the analysis sandbox mounted one
document tree.

The selection travels as sources on EvidenceState, beside the filter it
scopes with, so both capabilities read it the same way. clients_covering
is the one rule that turns a selection into clients, used by search, the
sandbox mount and the cite fallback, so a question scoped to some
databases cannot search, mount or cite another. Citations carry the
database they came from, and format_for_agent names it, so the model can
attribute evidence while it answers rather than only afterwards.

The sandbox keeps one flat /documents/{id}/ namespace and resolves each id
to the client holding it, which rests on ids being UUID4. A database
copied from another breaks that, so an id held twice is refused rather
than resolved to whichever arrived last.

On the CLI, search, ask and analyze cover the configured set and label
each result with its database. Every other command works on one, named
with --database NAME (a name reaches a database behind a URI, which --db
cannot) or --db PATH, and refuses a set it cannot choose from instead of
silently reading the default database. Cold databases open together, so a
first query costs the slowest open rather than their sum.
2026-08-24 10:03:46 +03:00
Yiorgis Gozadinos
397b553528
Search several configured databases and fuse the results
`lancedb.databases` maps a name to a location, mutually exclusive with `uri`.
`search(sources=[…])` selects which to search, `sources=None` searches all of them
and `sources=[]` searches none; `SearchResult.source` carries the configured name,
so a path or URI never leaves the configuration. A database named in config keeps
its name even when it is the only one configured; only a legacy single `uri`
leaves `source` unset.

Databases open on first use, not at entry. Which are searched is a per-query
choice, so a set of 25 queried a few at a time opens a few, and a database nobody
asked for can neither fail a query nor be opened for nothing.

A named database that fails to open raises `SourceUnavailableError` naming it,
raised outside the handler so the original is not attached at all. A local failure
spells out the absolute path and an object-store failure can carry the bucket;
`from None` would only stop that being printed, leaving it on `__context__` for
anything that walks the chain. A legacy `uri` client has no name to report
instead, so its error passes through unchanged.

Candidates are fetched concurrently, then fused before anything is ranked. A
configured reranker scores the union, which is what makes ranking across
databases tractable: it compares query against document and does not care where a
candidate came from. Without one, reciprocal rank fusion over the per-database
rankings, since scores from separate indexes are not comparable. Enrichment then
runs on the survivors through the database each came from, concurrently, so it
costs what a single-database search costs.

The over-fetch decision and the reranker belong to the federating client alone.
Deciding per database would have each consult its own, and a local reranker loads
model weights per instance. It is built only for a text query, and closed once by
the client that owns it.

A location without a scheme is opened as a local path rather than through
`lancedb.uri`. Routing it through `uri` had `ConnectionMode` classify it as object
storage, which opens a missing database instead of reporting it.

With several databases configured, `store` and the repositories are left unset:
they have no unambiguous meaning across a set, and picking one silently would be
worse than the error.
2026-08-24 10:03:45 +03:00
Lawrence Akka
47feeb32ee Linting, doc edits 2026-08-18 12:18:10 +02:00
Lawrence Akka
09ff101349 Expose all chunk metadata on search results and citations through SearchResult.chunk_meta and Citation.chunk_meta 2026-08-17 19:42:23 +02:00
Yiorgis Gozadinos
46f7ab8d97
Name the search result each page image belongs to
ToolReturn.content reaches the model as a user-role message and the pictures
arrive bare, so nothing connects a figure to the chunk it came from:
BinaryContent.identifier does not survive serialization to the vision API, and
the captions in the result text correlate only by position.

Precede each picture with its position, source chunk id and self_ref.
build_binary_parts_from_results becomes build_image_content_from_results and
returns the labels interleaved with the pictures, so both attachment sites emit
them the same way.

This does not stop a model narrating retrieved pictures as user-supplied.
Measured on gemma4-26b with a single note ahead of the batch, and again with
per-image labels: it quotes the label and still says the user provided them.
The message role wins over its text.
2026-08-13 13:00:01 +03:00
Yiorgis Gozadinos
7ba78fdde3
Normalize pragma comments to the codebase's single-line form 2026-07-26 20:18:19 +03:00
Yiorgis Gozadinos
f6acb65e95
Reach and enforce 100% coverage
Cover the remaining paths in the client, context, downloads, title
generation, document tools and store models, and add fail_under=100 so
uncovered lines fail CI.

Six lines that no test can reach get a pragma with its reason: the docling
import guard, the nameless PDF attachment, the FS symlink OSError guard that
resolve(strict=False) absorbs, the docling bbox and LanceDB document-id
shape guards, the tag-retention branch vacuum makes unreachable, and Monty's
Rust-thread print callback.

Fix test_find_config_file_user_config, which wrote its config into the cwd it
had chdir'd to, so the cwd branch answered first and the user-directory
lookup it names was never exercised.
2026-07-26 20:11:02 +03:00
Yiorgis Gozadinos
75a21c82c2
Search results and citations carry document metadata 2026-07-23 10:23:48 +03:00
Yiorgis Gozadinos
0bcf34363a
Carry merged chunk ids on SearchResult and Citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
514b0c51f8
surface figure captions in search results, lower search.limit to 5 2026-05-18 16:49:43 +03:00
Tres Seaver
ad0161e6da
chore: isort 2026-05-18 09:31:12 -04:00
Tres Seaver
f9ce3bc401
refactor: move 'SearchType' to 'store.models.chunk' 2026-05-18 09:27:34 -04:00
Yiorgis Gozadinos
c9227c649f
emit synthetic picture chunks at ingest under multimodal embedders.
processing.chunk() merges text chunks with one synthetic Chunk per PictureItem-with-bytes,
sorted by iterate_items() position so chunk.order is structural.
embed_chunks dispatches on a Chunk._picture_data PrivateAttr
(text through embed_documents, picture through embed_image_query)
2026-05-03 17:04:43 +03:00
Yiorgis Gozadinos
b01c649684
surface picture image bytes in SearchResult and emit multimodal ToolReturn from the agent search tool 2026-04-30 12:43:05 +03:00
Yiorgis Gozadinos
68c3fa0f79
Add order to SearchResult, add ChunkRepository.get_chunks_in_range(), to use them in _expand_with_chunks to fetch only nearby chunks 2026-04-07 12:04:05 +03:00
Yiorgis Gozadinos
ce320ca772
Update docling, docling-core and move to version 1.10.0 of DoclingDocument. 2026-03-23 17:34:06 +02:00
Yiorgis Gozadinos
7f51c63ad7
Search results now show rank position instead of raw scores. Consolidate cassettes 2026-01-21 14:54:15 +02:00
Yiorgis Gozadinos
1b14a4643d
Remove unused bounding box calculations 2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
9a5f8e4368
Add type-aware context expansion for search results 2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
c69b934eac
Incorporate additional docling meta in search(), expand_context() and relevant qa/research agents 2025-12-08 15:54:56 +02:00
Yiorgis Gozadinos
a5d8af07e9
Resolve DoclingDocument 2025-12-08 15:54:31 +02:00
Yiorgis Gozadinos
44e8d7b340
Update Document & Chunk schemas to include DoclingDocument references. 2025-12-08 15:54:30 +02:00
Yiorgis Gozadinos
2f9c907031
Restructure into uv workspace to support minimal and full installations 2025-11-04 17:59:12 +02:00
Renamed from src/haiku/rag/store/models/chunk.py (Browse further)