Hybrid search inside one database fuses its own vector and FTS rankings
with lancedb's RRFReranker over exactly the requested limit, and both
branch queries derive from the same inner query, so there is no
branch-depth knob. Below roughly 50 candidates the two rankings stop
overlapping, nothing sums, and the fusion degenerates: measured recall@5
on a single database was 0.000 at fetch 5, 10 and 20, then 0.267 at 50 and
0.350 at 100.
A dataset's retrieval_limit therefore fixes which regime it measures, and
comparing regimes would otherwise need one dataset per depth.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
Measures whether cross-database fusion reaches what a query needs, scored
on retrieval alone so no model or judge sits between the fusion and the
number.
The corpus is MTRAG ClapNQ partitioned by article title, whole titles to a
collection, so an article's passages never split and a query's gold stays
concentrated in one collection, which is the condition a per-collection
depth quota punishes. collection_of keys on sha256 rather than hash(),
which is salted per process: the partition is never stored, and scoring
recomputes it in a different process than the one that ingested.
The 148 titles holding a gold passage carry 10,723 passages between them,
so a budget near that floor leaves no cross-topic distractors and inflates
recall. The default is 40,000 and the build reports the gold/distractor
split, warning when there are none.
build_databases opens each collection by configured name with a scope of
one, since populate_db writes to a single database. The operator entry
point emits the config for the partition it just built, so a config cannot
search a differently-partitioned build.
Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc
