Commit graph

16 commits

Author SHA1 Message Date
Yiorgis Gozadinos
5b0444043a
Batch context expansion across documents
`expand_with_items` fetched its own inputs per document: one query to resolve
refs to positions, one for the window of items around them. A result set spanning
N documents cost 2N queries, which was 10 of the 18 measured for a limit=5 search
on a remote object-store corpus.

`expand_context` now does both fetches once for every document it is expanding,
and `expand_with_items` takes the positions and items it needs. Two queries for
one document, and two for five.

Each document keeps its own inclusive window in `get_items_in_ranges`. Positions
repeat across documents, so a shared range would splice one document's items into
another's context.
2026-08-18 16:35:17 +03:00
Lawrence Akka
09ff101349 Expose all chunk metadata on search results and citations through SearchResult.chunk_meta and Citation.chunk_meta 2026-08-17 19:42:23 +02:00
Yiorgis Gozadinos
dddaf0f84c
Remove unreachable code and fix two defects it surfaced
Delete repository methods with no callers (SettingsRepository CRUD,
ChunkRepository.update/delete/get_chunks_in_range), the DataFrame branch of
_process_search_results whose only caller always passes a query, and guards
that cannot be reached from their call sites: the rebuild mode=None default,
the staging drop already performed by _resolve_rebuild_recovery, the empty
batch skip, two context fast paths, the doctor prefix guard, the poller
_task attribute that is never assigned, and a docling caption fallback for
a field name no item class defines.

set_haiku_version built a recreated settings row from the process-global
Config rather than the store's own, so a store opened with a custom config
stamped global settings into the database.

check_source_accessible called urlparse outside its try block, so a stored
URI with a malformed IPv6 host raised ValueError instead of reporting the
source as inaccessible, aborting the whole rebuild sweep.
2026-07-26 18:45:48 +03:00
Yiorgis Gozadinos
75a21c82c2
Search results and citations carry document metadata 2026-07-23 10:23:48 +03:00
Yiorgis Gozadinos
82db15cf3c
Fill missing item-table pages from fully-visible inputs in context expansion 2026-07-14 10:32:45 +03:00
Yiorgis Gozadinos
ee5b7b7313
Split merged context-expansion groups that cannot afford every constituent 2026-07-14 10:15:28 +03:00
Yiorgis Gozadinos
c424f51056
Anchor and clip merged citations on the highest-scoring chunk
A merged search result took its chunk_id from whichever constituent
sorted earliest in the document, while its score was the max across the
group — so the citation's identity could point at a different, less
relevant chunk. Anchor chunk_id and the content/refs fallbacks on the
max-score constituent, clip the budget window around that same chunk so
its evidence is never trimmed away, and narrow page_numbers, doc_item_refs,
and attached image bytes to the items that survive the clip.
2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
0bcf34363a
Carry merged chunk ids on SearchResult and Citation 2026-07-09 11:05:41 +03:00
Yiorgis Gozadinos
b14152a45b
Keep picture and table hits section-bounded during context expansion 2026-07-09 11:05:40 +03:00
Yiorgis Gozadinos
b94e13083f
Hard-cap context expansion at max_context_chars
A single oversized document_items row (e.g. a spreadsheet converted to one
table) expanded far past search.max_context_chars and could overflow the
model context window. _expand_outward only used the budget as a soft
accumulation threshold and expand_with_items never capped the joined result.

Add _clip_to_budget to clip each expanded result to max_context_chars,
returning a window centered on the matched chunk (via _evidence_anchors) so
the retrieved evidence survives the cut.
2026-06-27 09:46:36 +03:00
Yiorgis Gozadinos
e3314e36fd
search: only attach picture bytes from pre-expansion chunks 2026-05-19 15:03:22 +03:00
Yiorgis Gozadinos
b01c649684
surface picture image bytes in SearchResult and emit multimodal ToolReturn from the agent search tool 2026-04-30 12:43:05 +03:00
Yiorgis Gozadinos
ff8ad0879c
fix context expansion: respect section boundaries, remove max_context_items 2026-04-20 15:18:16 +03:00
Yiorgis Gozadinos
d3c8ed62a8
preserve original chunk when expansion produces less content 2026-04-16 12:11:54 +03:00
Yiorgis Gozadinos
3c1a33779f
use windowed fetch for context expansion instead of loading all items 2026-04-16 12:11:53 +03:00
Yiorgis Gozadinos
b6113bf8ab
replace fixed-radius expansion with section-bounded algorithm
Context expansion is now automatic and structure-aware. For structured
documents, expands within the section containing the match. For sections
that exceed the budget or are too small, expands item-by-item outward
skipping noise labels. Unstructured documents use budget-based outward
expansion. Results sorted by relevance score.
2026-04-16 12:11:53 +03:00