diff --git a/CHANGELOG.md b/CHANGELOG.md index c063d37e..9bfc79a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,11 +5,12 @@ - `evaluations run --filter/-f CLAUSE`: SQL `WHERE` clause over document columns, applied to the retrieval benchmark's searches and to every capability search during QA. Recorded as `document_filter` in experiment metadata. - `mtrag_clapnq` / `mtrag_clapnq_rewrite` / `mtrag_clapnq_live` / `mtrag_clapnq_live_uncompacted` evaluation datasets: multi-turn QA with gold-prefix and live-session conversation replay, live arms with and without `EvidenceCompactionCapability`, Recall@k/nDCG@k retrieval metrics, eligibility-aware citation scoring, refusal precision/recall, per-turn tool-traffic attributes, and `citation_status` / `turn_citation_status` eval attributes. +- Raw chunk metadata is now exposed to search and citation results, through `SearchResult.chunk_meta` and `Citation.chunk_meta`. For context-expanded results, the metadata is that of the anchor chunk. + ### Changed - `import_documents` embeds chunks across the whole batch in one pass instead of per document. -- Raw chunk metadata is now exposed to search and citation results, through `SearchResult.chunk_meta` and `Citation.chunk_meta`. For context-expanded results, the metadata is that of the anchor chunk. ### Removed diff --git a/docs/custom-pipelines.md b/docs/custom-pipelines.md index aa003a85..e4ad3202 100644 --- a/docs/custom-pipelines.md +++ b/docs/custom-pipelines.md @@ -117,9 +117,7 @@ Chunks are returned with: - `embedding` - `None` (not yet embedded) - `document_id` - `None` (not yet stored) -A custom `DocumentChunker` can provide other keys and values in `metadata`. They will -be stored with the chunk and are accessible when it is returned in a search result or citation, within -`SearchResult.chunk_meta` / `Citation.chunk_meta`. +A custom `DocumentChunker` can provide other keys and values in `metadata`. They will be stored with the chunk and are accessible when it is returned in a search result or citation, within `SearchResult.chunk_meta` / `Citation.chunk_meta`. ## Embed diff --git a/docs/python.md b/docs/python.md index e261d6cd..f7ed4854 100644 --- a/docs/python.md +++ b/docs/python.md @@ -196,7 +196,7 @@ for result in results: print(f"Document ID: {result.document_id}") ``` -Each result carries the parent document's metadata in `result.document_meta`. It is not shown to the model during QA. +Each result carries the parent document's metadata in `result.document_meta` and the relevant chunk's verbatim metadata in `result.chunk_meta`. Neither is shown to the model during QA. Search with different search types: ```python diff --git a/haiku_rag_slim/haiku/rag/store/models/chunk.py b/haiku_rag_slim/haiku/rag/store/models/chunk.py index 9a6cbadb..2a75fd4a 100644 --- a/haiku_rag_slim/haiku/rag/store/models/chunk.py +++ b/haiku_rag_slim/haiku/rag/store/models/chunk.py @@ -133,7 +133,7 @@ class SearchResult(BaseModel): ``document_meta`` carries the parent document's metadata for citation consumers (UIs). Never part of ``format_for_agent`` output. - ``chunk_meta``is the anchor chunk's unparsed ``Chunk.metadata`` and does not + ``chunk_meta`` is the anchor chunk's unparsed ``Chunk.metadata`` and does not include the metadata of any other chunks merged with it. Never part of ``format_for_agent`` output. """