Merge pull request #413 from ggozad/feat/analysis-citation-improvement

Improve citation rate for analysis skill in lesser models (gemma4)

docs/benchmarks.md

@@ -122,13 +122,14 @@ Two approaches are benchmarked separately:
 
 ##### QA accuracy + citation retrieval
 
-| Embedding Model                          | Target          | Skill model                  | Cases | QA accuracy | Mean `cited_map` |
-|------------------------------------------|-----------------|------------------------------|------:|-------------|------------------|
-| `Qwen/Qwen3-VL-Embedding-8B`             | `rag-skill`     | `vllm:Gemma-4-26B-A4B-NVFP4` |  1409 | 0.89        | —                |
-| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `rag-skill`     | `vllm:Gemma-4-26B-A4B-NVFP4` |  3045 | 0.92        | 0.93             |
-| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `analysis-skill`| `vllm:Gemma-4-26B-A4B-NVFP4` |  3045 | 0.94        | 0.35             |
+| Embedding Model                          | Target          | Skill model                       | Cases | QA accuracy | Mean `cited_map` |
+|------------------------------------------|-----------------|-----------------------------------|------:|-------------|------------------|
+| `Qwen/Qwen3-VL-Embedding-8B`             | `rag-skill`     | `vllm:Gemma-4-26B-A4B-NVFP4`      |  1409 | 0.89        | —                |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `rag-skill`     | `vllm:Gemma-4-26B-A4B-NVFP4`      |  3045 | 0.92        | 0.93             |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `analysis-skill`| `vllm:Gemma-4-26B-A4B-NVFP4`      |  3045 | 0.94        | 0.78             |
+| `nvidia/llama-nemotron-embed-vl-1b-v2`   | `analysis-skill`| `vllm:Qwen3.6-35B-A3B-NVFP4`      |  3045 | 0.95        | 0.93             |
 
-*Measured on haiku.rag v0.50.0, no reranker (no multimodal reranker available), judged by `vllm:Qwen3.6-35B-A3B-NVFP4`. Qwen3-VL covered 1409 / 3045 cases and predates the `cited_map` evaluator. The analysis skill answers via Python sandbox over the document VFS and only cites in ~35 % of cases, which depresses `cited_map`.*
+*Measured on haiku.rag v0.52.0, no reranker, judged by `vllm:Qwen3.6-35B-A3B-NVFP4`. Qwen3-VL covered 1409 / 3045 cases.*
 
 #### Text embedder + VLM picture descriptions
 
@@ -148,7 +149,7 @@ Two approaches are benchmarked separately:
 | `qwen3-embedding:4b`                     | Ollama / ministral-3 | `vllm:Gemma-4-26B-A4B-NVFP4` |  3045 | 0.92        | 0.80             |
 | `nvidia/llama-nemotron-embed-vl-1b-v2`   | Ollama / ministral-3 | `vllm:Gemma-4-26B-A4B-NVFP4` |  2836 | 0.96        | 0.81             |
 
-*Measured on haiku.rag v0.50.0 with `mxbai-rerank-base-v2`, judged by `vllm:Qwen3.6-35B-A3B-NVFP4`. Nemotron ran 2836 / 3045 cases (run stopped early; numbers stable from ~12% onward).*
+*Measured on haiku.rag v0.50.0 with `mxbai-rerank-base-v2`, judged by `vllm:Qwen3.6-35B-A3B-NVFP4`. Nemotron covered 2836 / 3045 cases.*
 
 ### Wix
 

evaluations/evaluations/skill_runner.py

@@ -28,6 +28,7 @@ class SkillRunResult:
     cited_chunk_ids: list[str] = field(default_factory=list)
     searched_uris: list[str] = field(default_factory=list)
     n_searches: int = 0
+    n_executions: int = 0
 
 
 async def run_skill_question(
@@ -83,10 +84,14 @@ async def run_skill_question(
                 seen_searched.add(uri)
                 searched_uris.append(uri)
 
+    executions = getattr(state, "executions", None)
+    n_executions = len(executions) if executions is not None else 0
+
     return SkillRunResult(
         answer=answer,
         cited_uris=cited_uris,
         cited_chunk_ids=cited_chunk_ids,
         searched_uris=searched_uris,
         n_searches=len(typed.searches),
+        n_executions=n_executions,
     )

haiku_rag_slim/haiku/rag/skills/rag-analysis/SKILL.md

@@ -10,7 +10,12 @@ description: >
 
 # Analysis
 
-You answer questions over a document knowledge base. Most questions are answered directly with `search → cite → answer`. Reach for `execute_code` when a question requires computation, aggregation, or structural traversal that a single search cannot deliver.
+You answer questions over a document knowledge base. Two common workflows:
+
+- **`search → cite → answer`** when the answer is grounded on specific document content. Call `cite` with the supporting chunk_ids before writing the answer.
+- **`execute_code → answer`** when the answer is a count, aggregation, listing, or structural computation over the corpus (e.g. "how many documents?", "average page count"). No `cite` is needed when no specific chunks support the answer.
+
+You can mix the two. The rule: cite when grounded on retrieved evidence; don't fabricate citations for corpus-level computation.
 
 ## Tools
 
@@ -28,16 +33,16 @@ Not supported: class definitions, generators/yield, match statements, decorators
 Search the knowledge base directly (outside code execution). Each result has a `Type:` (paragraph, table, code, list_item, picture). When the Type is `picture`, the corresponding figure may also be attached to the tool response as an image alongside the text — use it directly to answer questions about figures, diagrams, charts, screenshots.
 
 ### cite
-Register the chunk IDs that ground your answer. Call this BEFORE writing your final answer.
+Register the chunk IDs that ground your answer. **You must call `cite` before writing any final answer that uses retrieved evidence — search results, items.jsonl rows, toc.json nodes, or content.txt content.** Skipping `cite` leaves the answer ungrounded and is treated as a failure.
+
+`cite` is **not** required when your answer is a corpus-level computation that doesn't draw on specific chunks — counts, aggregations, listings, averages across documents. Don't fabricate citations for these.
 
 Chunk IDs come from two places:
 - The `chunk_id` field on `search` / `await search(...)` results
-- The `chunk_ids` field on `items.jsonl` rows (when you ground via direct file reads)
+- The `chunk_ids` field on `items.jsonl` rows / `toc.json` nodes (when you ground via direct file reads)
 
 Do NOT cite `self_ref` (`#/texts/N` style refs), `position`, or any other identifier-shaped field. They are not chunk IDs and the tool will reject them. Copy chunk IDs verbatim — they are opaque UUIDs.
 
-Every answer that uses search or file-read evidence must be backed by `cite`.
-
 ## Document Filesystem (inside execute_code)
 
 All documents are mounted as a virtual filesystem at `/documents/`:
@@ -100,12 +105,12 @@ Search results include `doc_item_refs` (e.g. `["#/texts/48", "#/tables/0"]`) tha
 ## Strategy
 
 1. Search first.
-2. If the top results contain the answer, call `cite` with the supporting chunk_ids and write a concise answer.
+2. Identify the chunk_ids from the search results that support your answer and call `cite` with them. Then write a concise answer.
 3. Reach for `execute_code` when search results are insufficient or when the task requires computation, aggregation, traversal across documents, or section-scoped reading. From inside code you can search again with different terms, or read `items.jsonl` / `toc.json` / `content.txt` directly from the document filesystem.
 4. For questions about a *known document's* structure ("which section contains X", "list the sections of doc Y", "summarise section Z"), read `/documents/{id}/toc.json` first. Each node carries `item_range` (a slice into `items.jsonl`) and `chunk_ids` (citable). Prefer this over `search()` for in-document navigation — `search()` ranks across the whole corpus and can return chunks from unrelated documents.
-5. Call `cite` with the chunk_ids that ground your answer before writing the final response.
+5. Before writing your final response, call `cite` with the chunk_ids that ground your answer.
 
-You MUST call `cite` with at least one chunk ID before producing your final answer, **unless** you are refusing for lack of information. Answers without citations are considered ungrounded. In a refusal case do **not** call `cite` — there is nothing to cite.
+You MUST call `cite` with at least one chunk ID before producing your final answer **when your answer is grounded on retrieved evidence**. Skip `cite` in two cases: (a) you are refusing for lack of information, or (b) your answer is a corpus-level computation (count, aggregation, listing) that doesn't draw on specific chunks. In those cases do **not** fabricate citations.
 
 ## Important
 
@@ -114,4 +119,5 @@ You MUST call `cite` with at least one chunk ID before producing your final answ
 - When you write code, execute it — don't describe what code would do. But not every question needs code; simple lookups are best answered by `search → cite`.
 - Use `await` for all async functions inside execute_code (search, list_documents)
 - Use `Path.read_text()` to read files — do NOT use `open()`, `with` statements, or `collections` module
-- Do NOT include chunk IDs or UUIDs in your answer text — your answer should read naturally. Use the `cite` tool separately to register citations.
+- Do NOT include chunk IDs or UUIDs in your answer text — your answer should read naturally. Use the `cite` tool separately to register citations. `cite{...}` markdown-style inline references do nothing; only an actual `cite` tool call registers a citation.
+- **Before you write your final answer, invoke the `cite` tool with the supporting chunk_ids.** This is the last tool call before answering whenever your answer draws on retrieved evidence.