danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

0 releases 185 tags

RSS feed
  • archive/eval-mtrag-strip-speaker-tag a60d0f45d3

    Strip the speaker tag from MTRAG retrieval queries

    Ghost released this 2026-09-01 07:17:35 +02:00 | 42 commits to main since this release

    The query files encode the speaker into the text, so every retrieval query
    arrived as "|user|: How many teams are in the NFL?". That reaches the
    embedder, the BM25 query and the reranker's query.

    Measured paired over 777 queries on four domains: stripping is worth +3.60pp
    recall@5 with a reranker (94 queries better, 33 worse, 650 tied) and nothing
    without one (40 better, 40 worse). A cross-encoder scores query against
    document directly, so junk tokens on the query side hurt it where a
    bag-of-words branch and a pooled embedding absorb them.

    Confined to the retrieval query files: 208 of 208 in both lastturn and
    rewrite carry it, while QA turn texts, answers and live questions carry none.

    Changes retrieval scores for mtrag_clapnq, mtrag_clapnq_rewrite,
    mtrag_federated and mtrag_pooled. The single-database direction is small and
    signed: hybrid -0.36pp, vector -1.83pp, FTS +1.25pp, the branches moving
    oppositely and nearly cancelling.

    Claude-Session: https://claude.ai/code/session_01WhudUtZm6qqiuv8Y1sbwSc

    Downloads