Commit graph

12 commits

Author SHA1 Message Date
Yiorgis Gozadinos
bf94efc655
Simplify the eval harness and share the embed-fill path. 2026-08-17 10:56:26 +03:00
Yiorgis Gozadinos
db2b8fb883
Add compaction arms and grounding status to MTRAG live runs 2026-08-17 10:54:20 +03:00
Yiorgis Gozadinos
73d9d93db9
Add MTRAG ClapNQ multi-turn evaluation
IBM's MTRAG benchmark (ClapNQ domain, pinned repo SHA): retrieval with
Recall@k/nDCG@k against binary qrels, gold-prefix QA replaying reference
conversation prefixes as message history, and live-session replay
carrying the model's own answers and tool history across turns.

Corpus population gains a bounded, resumable batched ingest path.
ConversationInput case type with transcript rendering for the judge,
eligibility-aware citation scoring, refusal precision/recall via a
label-aware RefusalJudge, per-turn verdicts with judged-turn coverage,
and per-turn tool-traffic attributes counted from each turn's new
messages so the arrays survive prior-turn compaction.
2026-08-17 10:53:16 +03:00
Yiorgis Gozadinos
c2a48b55c0
Add deterministic Number-Match QA scoring for T²-RAGBench 2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
7bfb818d77
remove dataset-specific system prompts 2026-04-28 14:33:25 +03:00
Yiorgis Gozadinos
c785a54f5f
Respect config.prompts.qa in evaluations benchmark and optimization 2026-03-12 12:05:51 +02:00
Yiorgis Gozadinos
c7a9ad8583
Customize orb QA prompt to not use LaTeX as gpt-oss Ollama implementation fails to parse it properly 2026-01-24 13:48:09 +02:00
Yiorgis Gozadinos
e6b239bd6c
Add evaluation dataset for multi-modal q&a 2026-01-22 14:17:09 +02:00
Yiorgis Gozadinos
06b4f6a224
Add format parameter for text-to-DoclingDocument conversion 2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
4e1752a740
Default eval db location, evaluation script 2025-11-25 11:07:20 +02:00
Yiorgis Gozadinos
b5892699a0
Use Mean Reciprocal Rank for single document evaluation as metric. Use Mean Average Precision for variable document evaluation as metric 2025-11-24 14:49:30 +02:00
Yiorgis Gozadinos
2f9c907031
Restructure into uv workspace to support minimal and full installations 2025-11-04 17:59:12 +02:00
Renamed from src/evaluations/config.py (Browse further)