Yiorgis Gozadinos
295570797c
Disable Logfire scrubbing in evaluations
2026-06-08 11:51:13 +03:00
Yiorgis Gozadinos
e9bd56467c
Add T²-RAGBench leaderboard submission exporter
2026-06-08 11:51:02 +03:00
Yiorgis Gozadinos
a3a73f1331
Add --filter-ids to run QA on a case-id subset
2026-06-08 09:36:01 +03:00
Yiorgis Gozadinos
1d7d540270
Match the numeric scale convention in Number-Match
2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
d5b0aafa2b
Add T²-RAGBench TAT-DQA subset; generalize subset layout
2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
de9731d5f3
Score Number-Match on the declared ANSWER line, magnitude and ×100 scale
2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
390deb4203
Normalize unicode signs
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
80594cd38c
Handle percent/decimal convention in Number-Match
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
c2a48b55c0
Add deterministic Number-Match QA scoring for T²-RAGBench
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
9f1bd9940a
Add T²-RAGBench FinQA evaluation dataset
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
eb74525084
Lift analysis-skill citation rate via SKILL.md tightening
2026-06-01 18:58:52 +03:00
Yiorgis Gozadinos
e653c49fce
Remove unecessary Mean Reciprocal Rank metric
2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
00f2a40a60
Refresh benchmarks doc and remove unused eval datasets
2026-06-01 10:40:51 +03:00
Yiorgis Gozadinos
392c74039b
bump pydantic-ai-slim to 1.100 and haiku.skills to 0.17
...
Migrate off two APIs slated for removal in pydantic-ai 2.0:
- Agent(tool_retries=, output_retries=) -> Agent(retries={"tools": ...,
"output": ...}) in the LLM-as-judge evaluator.
- Evaluator.evaluation_name class attribute -> overriding
get_default_evaluation_name() on the citation MRR / MAP evaluators.
haiku.skills 0.17.0 already migrated its internal AGUIAdapter,
MCPToolset and ProcessEventStream usage; no further changes needed on
our side beyond the pin bumps.
2026-05-21 13:20:54 +03:00
Yiorgis Gozadinos
f9392cee46
evaluations: judge model moves to config.evaluations.judge
2026-05-20 12:56:22 +03:00
Yiorgis Gozadinos
8c57a3ca99
Drop the multi-agent research workflow
2026-05-20 12:46:48 +03:00
Yiorgis Gozadinos
ceb645564c
Drop GEPA prompt optimization and --target qa from evaluations
2026-05-19 11:01:07 +03:00
Yiorgis Gozadinos
74dea65f21
analysis.model inherits qa.model when unset; per-skill vision gate
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
1df6530d03
Bump pydantic-ai, prepare for 2.*
2026-05-18 15:14:29 +03:00
Yiorgis Gozadinos
cc59dc7203
Use upload_large_folder for hf upload
2026-05-06 13:12:08 +03:00
Yiorgis Gozadinos
ce7201271f
split open_rag_bench dataset into orb_text and orb_multimodal variants
2026-05-06 12:49:03 +03:00
Yiorgis Gozadinos
a45d82f206
Update benchmark
2026-05-05 13:58:48 +03:00
Yiorgis Gozadinos
5c3a864a0b
rename open_rag_bench eval DBs to text/multimodal variants
2026-05-05 12:55:06 +03:00
Yiorgis Gozadinos
75c8e83515
Allow uri override on create_document_from_source
2026-04-30 16:36:35 +03:00
Yiorgis Gozadinos
8962987ff9
pin judge model to ollama:qwen3.6
2026-04-29 12:24:06 +03:00
Yiorgis Gozadinos
7bfb818d77
remove dataset-specific system prompts
2026-04-28 14:33:25 +03:00
Yiorgis Gozadinos
7d288c525e
citation retrieval scoring
2026-04-28 12:46:16 +03:00
Yiorgis Gozadinos
d00befd0c4
benchmark RAG and analysis skills via --target
2026-04-28 12:09:19 +03:00
Yiorgis Gozadinos
ff8ad0879c
fix context expansion: respect section boundaries, remove max_context_items
2026-04-20 15:18:16 +03:00
Yiorgis Gozadinos
4f16714430
prevent stale files from accumulating on HF during uploads.
2026-04-16 13:45:24 +03:00
Yiorgis Gozadinos
e92f4774fa
Improve judge prompt
2026-04-16 12:11:54 +03:00
Yiorgis Gozadinos
b6113bf8ab
replace fixed-radius expansion with section-bounded algorithm
...
Context expansion is now automatic and structure-aware. For structured
documents, expands within the section containing the match. For sections
that exceed the budget or are too small, expands item-by-item outward
skipping noise labels. Unstructured documents use budget-based outward
expansion. Results sorted by relevance score.
2026-04-16 12:11:53 +03:00
Yiorgis Gozadinos
e4dc60c4f1
Update core dependencies
2026-04-03 10:14:27 +03:00
Yiorgis Gozadinos
fd327996b8
Configurable judge and reflect models for evaluations
2026-03-18 17:14:11 +02:00
Yiorgis Gozadinos
6ac5f277d8
Rename iterations to num_candidates
2026-03-12 13:00:51 +02:00
Yiorgis Gozadinos
5b3ad9bae7
Refactor QAPromptAdapter to accept QA agent directly, rewrite mock-heavy tests
2026-03-12 12:35:39 +02:00
Yiorgis Gozadinos
24275d2ef9
Fixes
2026-03-12 12:19:21 +02:00
Yiorgis Gozadinos
c785a54f5f
Respect config.prompts.qa in evaluations benchmark and optimization
2026-03-12 12:05:51 +02:00
Yiorgis Gozadinos
6812435ba0
cleanup
2026-03-12 12:05:50 +02:00
Yiorgis Gozadinos
7c399d1ff1
Remove evaluation ceremony from gepa judge
2026-03-12 12:05:50 +02:00
Yiorgis Gozadinos
8bac8b4104
Properly choose training/eval set
2026-03-12 12:05:50 +02:00
Yiorgis Gozadinos
45f7ac8d1b
run_optimization should be sync
2026-03-12 12:05:25 +02:00
Yiorgis Gozadinos
60b1d3c013
Add GEPA prompt optimization for QA evaluations
2026-03-12 12:05:25 +02:00
Yiorgis Gozadinos
e2749ad2a6
Cap QA agent search iterations to reduce response time
2026-03-11 12:13:43 +02:00
Yiorgis Gozadinos
40ff54eac0
Set params for evaluations judge
2026-03-05 13:59:20 +02:00
Yiorgis Gozadinos
30ebd6694b
Update dependencies
2026-03-05 13:01:53 +02:00
Yiorgis Gozadinos
4855051936
Add research() to client and add haiku.skills dependency, remove --deep flag and simplify app
2026-02-20 16:36:47 +02:00
Yiorgis Gozadinos
42f2d0fd0e
docs
2026-01-31 10:13:22 +02:00
Yiorgis Gozadinos
2a99cb09bf
Evaluate using ask --deep
2026-01-29 14:22:18 +02:00
Yiorgis Gozadinos
39f4937916
Handle missing datasets from huggingface
2026-01-29 13:31:31 +02:00
