Yiorgis Gozadinos
de9731d5f3
Score Number-Match on the declared ANSWER line, magnitude and ×100 scale
2026-06-06 14:52:05 +03:00
Yiorgis Gozadinos
390deb4203
Normalize unicode signs
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
80594cd38c
Handle percent/decimal convention in Number-Match
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
c2a48b55c0
Add deterministic Number-Match QA scoring for T²-RAGBench
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
9f1bd9940a
Add T²-RAGBench FinQA evaluation dataset
2026-06-06 14:52:04 +03:00
Yiorgis Gozadinos
916b84bff4
vb
2026-06-05 12:09:57 +03:00
Yiorgis Gozadinos
e70f271ac7
vb
2026-06-04 14:13:55 +03:00
Yiorgis Gozadinos
4eae86225e
vb
2026-06-03 14:26:58 +03:00
Yiorgis Gozadinos
eb74525084
Lift analysis-skill citation rate via SKILL.md tightening
2026-06-01 18:58:52 +03:00
Yiorgis Gozadinos
8dbcd4bd7a
vb, changelog
2026-06-01 18:54:22 +03:00
Yiorgis Gozadinos
e653c49fce
Remove unecessary Mean Reciprocal Rank metric
2026-06-01 10:40:52 +03:00
Yiorgis Gozadinos
00f2a40a60
Refresh benchmarks doc and remove unused eval datasets
2026-06-01 10:40:51 +03:00
Yiorgis Gozadinos
62be23d130
vb
2026-05-29 11:47:12 +03:00
Yiorgis Gozadinos
3828a91c37
vb
2026-05-27 16:23:43 +03:00
Yiorgis Gozadinos
14d9951016
vb
2026-05-21 14:32:13 +03:00
Yiorgis Gozadinos
392c74039b
bump pydantic-ai-slim to 1.100 and haiku.skills to 0.17
...
Migrate off two APIs slated for removal in pydantic-ai 2.0:
- Agent(tool_retries=, output_retries=) -> Agent(retries={"tools": ...,
"output": ...}) in the LLM-as-judge evaluator.
- Evaluator.evaluation_name class attribute -> overriding
get_default_evaluation_name() on the citation MRR / MAP evaluators.
haiku.skills 0.17.0 already migrated its internal AGUIAdapter,
MCPToolset and ProcessEventStream usage; no further changes needed on
our side beyond the pin bumps.
2026-05-21 13:20:54 +03:00
Yiorgis Gozadinos
98b0c068e8
vb
2026-05-20 14:34:33 +03:00
Yiorgis Gozadinos
f9392cee46
evaluations: judge model moves to config.evaluations.judge
2026-05-20 12:56:22 +03:00
Yiorgis Gozadinos
8c57a3ca99
Drop the multi-agent research workflow
2026-05-20 12:46:48 +03:00
Yiorgis Gozadinos
ceb645564c
Drop GEPA prompt optimization and --target qa from evaluations
2026-05-19 11:01:07 +03:00
Yiorgis Gozadinos
74dea65f21
analysis.model inherits qa.model when unset; per-skill vision gate
2026-05-18 16:49:43 +03:00
Yiorgis Gozadinos
1df6530d03
Bump pydantic-ai, prepare for 2.*
2026-05-18 15:14:29 +03:00
Yiorgis Gozadinos
97a7eb246c
vb
2026-05-14 16:03:33 +03:00
Yiorgis Gozadinos
862a209137
vb
2026-05-13 16:53:35 +03:00
Yiorgis Gozadinos
cc59dc7203
Use upload_large_folder for hf upload
2026-05-06 13:12:08 +03:00
Yiorgis Gozadinos
ce7201271f
split open_rag_bench dataset into orb_text and orb_multimodal variants
2026-05-06 12:49:03 +03:00
Yiorgis Gozadinos
a45d82f206
Update benchmark
2026-05-05 13:58:48 +03:00
Yiorgis Gozadinos
5c3a864a0b
rename open_rag_bench eval DBs to text/multimodal variants
2026-05-05 12:55:06 +03:00
Yiorgis Gozadinos
75c8e83515
Allow uri override on create_document_from_source
2026-04-30 16:36:35 +03:00
Yiorgis Gozadinos
6a77ce92a9
extract picture bytes to document_items.picture_data at ingest, strip them from the docling_document blob, and add 0.45.0 migration to backfill existing
...
databases
2026-04-30 11:26:37 +03:00
Yiorgis Gozadinos
29bc42514e
vb
2026-04-29 13:45:36 +03:00
Yiorgis Gozadinos
8962987ff9
pin judge model to ollama:qwen3.6
2026-04-29 12:24:06 +03:00
Yiorgis Gozadinos
5e31c15907
docs
2026-04-28 14:44:27 +03:00
Yiorgis Gozadinos
7bfb818d77
remove dataset-specific system prompts
2026-04-28 14:33:25 +03:00
Yiorgis Gozadinos
7d288c525e
citation retrieval scoring
2026-04-28 12:46:16 +03:00
Yiorgis Gozadinos
d00befd0c4
benchmark RAG and analysis skills via --target
2026-04-28 12:09:19 +03:00
Yiorgis Gozadinos
6807f082ec
vb
2026-04-25 09:35:47 +03:00
Yiorgis Gozadinos
81ba248483
vb
2026-04-24 17:47:57 +03:00
Yiorgis Gozadinos
4f058bf484
vb
2026-04-22 17:00:03 +03:00
Yiorgis Gozadinos
bb9e7b28ca
vb
2026-04-22 16:18:44 +03:00
Yiorgis Gozadinos
8a23108fbd
vb
2026-04-20 16:36:32 +03:00
Yiorgis Gozadinos
ff8ad0879c
fix context expansion: respect section boundaries, remove max_context_items
2026-04-20 15:18:16 +03:00
Yiorgis Gozadinos
1049586469
vb
2026-04-17 16:24:25 +03:00
Yiorgis Gozadinos
4f16714430
prevent stale files from accumulating on HF during uploads.
2026-04-16 13:45:24 +03:00
Yiorgis Gozadinos
e92f4774fa
Improve judge prompt
2026-04-16 12:11:54 +03:00
Yiorgis Gozadinos
b6113bf8ab
replace fixed-radius expansion with section-bounded algorithm
...
Context expansion is now automatic and structure-aware. For structured
documents, expands within the section containing the match. For sections
that exceed the budget or are too small, expands item-by-item outward
skipping noise labels. Unstructured documents use budget-based outward
expansion. Results sorted by relevance score.
2026-04-16 12:11:53 +03:00
Yiorgis Gozadinos
0d6df472af
bump ag-ui protocol deps, fix duplicate React key in chat
2026-04-14 12:50:25 +03:00
Yiorgis Gozadinos
dcde86a5f1
vb
2026-04-09 15:33:19 +03:00
Yiorgis Gozadinos
62a436dd9d
vb
2026-04-08 14:04:32 +03:00
Yiorgis Gozadinos
3d30840e9d
vb
2026-04-07 16:36:27 +03:00
Yiorgis Gozadinos
e4dc60c4f1
Update core dependencies
2026-04-03 10:14:27 +03:00
Yiorgis Gozadinos
5c4164d224
vb
2026-04-01 13:32:56 +03:00
Yiorgis Gozadinos
127535bc56
vb
2026-03-28 10:05:50 +02:00
Yiorgis Gozadinos
13a71e34a7
vb
2026-03-27 16:39:22 +02:00
Yiorgis Gozadinos
f46b8195ed
vb
2026-03-26 14:15:09 +02:00
Yiorgis Gozadinos
f5014f39bf
vb
2026-03-24 17:46:51 +02:00
Yiorgis Gozadinos
8a5188aa01
vb
2026-03-24 14:23:29 +02:00
Yiorgis Gozadinos
83ea6347b6
Fix CI failures: update tests for pydantic-ai 1.70.0 and FastMCP API changes
2026-03-24 14:02:03 +02:00
Yiorgis Gozadinos
fd327996b8
Configurable judge and reflect models for evaluations
2026-03-18 17:14:11 +02:00
Yiorgis Gozadinos
9243f56e1c
vb
2026-03-16 14:30:08 +02:00
Yiorgis Gozadinos
f961f00f2c
vb
2026-03-13 13:24:31 +02:00
Yiorgis Gozadinos
a0025e9203
Fix ty
2026-03-13 12:51:35 +02:00
Yiorgis Gozadinos
7696b3f98f
vb
2026-03-12 14:46:13 +02:00
Yiorgis Gozadinos
6ac5f277d8
Rename iterations to num_candidates
2026-03-12 13:00:51 +02:00
Yiorgis Gozadinos
5b3ad9bae7
Refactor QAPromptAdapter to accept QA agent directly, rewrite mock-heavy tests
2026-03-12 12:35:39 +02:00
Yiorgis Gozadinos
24275d2ef9
Fixes
2026-03-12 12:19:21 +02:00
Yiorgis Gozadinos
c785a54f5f
Respect config.prompts.qa in evaluations benchmark and optimization
2026-03-12 12:05:51 +02:00
Yiorgis Gozadinos
6812435ba0
cleanup
2026-03-12 12:05:50 +02:00
Yiorgis Gozadinos
7c399d1ff1
Remove evaluation ceremony from gepa judge
2026-03-12 12:05:50 +02:00
Yiorgis Gozadinos
8bac8b4104
Properly choose training/eval set
2026-03-12 12:05:50 +02:00
Yiorgis Gozadinos
66f88f7872
Test evaluations
2026-03-12 12:05:50 +02:00
Yiorgis Gozadinos
45f7ac8d1b
run_optimization should be sync
2026-03-12 12:05:25 +02:00
Yiorgis Gozadinos
adc83e5f81
Tests for gepa
2026-03-12 12:05:25 +02:00
Yiorgis Gozadinos
60b1d3c013
Add GEPA prompt optimization for QA evaluations
2026-03-12 12:05:25 +02:00
Yiorgis Gozadinos
8d6617d63f
vb
2026-03-11 17:08:22 +02:00
Yiorgis Gozadinos
e2749ad2a6
Cap QA agent search iterations to reduce response time
2026-03-11 12:13:43 +02:00
Yiorgis Gozadinos
f366aea905
vb
2026-03-06 11:12:54 +02:00
Yiorgis Gozadinos
40ff54eac0
Set params for evaluations judge
2026-03-05 13:59:20 +02:00
Yiorgis Gozadinos
30ebd6694b
Update dependencies
2026-03-05 13:01:53 +02:00
Yiorgis Gozadinos
878423ea2e
vb
2026-03-04 15:02:34 +02:00
Yiorgis Gozadinos
8a859fbe3c
vb
2026-03-03 17:25:53 +02:00
Yiorgis Gozadinos
99ca3605e7
vb
2026-02-28 14:11:25 +02:00
Yiorgis Gozadinos
50116bb49a
vb
2026-02-26 11:21:46 +02:00
Yiorgis Gozadinos
2acb0292e4
vb
2026-02-24 13:30:01 +02:00
Yiorgis Gozadinos
2cafb5c8cc
vb
2026-02-20 19:41:17 +02:00
Yiorgis Gozadinos
e186e13cd2
vb
2026-02-20 19:01:16 +02:00
Yiorgis Gozadinos
4855051936
Add research() to client and add haiku.skills dependency, remove --deep flag and simplify app
2026-02-20 16:36:47 +02:00
Yiorgis Gozadinos
0c270ba42f
vb
2026-02-19 14:28:13 +02:00
Yiorgis Gozadinos
c62fe5fce8
vb
2026-02-17 13:56:43 +02:00
Yiorgis Gozadinos
2d5fbd06f2
Update dependencies
2026-02-17 13:12:08 +02:00
Yiorgis Gozadinos
53f6cd3579
vb
2026-02-16 13:12:06 +02:00
Yiorgis Gozadinos
aaa9b6a32f
vb
2026-02-10 13:29:30 +02:00
Yiorgis Gozadinos
17c2136b4e
vb
2026-02-06 12:48:16 +01:00
Yiorgis Gozadinos
1e22937482
vb
2026-01-31 11:12:38 +02:00
Yiorgis Gozadinos
42f2d0fd0e
docs
2026-01-31 10:13:22 +02:00
Yiorgis Gozadinos
4264cdcc24
vb
2026-01-29 15:35:42 +02:00
Yiorgis Gozadinos
2a99cb09bf
Evaluate using ask --deep
2026-01-29 14:22:18 +02:00
Yiorgis Gozadinos
39f4937916
Handle missing datasets from huggingface
2026-01-29 13:31:31 +02:00
Yiorgis Gozadinos
bb634e68a4
vb
2026-01-27 11:17:17 +02:00
Yiorgis Gozadinos
7c9d70fa62
vb
2026-01-26 16:46:33 +02:00
Yiorgis Gozadinos
369b3a4bf7
No zip
2026-01-26 16:28:19 +02:00
Yiorgis Gozadinos
9b166969e2
Introduce huggingface dataset for sharing evaluation dbs
2026-01-26 15:53:37 +02:00
Yiorgis Gozadinos
c7a9ad8583
Customize orb QA prompt to not use LaTeX as gpt-oss Ollama implementation fails to parse it properly
2026-01-24 13:48:09 +02:00
Yiorgis Gozadinos
47d355ed0a
Update dependencies
2026-01-23 11:57:02 +02:00
Yiorgis Gozadinos
35ebdd67e2
vb
2026-01-22 17:48:39 +02:00
Yiorgis Gozadinos
0d3ac5142d
Fix answer in orb
2026-01-22 15:08:33 +02:00
Yiorgis Gozadinos
de022639b6
Correctly lookup docs for the orb dataset
2026-01-22 14:17:09 +02:00
Yiorgis Gozadinos
e6b239bd6c
Add evaluation dataset for multi-modal q&a
2026-01-22 14:17:09 +02:00
Yiorgis Gozadinos
32c64d3d11
vb
2026-01-22 12:59:32 +02:00
Yiorgis Gozadinos
ea7a4a9a8f
apply dotenv usecwd fix to remaining files
...
Apply the same find_dotenv(usecwd=True) fix from cli.py to:
- evaluations/evaluations/benchmark.py
- app/backend/main.py
Co-Authored-By: Tianyi Cui <contact@tianyicui.com>
2026-01-22 12:41:35 +02:00
Yiorgis Gozadinos
b96e5a3be1
Additional prompts needing adapting. Missing cassettes.
2026-01-21 15:17:56 +02:00
Yiorgis Gozadinos
e7ce9694b5
vb
2026-01-20 15:47:37 +02:00
Yiorgis Gozadinos
1022d5dafd
Update dependencies
2026-01-19 16:23:26 +02:00
Yiorgis Gozadinos
5e55c0df54
replace pyright with ty type checker
2026-01-19 15:50:14 +02:00
Yiorgis Gozadinos
2c264f4afc
vb
2026-01-19 14:54:18 +02:00
Yiorgis Gozadinos
5f2f180960
vb
2026-01-16 15:27:55 +02:00
Yiorgis Gozadinos
12e5e9ebfb
vb
2026-01-15 16:24:03 +02:00
Yiorgis Gozadinos
157773ef15
vb
2026-01-15 11:47:49 +02:00
Yiorgis Gozadinos
2930679dc1
vb
2026-01-13 19:47:27 +02:00
Yiorgis Gozadinos
275b09c8c5
vb
2026-01-13 18:34:45 +02:00
Yiorgis Gozadinos
eb5d0b0622
Use tui as an optional group. vb
2026-01-13 12:30:19 +02:00
Yiorgis Gozadinos
7cc561d1db
Refactor to make agents a top-level module. Bring in the conversational agent from the app
2026-01-12 12:37:08 +02:00
Yiorgis Gozadinos
747acd6f6a
vb
2026-01-12 12:08:28 +02:00
Yiorgis Gozadinos
ff1f19bb08
vb
2026-01-08 18:50:39 +02:00
Yiorgis Gozadinos
92f0fd956f
vb
2026-01-08 10:25:25 +02:00
Yiorgis Gozadinos
ef187767ed
vb
2026-01-07 11:20:48 +02:00
Yiorgis Gozadinos
af340ac410
vb
2026-01-05 12:15:41 +02:00
Yiorgis Gozadinos
271af6cb3c
Update core dependencies
2026-01-05 11:52:38 +02:00
Yiorgis Gozadinos
c9cb005d07
Use FTS on contextualized text
2025-12-29 14:50:07 +02:00
Yiorgis Gozadinos
1f654917b8
vb
2025-12-26 13:56:29 +02:00
Yiorgis Gozadinos
5c7f033d2c
vb
2025-12-19 12:35:35 +02:00
Yiorgis Gozadinos
e8e67d336c
Fix get_model() uses where default Config is forced
2025-12-18 15:10:31 +02:00
Yiorgis Gozadinos
1dbbaf6dd8
vb
2025-12-18 12:21:16 +02:00
Yiorgis Gozadinos
40e0ea090d
Add option to override vacuum interval
2025-12-12 16:04:06 +02:00
Yiorgis Gozadinos
056b9ad090
Use periodic vacuum to prevent disk exhaustion with large datasets
2025-12-12 15:53:23 +02:00
Yiorgis Gozadinos
7526735059
hotpotqa adapter
2025-12-12 10:56:37 +02:00
Yiorgis Gozadinos
f6073a0db9
vb
2025-12-12 10:48:21 +02:00
Yiorgis Gozadinos
00d621e211
vb
2025-12-11 16:31:21 +02:00
Yiorgis Gozadinos
802f058205
Move search related config under SearchConfig
2025-12-09 12:28:28 +02:00
Yiorgis Gozadinos
06b4f6a224
Add format parameter for text-to-DoclingDocument conversion
2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
9a5f8e4368
Add type-aware context expansion for search results
2025-12-08 15:56:21 +02:00
Yiorgis Gozadinos
dab04668bd
Dependency updates
2025-12-08 15:56:02 +02:00
Yiorgis Gozadinos
a57203065a
Fix test
2025-12-08 15:55:28 +02:00
Yiorgis Gozadinos
d808c6c425
Simplify citations in qa & graph agents
2025-12-08 15:55:27 +02:00
Yiorgis Gozadinos
ace58a7034
Update evaluations and a2a
2025-12-08 15:54:56 +02:00
Yiorgis Gozadinos
fe4ac530be
DB migration
2025-12-08 15:54:56 +02:00
Yiorgis Gozadinos
3525fae625
Use EmbeddingModelConfig similar to ModelConfig for embeddings
2025-12-02 11:55:31 +02:00
Yiorgis Gozadinos
f07590bd5b
vb
2025-12-01 17:58:47 +02:00
Yiorgis Gozadinos
10aa2a7cf2
vb
2025-11-28 10:15:59 +02:00
Yiorgis Gozadinos
eee29578ba
vb
2025-11-27 15:24:39 +02:00
