Merge pull request #307 from ggozad/feat/evals-gepa
Add GEPA prompt optimization for QA evaluations
This commit is contained in:
Yiorgis Gozadinos
GitHub
commit
1bee8723ae
17 changed files with 1578 additions and 368 deletions
|
|
@ -7,17 +7,18 @@ repos:
|
|||
- id: check-merge-conflict
|
||||
- id: check-toml
|
||||
- id: debug-statements
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
# Ruff version.
|
||||
rev: v0.14.8
|
||||
hooks:
|
||||
# Run the linter.
|
||||
- id: ruff
|
||||
# Run the formatter.
|
||||
- id: ruff-format
|
||||
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: ruff
|
||||
name: ruff check
|
||||
entry: uv run ruff check --force-exclude
|
||||
language: system
|
||||
types: [python]
|
||||
- id: ruff-format
|
||||
name: ruff format
|
||||
entry: uv run ruff format --force-exclude --check
|
||||
language: system
|
||||
types: [python]
|
||||
- id: ty
|
||||
name: ty check
|
||||
entry: uv run ty check
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
# Changelog
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **GEPA prompt optimization**: `evaluations optimize` command for automated QA system prompt improvement using evolutionary optimization with LLM-judged scoring. Cases are split 50/50 into train/val sets; GEPA budget is auto-computed from `--num-candidates` and dataset size.
|
||||
- **Tuning docs**: Added step 7 (Optimize QA Prompts) to the tuning workflow in `docs/tuning.md`
|
||||
- **Evaluations test coverage**: Tests for evaluators (MAP, MRR), config, benchmark helpers, dataset mappers/builders, and optimization
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Read-only mode table creation**: `--read-only` no longer creates lance tables when pointed at an empty directory. `Store._init_tables()` now raises `ReadOnlyError` when tables are missing in read-only mode.
|
||||
|
|
|
|||
333
docs/tuning.md
333
docs/tuning.md
|
|
@ -1,320 +1,101 @@
|
|||
# Tuning haiku.rag for Your Corpus
|
||||
# Tuning
|
||||
|
||||
This guide explains how to tune haiku.rag settings based on your document corpus characteristics. The right settings depend on your document types, query patterns, and accuracy requirements.
|
||||
How to adjust haiku.rag's pipeline for better retrieval and answer quality. For individual setting definitions and defaults, see [Configuration](configuration/index.md).
|
||||
|
||||
## Key Concepts
|
||||
## Pipeline Overview
|
||||
|
||||
### Retrieval vs Generation
|
||||
Documents flow through: **chunking → embedding → hybrid search (vector + FTS) → reranking → context expansion → LLM generation**. Retrieval tuning (chunking through reranking) is highest-leverage — if the LLM never sees the right chunks, no prompt or model change will help.
|
||||
|
||||
RAG has two phases:
|
||||
## Tuning Retrieval
|
||||
|
||||
1. **Retrieval**: Finding relevant chunks from your corpus
|
||||
2. **Generation**: Using those chunks to answer questions
|
||||
### Chunking
|
||||
|
||||
Poor retrieval means the LLM never sees the relevant content, regardless of how good the model is. Tuning retrieval is usually more impactful than tuning generation.
|
||||
`chunk_size` controls the granularity of retrieval. Smaller chunks match queries more precisely but carry less context each; larger chunks provide more surrounding information but dilute relevance signals. On the Wix benchmark, increasing from 256 to 512 tokens raised MAP from 0.43 to 0.45 on plain text — a modest gain that also increases token cost per result. See [Processing](configuration/processing.md#chunk-size) for configuration.
|
||||
|
||||
### Recall vs Precision
|
||||
`chunker_type` selects between `hybrid` (default) and `hierarchical` chunking. Hierarchical chunking preserves the document's heading structure and works better for deeply nested or structured content. See [Chunking Strategies](configuration/processing.md#chunking-strategies).
|
||||
|
||||
- **Recall**: What fraction of relevant documents did we find?
|
||||
- **Precision**: What fraction of retrieved documents are relevant?
|
||||
### Embedding Model
|
||||
|
||||
For RAG, recall matters more than precision. Missing a relevant chunk means wrong answers. Including an extra irrelevant chunk just wastes context tokens.
|
||||
Larger embedding models produce better representations at the cost of slower indexing and more storage. The choice of embedding model has a larger impact on retrieval quality than most other settings. See [Providers](configuration/providers.md) for available options and [Benchmarks](benchmarks.md) for real comparisons across models.
|
||||
|
||||
## Search Settings
|
||||
### Reranking
|
||||
|
||||
### `search.limit`
|
||||
When configured, a cross-encoder reranker re-scores 10x the requested candidates and returns the top results. This adds latency but improves precision — on the Wix benchmark, adding `mxbai-rerank-base-v2` raised MAP from 0.34 to 0.39 on HTML content. See [Search Settings](configuration/qa-research.md#search-settings) for how reranking integrates with search.
|
||||
|
||||
Default number of chunks to retrieve.
|
||||
### Search Settings
|
||||
|
||||
```yaml
|
||||
search:
|
||||
limit: 5 # Default
|
||||
```
|
||||
`limit` controls how many results reach the LLM. More candidates improve recall but increase token usage. See [Search Settings](configuration/qa-research.md#search-settings).
|
||||
|
||||
**When to increase:**
|
||||
`context_radius` expands text chunks with neighboring document items. Structural content (tables, code blocks, lists) expands automatically to include the complete structure. This setting matters most with small `chunk_size` values, where individual chunks may lack sufficient context. `max_context_items` and `max_context_chars` cap expansion to prevent context bloat.
|
||||
|
||||
- Complex questions requiring information from multiple sources
|
||||
- Broad topics spread across many documents
|
||||
## Tuning Generation
|
||||
|
||||
**When to decrease:**
|
||||
Model and temperature selection affect answer quality directly — see [Providers](configuration/providers.md#model-settings) for options.
|
||||
|
||||
- Simple factual questions
|
||||
- Highly focused corpus where top results are usually correct
|
||||
- Cost-sensitive deployments (fewer chunks = fewer tokens)
|
||||
`domain_preamble` prepends domain context to all agent prompts. Use it to clarify terminology, set tone, or describe what the knowledge base contains. For full prompt replacement, set `prompts.qa` directly. See [Prompt Customization](configuration/prompts.md).
|
||||
|
||||
**Typical values:** 3-10
|
||||
For automated prompt optimization, see [Prompt Optimization (GEPA)](#prompt-optimization-gepa) below.
|
||||
|
||||
### `search.context_radius`
|
||||
## What Requires a Rebuild
|
||||
|
||||
Number of adjacent DocItems to include when expanding search results. Only applies to text content (paragraphs). Tables, code blocks, and lists use structural expansion automatically.
|
||||
| Change | Rebuild required? |
|
||||
|--------|:-:|
|
||||
| `chunk_size`, `chunker_type`, `chunking_merge_peers` | Yes — `haiku-rag rebuild` |
|
||||
| Embedding model | Yes — `haiku-rag rebuild` |
|
||||
| Search settings, reranking, prompts | No |
|
||||
|
||||
```yaml
|
||||
search:
|
||||
context_radius: 0 # Default: no expansion
|
||||
```
|
||||
## Measuring Changes
|
||||
|
||||
**When to increase:**
|
||||
|
||||
- Answers require surrounding context (definitions, explanations)
|
||||
- Chunks are small and queries need more context
|
||||
- Documents have strong local coherence (adjacent paragraphs relate)
|
||||
|
||||
**When to keep at 0:**
|
||||
|
||||
- Large chunks that already contain sufficient context
|
||||
- Documents where adjacent content is often unrelated
|
||||
- When chunk boundaries align well with semantic units
|
||||
|
||||
**Typical values:** 0-3
|
||||
|
||||
### `search.max_context_items` and `search.max_context_chars`
|
||||
|
||||
Safety limits on context expansion to prevent runaway expansion.
|
||||
|
||||
```yaml
|
||||
search:
|
||||
max_context_items: 10 # Max DocItems per expanded result
|
||||
max_context_chars: 10000 # Max characters per expanded result
|
||||
```
|
||||
|
||||
Increase if expansion is being truncated and you need more context. Decrease if expanded results are too long for your LLM context window.
|
||||
|
||||
## Processing Settings
|
||||
|
||||
### `processing.chunk_size`
|
||||
|
||||
Maximum tokens per chunk (using the configured tokenizer).
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunk_size: 256 # Default
|
||||
```
|
||||
|
||||
**Trade-offs:**
|
||||
|
||||
| Smaller chunks (128-256) | Larger chunks (512-1024) |
|
||||
|-------------------------|-------------------------|
|
||||
| More precise retrieval | Better context per chunk |
|
||||
| May miss spanning content | Better recall |
|
||||
| More chunks to search | Faster search |
|
||||
| Better for specific queries | Better for broad queries |
|
||||
|
||||
**Guidance by corpus type:**
|
||||
|
||||
- **Technical documentation**: 256-512 (specific lookups)
|
||||
- **Long-form articles**: 512-1024 (need context)
|
||||
- **FAQs/short answers**: 128-256 (discrete answers)
|
||||
- **Code documentation**: 256-512 (function-level)
|
||||
|
||||
### `processing.chunker_type`
|
||||
|
||||
Chunking strategy.
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunker_type: hybrid # Default
|
||||
```
|
||||
|
||||
- **`hybrid`**: Structure-aware with token limits. Best for most documents.
|
||||
- **`hierarchical`**: Preserves document hierarchy strictly. Use for highly structured documents where hierarchy matters.
|
||||
|
||||
|
||||
### `processing.chunking_merge_peers`
|
||||
|
||||
Whether to merge adjacent small chunks that share the same section.
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunking_merge_peers: true # Default
|
||||
```
|
||||
|
||||
Keep `true` unless you specifically want very granular chunks. Merging improves embedding quality by ensuring chunks have sufficient context.
|
||||
|
||||
## Embedding Settings
|
||||
|
||||
### Model Selection
|
||||
|
||||
Embedding model choice significantly impacts retrieval quality.
|
||||
|
||||
```yaml
|
||||
embeddings:
|
||||
model:
|
||||
provider: ollama
|
||||
name: qwen3-embedding:4b
|
||||
vector_dim: 2560
|
||||
```
|
||||
|
||||
**Considerations:**
|
||||
|
||||
- Larger models generally produce better embeddings but are slower
|
||||
- Match `vector_dim` to your model's actual output dimension
|
||||
- Local models (Ollama) vs API models (OpenAI, VoyageAI) trade-off cost vs quality
|
||||
|
||||
### Contextualizing Embeddings
|
||||
|
||||
Chunks are embedded with section headings prepended (via `contextualize()`). This improves retrieval by including structural context in the embedding.
|
||||
|
||||
If your documents lack clear headings, embeddings will be based on chunk content alone.
|
||||
|
||||
## Reranking
|
||||
|
||||
Reranking retrieves more candidates than needed, then uses a cross-encoder to re-score them.
|
||||
|
||||
```yaml
|
||||
reranking:
|
||||
model:
|
||||
provider: mxbai # or cohere, zeroentropy, vllm
|
||||
name: mixedbread-ai/mxbai-rerank-base-v2
|
||||
```
|
||||
|
||||
**When to use reranking:**
|
||||
|
||||
- Embedding model has limited accuracy
|
||||
- Queries are complex or ambiguous
|
||||
- You can afford the latency (adds ~100-500ms)
|
||||
|
||||
**When to skip reranking:**
|
||||
|
||||
- Simple, specific queries
|
||||
- High-quality embedding model
|
||||
- Latency-sensitive applications
|
||||
|
||||
When reranking is enabled, haiku.rag automatically retrieves 10x the requested limit, then reranks to the final count. You don't need to adjust `search.limit` for reranking.
|
||||
|
||||
## Tuning Workflow
|
||||
|
||||
### 1. Use the Inspector
|
||||
|
||||
The inspector is your best tool for understanding how your corpus is chunked and how search behaves:
|
||||
Use the inspector for ad-hoc exploration:
|
||||
|
||||
```bash
|
||||
haiku-rag inspect
|
||||
```
|
||||
|
||||
**What to look for:**
|
||||
|
||||
- Browse documents and their chunks to see how content is split
|
||||
- Use the search modal (`/`) to test queries and see which chunks are retrieved
|
||||
- Press `c` on a chunk to view expanded context - see what additional content would be included with `context_radius > 0`
|
||||
- Check chunk sizes - are they too small (fragmented) or too large (unfocused)?
|
||||
|
||||
### 2. Test Search Manually
|
||||
|
||||
Before changing settings, run searches from the CLI to understand current behavior:
|
||||
For systematic measurement, use the `evaluations/` workspace which provides retrieval metrics (MRR, MAP) and LLM-judged QA accuracy via `pydantic-evals`:
|
||||
|
||||
```bash
|
||||
# Search and see results
|
||||
haiku-rag search "your test query" --limit 10
|
||||
# Run retrieval + QA benchmarks
|
||||
evaluations run <dataset>
|
||||
|
||||
# Try the QA to see end-to-end behavior
|
||||
haiku-rag ask "your question"
|
||||
# Skip database rebuild when only changing search/reranking/prompt settings
|
||||
evaluations run <dataset> --skip-db
|
||||
|
||||
# Limit test cases for faster iteration
|
||||
evaluations run <dataset> --limit 50
|
||||
```
|
||||
|
||||
### 3. Identify the Bottleneck
|
||||
See [Benchmarks](benchmarks.md) for dataset details, methodology, and baseline results.
|
||||
|
||||
- **Relevant chunks not retrieved**: Try larger `search.limit`, smaller `chunk_size`, or a different embedding model
|
||||
- **Too many irrelevant chunks**: Try reranking or larger `chunk_size`
|
||||
- **Chunks found but answers wrong**: Try `context_radius` expansion or a better QA model
|
||||
## Prompt Optimization (GEPA)
|
||||
|
||||
### 4. Test One Change at a Time
|
||||
The `evaluations optimize` command uses GEPA (Generalized Evolutionary Prompt Algorithm) to evolve the QA system prompt. It evaluates candidates on minibatches scored by an LLM judge, reflects on failures, proposes mutations, and accepts improvements.
|
||||
|
||||
```bash
|
||||
# After changing chunk_size, rebuild is required
|
||||
haiku-rag rebuild
|
||||
# Basic optimization
|
||||
evaluations optimize wix
|
||||
|
||||
# After changing search settings, no rebuild needed - just test again
|
||||
haiku-rag search "your test query"
|
||||
# Constrained run
|
||||
evaluations optimize repliqa --limit 40 --num-candidates 30
|
||||
|
||||
# Save result
|
||||
evaluations optimize wix --output optimized_prompt.txt
|
||||
```
|
||||
|
||||
### 5. Build Dataset-Specific Evaluations
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `--limit` | all cases | QA cases to use (split 50/50 train/val) |
|
||||
| `--num-candidates` | `50` | Number of candidate prompts to evaluate |
|
||||
| `--output` | — | Save optimized prompt to file |
|
||||
| `--config` | auto | haiku.rag YAML config path |
|
||||
| `--db` | auto | Database path override |
|
||||
|
||||
For systematic tuning, create evaluations specific to your corpus. See the `evaluations/` directory in the repository for examples of how to:
|
||||
|
||||
- Define test cases with questions and expected answers
|
||||
- Run retrieval benchmarks (MRR, MAP)
|
||||
- Run QA accuracy benchmarks with LLM judges
|
||||
|
||||
Custom evaluations let you measure the impact of configuration changes objectively rather than relying on intuition.
|
||||
|
||||
### 6. Consider Your Corpus
|
||||
|
||||
| Corpus Type | Suggested Starting Point |
|
||||
|-------------|-------------------------|
|
||||
| Technical docs | `chunk_size: 256`, `limit: 10`, `context_radius: 1` |
|
||||
| Legal/contracts | `chunk_size: 512`, `limit: 5`, `context_radius: 2` |
|
||||
| News articles | `chunk_size: 512`, `limit: 5`, `context_radius: 0` |
|
||||
| Scientific papers | `chunk_size: 256`, `limit: 5`, reranking enabled |
|
||||
| FAQs | `chunk_size: 128`, `limit: 5`, `context_radius: 0` |
|
||||
| Code repos | `chunk_size: 256`, `limit: 10`, `context_radius: 1` |
|
||||
|
||||
## Common Issues
|
||||
|
||||
### "Relevant content not being retrieved"
|
||||
|
||||
1. Check chunk boundaries - is the content split awkwardly?
|
||||
2. Try smaller chunks for more granular matching
|
||||
3. Increase `search.limit`
|
||||
4. Consider a different embedding model
|
||||
|
||||
### "Retrieved chunks lack context"
|
||||
|
||||
1. Increase `context_radius` for text content
|
||||
2. Increase `chunk_size` for more context per chunk
|
||||
3. Structural content (tables, code) expands automatically
|
||||
|
||||
### "Search is slow"
|
||||
|
||||
1. Create a vector index: `haiku-rag create-index`
|
||||
2. Reduce `search.limit`
|
||||
3. Consider a smaller embedding model
|
||||
|
||||
### "QA answers are wrong despite good retrieval"
|
||||
|
||||
1. Check if chunks are being truncated by LLM context limits
|
||||
2. Try a more capable QA model
|
||||
3. Reduce number of chunks or expansion to fit context window
|
||||
|
||||
## Example Configurations
|
||||
|
||||
### High-Precision Technical Documentation
|
||||
Apply the result in your config:
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunk_size: 256
|
||||
chunker_type: hybrid
|
||||
|
||||
search:
|
||||
limit: 10
|
||||
context_radius: 1
|
||||
max_context_items: 15
|
||||
|
||||
reranking:
|
||||
model:
|
||||
provider: mxbai
|
||||
name: mixedbread-ai/mxbai-rerank-base-v2
|
||||
prompts:
|
||||
qa: |
|
||||
Your optimized prompt text here...
|
||||
```
|
||||
|
||||
### Long-Form Content (Articles, Reports)
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunk_size: 512
|
||||
chunker_type: hybrid
|
||||
|
||||
search:
|
||||
limit: 5
|
||||
context_radius: 2
|
||||
max_context_items: 10
|
||||
```
|
||||
|
||||
### FAQ/Knowledge Base
|
||||
|
||||
```yaml
|
||||
processing:
|
||||
chunk_size: 128
|
||||
chunker_type: hybrid
|
||||
|
||||
search:
|
||||
limit: 5
|
||||
context_radius: 0
|
||||
```
|
||||
Or programmatically: `get_qa_agent(client, config, system_prompt=optimized_prompt)`.
|
||||
|
|
|
|||
|
|
@ -6,31 +6,68 @@ This package is not published to PyPI and is only used for development and testi
|
|||
|
||||
## Overview
|
||||
|
||||
Contains evaluation scripts for benchmarking RAG performance using datasets like:
|
||||
Contains evaluation scripts for benchmarking RAG retrieval and QA performance, plus GEPA-based prompt optimization. Available datasets:
|
||||
|
||||
- RepliQA
|
||||
- WiX
|
||||
- HotpotQA
|
||||
- OpenRAG Bench
|
||||
|
||||
## Usage
|
||||
|
||||
After installing the package, you can run evaluations using the `evaluations` command:
|
||||
|
||||
```bash
|
||||
# Run evaluations with default settings
|
||||
evaluations repliqa
|
||||
# Run retrieval + QA benchmarks
|
||||
evaluations run repliqa
|
||||
evaluations run wix
|
||||
|
||||
# Use a custom config file
|
||||
evaluations repliqa --config /path/to/haiku.rag.yaml
|
||||
evaluations run repliqa --config /path/to/haiku.rag.yaml
|
||||
|
||||
# Override the database path
|
||||
evaluations repliqa --db /path/to/custom.lancedb
|
||||
evaluations run repliqa --db /path/to/custom.lancedb
|
||||
|
||||
# Skip database population and run only benchmarks
|
||||
evaluations repliqa --skip-db
|
||||
evaluations run repliqa --skip-db
|
||||
|
||||
# Skip specific benchmarks
|
||||
evaluations run repliqa --skip-retrieval
|
||||
evaluations run repliqa --skip-qa
|
||||
|
||||
# Limit the number of test cases
|
||||
evaluations repliqa --limit 100
|
||||
evaluations run repliqa --limit 100
|
||||
```
|
||||
|
||||
### Pre-built Databases
|
||||
|
||||
Download pre-built evaluation databases from HuggingFace:
|
||||
|
||||
```bash
|
||||
evaluations download repliqa
|
||||
evaluations download all
|
||||
evaluations download repliqa --force
|
||||
```
|
||||
|
||||
Upload databases (maintainer only):
|
||||
|
||||
```bash
|
||||
evaluations upload repliqa
|
||||
evaluations upload all
|
||||
```
|
||||
|
||||
### Prompt Optimization
|
||||
|
||||
Optimize QA system prompts using GEPA (Generalized Evolutionary Prompt Algorithm):
|
||||
|
||||
```bash
|
||||
evaluations optimize wix
|
||||
evaluations optimize repliqa --limit 40 --num-candidates 30
|
||||
evaluations optimize wix --output optimized_prompt.txt
|
||||
```
|
||||
|
||||
See [Tuning docs](https://ggozad.github.io/haiku.rag/tuning/#prompt-optimization-gepa) for details on applying results.
|
||||
|
||||
## Database Storage
|
||||
|
||||
By default, evaluation databases are stored in the haiku.rag data directory:
|
||||
|
|
|
|||
|
|
@ -28,6 +28,10 @@ load_dotenv(find_dotenv(usecwd=True))
|
|||
|
||||
HF_REPO_ID = "ggozad/haiku-rag-eval-dbs"
|
||||
|
||||
JUDGE_MODEL_CONFIG = ModelConfig(
|
||||
provider="ollama", name="gpt-oss", enable_thinking=False, temperature=0.0
|
||||
)
|
||||
|
||||
logfire.configure(send_to_logfire="if-token-present", service_name="evals")
|
||||
logfire.instrument_pydantic_ai()
|
||||
configure_cli_logging()
|
||||
|
|
@ -278,10 +282,7 @@ async def run_qa_benchmark(
|
|||
for index, doc in enumerate(corpus, start=1)
|
||||
]
|
||||
|
||||
judge_config = ModelConfig(
|
||||
provider="ollama", name="gpt-oss", enable_thinking=False, temperature=0.0
|
||||
)
|
||||
judge_model = get_model(judge_config, config)
|
||||
judge_model = get_model(JUDGE_MODEL_CONFIG, config)
|
||||
|
||||
evaluation_dataset = EvalDataset[str, str, dict[str, str]](
|
||||
name=spec.key,
|
||||
|
|
@ -302,7 +303,7 @@ async def run_qa_benchmark(
|
|||
|
||||
db = spec.db_path(db_path)
|
||||
async with HaikuRAG(db, config=config) as rag:
|
||||
qa = get_qa_agent(rag, system_prompt=spec.system_prompt)
|
||||
qa = get_qa_agent(rag, config, system_prompt=spec.resolve_system_prompt(config))
|
||||
|
||||
async def answer_question(question: str) -> str:
|
||||
answer, _ = await qa.answer(question)
|
||||
|
|
@ -314,7 +315,7 @@ async def run_qa_benchmark(
|
|||
dataset_key=spec.key,
|
||||
test_cases=len(cases),
|
||||
config=config,
|
||||
judge_config=judge_config,
|
||||
judge_config=JUDGE_MODEL_CONFIG,
|
||||
)
|
||||
|
||||
report = await evaluation_dataset.evaluate(
|
||||
|
|
@ -334,11 +335,10 @@ async def run_qa_benchmark(
|
|||
total_processed = len(report.cases)
|
||||
failures = report.failures
|
||||
|
||||
total_cases = total_processed
|
||||
accuracy = passing_cases / total_cases if total_cases > 0 else 0
|
||||
accuracy = passing_cases / total_processed if total_processed > 0 else 0
|
||||
|
||||
console.print("\n=== QA Benchmark Results ===", style="bold cyan")
|
||||
console.print(f"Total questions: {total_cases}")
|
||||
console.print(f"Total questions: {total_processed}")
|
||||
console.print(f"Correct answers: {passing_cases}")
|
||||
console.print(f"QA Accuracy: {accuracy:.4f} ({accuracy * 100:.2f}%)")
|
||||
|
||||
|
|
@ -390,6 +390,43 @@ async def evaluate_dataset(
|
|||
app = typer.Typer(help="Run retrieval and QA benchmarks for configured datasets.")
|
||||
|
||||
|
||||
def _load_config(config_path: Path | None) -> AppConfig:
|
||||
"""Load AppConfig from a file path or standard search path."""
|
||||
if config_path:
|
||||
if not config_path.exists():
|
||||
raise typer.BadParameter(f"Config file not found: {config_path}")
|
||||
console.print(f"Loading config from: {config_path}", style="dim")
|
||||
yaml_data = load_yaml_config(config_path)
|
||||
return AppConfig.model_validate(yaml_data)
|
||||
|
||||
found = find_config_file(None)
|
||||
if found:
|
||||
console.print(f"Loading config from: {found}", style="dim")
|
||||
yaml_data = load_yaml_config(found)
|
||||
return AppConfig.model_validate(yaml_data)
|
||||
|
||||
console.print("No config file found, using defaults", style="dim")
|
||||
return AppConfig()
|
||||
|
||||
|
||||
def _resolve_dataset(dataset: str) -> DatasetSpec:
|
||||
"""Resolve a dataset key to a DatasetSpec or raise BadParameter."""
|
||||
spec = DATASETS.get(dataset.lower())
|
||||
if spec is None:
|
||||
valid_datasets = ", ".join(sorted(DATASETS))
|
||||
raise typer.BadParameter(
|
||||
f"Unknown dataset '{dataset}'. Choose from: {valid_datasets}"
|
||||
)
|
||||
return spec
|
||||
|
||||
|
||||
def _resolve_datasets(dataset: str) -> list[DatasetSpec]:
|
||||
"""Resolve 'all' or a single dataset key to a list of DatasetSpecs."""
|
||||
if dataset.lower() == "all":
|
||||
return list(DATASETS.values())
|
||||
return [_resolve_dataset(dataset)]
|
||||
|
||||
|
||||
@app.command()
|
||||
def run(
|
||||
dataset: str = typer.Argument(..., help="Dataset key to evaluate."),
|
||||
|
|
@ -398,7 +435,7 @@ def run(
|
|||
),
|
||||
db: Path | None = typer.Option(None, "--db", help="Override the database path."),
|
||||
skip_db: bool = typer.Option(
|
||||
False, "--skip-db", help="Skip updateing the evaluation db."
|
||||
False, "--skip-db", help="Skip updating the evaluation db."
|
||||
),
|
||||
skip_retrieval: bool = typer.Option(
|
||||
False, "--skip-retrieval", help="Skip retrieval benchmark."
|
||||
|
|
@ -417,30 +454,8 @@ def run(
|
|||
help="Only evaluate queries requiring image understanding.",
|
||||
),
|
||||
) -> None:
|
||||
spec = DATASETS.get(dataset.lower())
|
||||
if spec is None:
|
||||
valid_datasets = ", ".join(sorted(DATASETS))
|
||||
raise typer.BadParameter(
|
||||
f"Unknown dataset '{dataset}'. Choose from: {valid_datasets}"
|
||||
)
|
||||
|
||||
# Load config from file or use defaults
|
||||
if config:
|
||||
if not config.exists():
|
||||
raise typer.BadParameter(f"Config file not found: {config}")
|
||||
console.print(f"Loading config from: {config}", style="dim")
|
||||
yaml_data = load_yaml_config(config)
|
||||
app_config = AppConfig.model_validate(yaml_data)
|
||||
else:
|
||||
# Try to find config file using standard search path
|
||||
config_path = find_config_file(None)
|
||||
if config_path:
|
||||
console.print(f"Loading config from: {config_path}", style="dim")
|
||||
yaml_data = load_yaml_config(config_path)
|
||||
app_config = AppConfig.model_validate(yaml_data)
|
||||
else:
|
||||
console.print("No config file found, using defaults", style="dim")
|
||||
app_config = AppConfig()
|
||||
spec = _resolve_dataset(dataset)
|
||||
app_config = _load_config(config)
|
||||
|
||||
asyncio.run(
|
||||
evaluate_dataset(
|
||||
|
|
@ -458,22 +473,55 @@ def run(
|
|||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def optimize(
|
||||
dataset: str = typer.Argument(..., help="Dataset key to optimize prompt for."),
|
||||
config: Path | None = typer.Option(
|
||||
None, "--config", help="Path to haiku.rag YAML config file."
|
||||
),
|
||||
db: Path | None = typer.Option(None, "--db", help="Override the database path."),
|
||||
limit: int | None = typer.Option(
|
||||
None, "--limit", help="Limit QA cases (split 50/50 into train/val)."
|
||||
),
|
||||
num_candidates: int = typer.Option(
|
||||
50, "--num-candidates", help="Number of candidate prompts to evaluate."
|
||||
),
|
||||
output: Path | None = typer.Option(
|
||||
None, "--output", help="Save optimized prompt to file."
|
||||
),
|
||||
) -> None:
|
||||
"""Optimize QA system prompt using GEPA evolutionary optimization."""
|
||||
from evaluations.optimization import run_optimization
|
||||
|
||||
spec = _resolve_dataset(dataset)
|
||||
app_config = _load_config(config)
|
||||
|
||||
corpus = spec.qa_loader()
|
||||
if limit is not None:
|
||||
corpus = corpus.select(range(min(limit, len(corpus))))
|
||||
|
||||
cases: list[Case[str, str, dict[str, str]]] = [
|
||||
spec.qa_case_builder(index, cast(Mapping[str, Any], doc))
|
||||
for index, doc in enumerate(corpus, start=1)
|
||||
]
|
||||
|
||||
run_optimization(
|
||||
spec=spec,
|
||||
config=app_config,
|
||||
cases=cases,
|
||||
num_candidates=num_candidates,
|
||||
db_path=db,
|
||||
output=output,
|
||||
)
|
||||
|
||||
|
||||
@app.command()
|
||||
def download(
|
||||
dataset: str = typer.Argument(..., help="Dataset key or 'all' to download all."),
|
||||
force: bool = typer.Option(False, "--force", help="Overwrite existing database."),
|
||||
) -> None:
|
||||
"""Download pre-built evaluation database from HuggingFace."""
|
||||
if dataset.lower() == "all":
|
||||
specs = list(DATASETS.values())
|
||||
else:
|
||||
spec = DATASETS.get(dataset.lower())
|
||||
if spec is None:
|
||||
valid_datasets = ", ".join(sorted(DATASETS))
|
||||
raise typer.BadParameter(
|
||||
f"Unknown dataset '{dataset}'. Choose from: {valid_datasets}, all"
|
||||
)
|
||||
specs = [spec]
|
||||
specs = _resolve_datasets(dataset)
|
||||
|
||||
for spec in specs:
|
||||
db = spec.db_path()
|
||||
|
|
@ -524,16 +572,7 @@ def upload(
|
|||
dataset: str = typer.Argument(..., help="Dataset key or 'all' to upload all."),
|
||||
) -> None:
|
||||
"""Upload evaluation database to HuggingFace (maintainer only)."""
|
||||
if dataset.lower() == "all":
|
||||
specs = list(DATASETS.values())
|
||||
else:
|
||||
spec = DATASETS.get(dataset.lower())
|
||||
if spec is None:
|
||||
valid_datasets = ", ".join(sorted(DATASETS))
|
||||
raise typer.BadParameter(
|
||||
f"Unknown dataset '{dataset}'. Choose from: {valid_datasets}, all"
|
||||
)
|
||||
specs = [spec]
|
||||
specs = _resolve_datasets(dataset)
|
||||
|
||||
api = HfApi()
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,8 @@ from datasets import Dataset
|
|||
from pydantic_evals import Case
|
||||
from pydantic_evals.evaluators import Evaluator
|
||||
|
||||
from haiku.rag.config.models import AppConfig
|
||||
|
||||
|
||||
@dataclass
|
||||
class DocumentPayload:
|
||||
|
|
@ -63,3 +65,11 @@ class DatasetSpec:
|
|||
|
||||
data_dir = get_default_data_dir()
|
||||
return data_dir / "evaluations" / "dbs" / self.db_filename
|
||||
|
||||
def resolve_system_prompt(self, config: AppConfig) -> str | None:
|
||||
"""Resolve the QA system prompt.
|
||||
|
||||
Precedence: config.prompts.qa > spec.system_prompt > None
|
||||
(get_qa_agent handles the final fallback to QA_SYSTEM_PROMPT)
|
||||
"""
|
||||
return config.prompts.qa or self.system_prompt
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@ from evaluations.config import DatasetSpec
|
|||
|
||||
from .hotpotqa import HOTPOTQA_SPEC
|
||||
from .open_rag_bench import OPEN_RAG_BENCH_SPEC
|
||||
from .repliqa import REPLIQ_SPEC
|
||||
from .repliqa import REPLIQA_SPEC
|
||||
from .wix import WIX_SPEC
|
||||
|
||||
DATASETS: dict[str, DatasetSpec] = {
|
||||
spec.key: spec
|
||||
for spec in (REPLIQ_SPEC, WIX_SPEC, HOTPOTQA_SPEC, OPEN_RAG_BENCH_SPEC)
|
||||
for spec in (REPLIQA_SPEC, WIX_SPEC, HOTPOTQA_SPEC, OPEN_RAG_BENCH_SPEC)
|
||||
}
|
||||
|
||||
__all__ = ["DATASETS"]
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ def build_repliqa_case(
|
|||
)
|
||||
|
||||
|
||||
REPLIQ_SPEC = DatasetSpec(
|
||||
REPLIQA_SPEC = DatasetSpec(
|
||||
key="repliqa",
|
||||
db_filename="repliqa.lancedb",
|
||||
document_loader=load_repliqa_corpus,
|
||||
|
|
|
|||
289
evaluations/evaluations/optimization.py
Normal file
289
evaluations/evaluations/optimization.py
Normal file
|
|
@ -0,0 +1,289 @@
|
|||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pydantic_ai.models import Model
|
||||
from pydantic_evals import Case
|
||||
from pydantic_evals.evaluators.llm_as_a_judge import judge_input_output_expected
|
||||
|
||||
from gepa.core.adapter import EvaluationBatch
|
||||
|
||||
from evaluations.benchmark import JUDGE_MODEL_CONFIG
|
||||
from evaluations.config import DatasetSpec
|
||||
from haiku.rag.agents.qa import QuestionAnswerAgent, get_qa_agent
|
||||
from haiku.rag.agents.qa.prompts import QA_SYSTEM_PROMPT
|
||||
from haiku.rag.client import HaikuRAG
|
||||
from haiku.rag.config.models import AppConfig, ModelConfig
|
||||
from haiku.rag.utils import get_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
OPTIMIZATION_SCORING_RUBRIC = """You are evaluating the quality of an answer to a question,
|
||||
comparing it against a reference answer.
|
||||
|
||||
Score on a scale of 0.0 to 1.0:
|
||||
- 1.0: The answer is factually correct, complete, and concise. It covers all key points
|
||||
from the reference answer without contradictions or significant omissions.
|
||||
- 0.7-0.9: The answer is mostly correct and addresses the core question, but may miss
|
||||
some secondary details or include minor inaccuracies.
|
||||
- 0.4-0.6: The answer is partially correct — it addresses some aspects of the question
|
||||
but misses key information or contains notable inaccuracies.
|
||||
- 0.1-0.3: The answer is mostly incorrect or fails to address the core question,
|
||||
though it may contain some tangentially relevant information.
|
||||
- 0.0: The answer is completely wrong, irrelevant, or empty.
|
||||
|
||||
GUIDELINES:
|
||||
- Focus on factual correctness relative to the reference answer
|
||||
- Ignore differences in phrasing, style, or formatting
|
||||
- A concise correct answer scores higher than a verbose partially correct one
|
||||
- "I cannot find enough information" when the reference has an answer scores 0.0
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class EvalTrajectory:
|
||||
"""Per-case evaluation result for GEPA reflection."""
|
||||
|
||||
question: str
|
||||
expected_answer: str
|
||||
actual_answer: str | None
|
||||
score: float
|
||||
judge_reason: str | None = None
|
||||
|
||||
|
||||
QACase = Case[str, str, dict[str, str]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class QAPromptAdapter:
|
||||
"""GEPA adapter that evaluates QA prompt candidates against a dataset.
|
||||
|
||||
Implements the GEPAAdapter protocol:
|
||||
- evaluate(): Run QA agent with candidate prompt, score with LLMJudge
|
||||
- make_reflective_dataset(): Build failure records for the GEPA proposer
|
||||
"""
|
||||
|
||||
config: AppConfig
|
||||
db_path: Path
|
||||
judge_model: Model
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
batch: list[QACase],
|
||||
candidate: dict[str, str],
|
||||
capture_traces: bool = False,
|
||||
) -> EvaluationBatch[EvalTrajectory, str | None]:
|
||||
instructions = candidate["instructions"]
|
||||
return asyncio.run(
|
||||
self._evaluate_with_setup(batch, instructions, capture_traces)
|
||||
)
|
||||
|
||||
async def _evaluate_with_setup(
|
||||
self,
|
||||
batch: list[QACase],
|
||||
instructions: str,
|
||||
capture_traces: bool,
|
||||
) -> EvaluationBatch[EvalTrajectory, str | None]:
|
||||
async with HaikuRAG(self.db_path, config=self.config) as rag:
|
||||
qa = get_qa_agent(rag, self.config, system_prompt=instructions)
|
||||
return await self._evaluate_async(batch, qa, capture_traces)
|
||||
|
||||
async def _evaluate_async(
|
||||
self,
|
||||
batch: list[QACase],
|
||||
qa: QuestionAnswerAgent,
|
||||
capture_traces: bool,
|
||||
) -> EvaluationBatch[EvalTrajectory, str | None]:
|
||||
outputs: list[str | None] = []
|
||||
scores: list[float] = []
|
||||
trajectories: list[EvalTrajectory] | None = [] if capture_traces else None
|
||||
|
||||
for case in batch:
|
||||
question = case.inputs
|
||||
expected = case.expected_output or ""
|
||||
|
||||
try:
|
||||
answer, _ = await qa.answer(question)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"QA agent failed for question: %s", question, exc_info=True
|
||||
)
|
||||
answer = None
|
||||
|
||||
if answer is not None:
|
||||
score, reason = await self._judge(question, answer, expected)
|
||||
else:
|
||||
score, reason = 0.0, "QA agent failed to produce an answer"
|
||||
|
||||
outputs.append(answer)
|
||||
scores.append(score)
|
||||
|
||||
if capture_traces and trajectories is not None:
|
||||
trajectories.append(
|
||||
EvalTrajectory(
|
||||
question=question,
|
||||
expected_answer=expected,
|
||||
actual_answer=answer,
|
||||
score=score,
|
||||
judge_reason=reason,
|
||||
)
|
||||
)
|
||||
|
||||
return EvaluationBatch(
|
||||
outputs=outputs,
|
||||
scores=scores,
|
||||
trajectories=trajectories,
|
||||
)
|
||||
|
||||
async def _judge(
|
||||
self, question: str, answer: str, expected: str
|
||||
) -> tuple[float, str | None]:
|
||||
"""Score an answer using pydantic-evals LLMJudge with float scoring."""
|
||||
result = await judge_input_output_expected(
|
||||
inputs=question,
|
||||
output=answer,
|
||||
expected_output=expected,
|
||||
rubric=OPTIMIZATION_SCORING_RUBRIC,
|
||||
model=self.judge_model,
|
||||
)
|
||||
return result.score, result.reason
|
||||
|
||||
def make_reflective_dataset(
|
||||
self,
|
||||
candidate: dict[str, str],
|
||||
eval_batch: EvaluationBatch[EvalTrajectory, str | None],
|
||||
components_to_update: list[str],
|
||||
) -> Mapping[str, Sequence[Mapping[str, Any]]]:
|
||||
if eval_batch.trajectories is None:
|
||||
return {}
|
||||
|
||||
records: list[dict[str, Any]] = []
|
||||
for traj in eval_batch.trajectories:
|
||||
records.append(
|
||||
{
|
||||
"Inputs": {"question": traj.question},
|
||||
"Generated Outputs": {
|
||||
"answer": traj.actual_answer or "(no answer)"
|
||||
},
|
||||
"Feedback": (
|
||||
f"Expected answer: {traj.expected_answer}\n"
|
||||
f"Score: {traj.score:.2f}\n"
|
||||
f"Judge reasoning: {traj.judge_reason or 'N/A'}"
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
return {"instructions": records}
|
||||
|
||||
propose_new_texts = None
|
||||
|
||||
|
||||
class ReflectionLM:
|
||||
"""LanguageModel implementation for GEPA's ReflectiveMutationProposer.
|
||||
|
||||
Wraps a pydantic-ai Agent to satisfy GEPA's LanguageModel protocol.
|
||||
"""
|
||||
|
||||
def __init__(self, model_config: ModelConfig, config: AppConfig) -> None:
|
||||
from pydantic_ai import Agent
|
||||
|
||||
model = get_model(model_config, config)
|
||||
self._agent: Agent[None, str] = Agent(model=model, output_type=str)
|
||||
|
||||
def __call__(self, prompt: str | list[dict[str, Any]]) -> str:
|
||||
if isinstance(prompt, list):
|
||||
text = "\n".join(
|
||||
f"{msg.get('role', 'user')}: {msg.get('content', '')}" for msg in prompt
|
||||
)
|
||||
else:
|
||||
text = prompt
|
||||
result = self._agent.run_sync(text)
|
||||
return result.output
|
||||
|
||||
|
||||
# Cases per GEPA reflection minibatch (used for budget calculation)
|
||||
REFLECTION_MINIBATCH_SIZE = 3
|
||||
|
||||
|
||||
def run_optimization(
|
||||
spec: DatasetSpec,
|
||||
config: AppConfig,
|
||||
cases: list[QACase],
|
||||
num_candidates: int,
|
||||
db_path: Path | None = None,
|
||||
output: Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Run GEPA optimization and return results summary."""
|
||||
from rich.console import Console
|
||||
|
||||
console = Console()
|
||||
|
||||
judge_model = get_model(JUDGE_MODEL_CONFIG, config)
|
||||
|
||||
db = spec.db_path(db_path)
|
||||
adapter = QAPromptAdapter(
|
||||
config=config,
|
||||
db_path=db,
|
||||
judge_model=judge_model,
|
||||
)
|
||||
|
||||
reflection_lm = ReflectionLM(config.qa.model, config)
|
||||
|
||||
seed_prompt = spec.resolve_system_prompt(config) or QA_SYSTEM_PROMPT
|
||||
seed_candidate = {"instructions": seed_prompt}
|
||||
|
||||
mid = len(cases) // 2
|
||||
trainset = cases[:mid]
|
||||
valset = cases[mid:]
|
||||
|
||||
# Budget: initial valset eval + per-candidate worst case
|
||||
# (each candidate: 2 minibatch evals + full valset if accepted)
|
||||
max_metric_calls = len(valset) + num_candidates * (
|
||||
2 * REFLECTION_MINIBATCH_SIZE + len(valset)
|
||||
)
|
||||
|
||||
console.print(f"Optimizing prompt for dataset: {spec.key}", style="bold magenta")
|
||||
console.print(
|
||||
f"Train: {len(trainset)}, Val: {len(valset)}, "
|
||||
f"Candidates: {num_candidates}, Budget: {max_metric_calls} eval calls"
|
||||
)
|
||||
console.print(f"Seed prompt length: {len(seed_prompt)} chars")
|
||||
|
||||
from gepa import optimize as gepa_optimize
|
||||
|
||||
result = gepa_optimize(
|
||||
seed_candidate=seed_candidate,
|
||||
trainset=trainset,
|
||||
valset=valset,
|
||||
adapter=adapter,
|
||||
reflection_lm=reflection_lm,
|
||||
max_metric_calls=max_metric_calls,
|
||||
display_progress_bar=True,
|
||||
)
|
||||
|
||||
best_score = result.val_aggregate_scores[result.best_idx]
|
||||
best_prompt = result.best_candidate
|
||||
if isinstance(best_prompt, dict):
|
||||
best_prompt = best_prompt["instructions"]
|
||||
total_calls = result.total_metric_calls or "unknown"
|
||||
|
||||
console.print("\n=== Optimization Results ===", style="bold cyan")
|
||||
console.print(f"Total metric calls: {total_calls}")
|
||||
console.print(f"Candidates explored: {result.num_candidates}")
|
||||
console.print(f"Best score: {best_score:.4f}")
|
||||
console.print(f"\nOptimized prompt:\n{best_prompt}")
|
||||
|
||||
if output:
|
||||
output.write_text(best_prompt)
|
||||
console.print(f"\nSaved to: {output}", style="green")
|
||||
|
||||
return {
|
||||
"best_score": best_score,
|
||||
"best_prompt": best_prompt,
|
||||
"total_calls": total_calls,
|
||||
"num_candidates": result.num_candidates,
|
||||
}
|
||||
|
|
@ -14,6 +14,7 @@ dependencies = [
|
|||
"huggingface_hub>=0.20.0",
|
||||
"typer>=0.21.0,<0.22.0",
|
||||
"python-dotenv>=1.2.2",
|
||||
"gepa>=0.1.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
|
@ -26,5 +27,9 @@ build-backend = "hatchling.build"
|
|||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["evaluations"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["tests"]
|
||||
|
||||
[tool.ruff.lint.isort]
|
||||
known-first-party = ["haiku", "evaluations"]
|
||||
|
|
|
|||
0
evaluations/tests/__init__.py
Normal file
0
evaluations/tests/__init__.py
Normal file
112
evaluations/tests/test_benchmark.py
Normal file
112
evaluations/tests/test_benchmark.py
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
import typer
|
||||
|
||||
from evaluations.benchmark import (
|
||||
_load_config,
|
||||
_resolve_dataset,
|
||||
build_experiment_metadata,
|
||||
)
|
||||
from haiku.rag.config.models import AppConfig, ModelConfig
|
||||
|
||||
|
||||
class TestBuildExperimentMetadata:
|
||||
def test_basic_metadata(self) -> None:
|
||||
config = AppConfig()
|
||||
result = build_experiment_metadata(
|
||||
dataset_key="test",
|
||||
test_cases=42,
|
||||
config=config,
|
||||
)
|
||||
|
||||
assert result["dataset"] == "test"
|
||||
assert result["test_cases"] == 42
|
||||
assert result["embedder_provider"] == config.embeddings.model.provider
|
||||
assert result["embedder_model"] == config.embeddings.model.name
|
||||
assert result["embedder_dim"] == config.embeddings.model.vector_dim
|
||||
assert result["chunk_size"] == config.processing.chunk_size
|
||||
assert result["search_limit"] == config.search.limit
|
||||
assert result["context_radius"] == config.search.context_radius
|
||||
assert result["qa_provider"] == config.qa.model.provider
|
||||
assert result["qa_model"] == config.qa.model.name
|
||||
assert "judge_provider" not in result
|
||||
|
||||
def test_with_judge_config(self) -> None:
|
||||
config = AppConfig()
|
||||
judge = ModelConfig(
|
||||
provider="ollama", name="gpt-oss", enable_thinking=False, temperature=0.0
|
||||
)
|
||||
result = build_experiment_metadata(
|
||||
dataset_key="test",
|
||||
test_cases=10,
|
||||
config=config,
|
||||
judge_config=judge,
|
||||
)
|
||||
|
||||
assert result["judge_provider"] == "ollama"
|
||||
assert result["judge_model"] == "gpt-oss"
|
||||
assert result["judge_temperature"] == 0.0
|
||||
assert result["judge_enable_thinking"] is False
|
||||
|
||||
def test_no_reranker(self) -> None:
|
||||
config = AppConfig()
|
||||
result = build_experiment_metadata(
|
||||
dataset_key="test", test_cases=1, config=config
|
||||
)
|
||||
assert result["rerank_provider"] is None
|
||||
assert result["rerank_model"] is None
|
||||
|
||||
def test_with_reranker(self) -> None:
|
||||
config = AppConfig()
|
||||
config.reranking.model = ModelConfig(
|
||||
provider="mxbai", name="mixedbread-ai/mxbai-rerank-base-v2"
|
||||
)
|
||||
result = build_experiment_metadata(
|
||||
dataset_key="test", test_cases=1, config=config
|
||||
)
|
||||
assert result["rerank_provider"] == "mxbai"
|
||||
assert result["rerank_model"] == "mixedbread-ai/mxbai-rerank-base-v2"
|
||||
|
||||
|
||||
class TestResolveDataset:
|
||||
def test_valid_dataset(self) -> None:
|
||||
spec = _resolve_dataset("repliqa")
|
||||
assert spec.key == "repliqa"
|
||||
|
||||
def test_case_insensitive(self) -> None:
|
||||
spec = _resolve_dataset("REPLIQA")
|
||||
assert spec.key == "repliqa"
|
||||
|
||||
def test_unknown_dataset_raises(self) -> None:
|
||||
with pytest.raises(typer.BadParameter, match="Unknown dataset 'nonexistent'"):
|
||||
_resolve_dataset("nonexistent")
|
||||
|
||||
def test_error_lists_valid_datasets(self) -> None:
|
||||
with pytest.raises(typer.BadParameter, match="repliqa"):
|
||||
_resolve_dataset("nonexistent")
|
||||
|
||||
|
||||
class TestLoadConfig:
|
||||
def test_explicit_path(self, tmp_path: Path) -> None:
|
||||
config_file = tmp_path / "test.yaml"
|
||||
config_file.write_text("search:\n limit: 42\n")
|
||||
config = _load_config(config_file)
|
||||
assert config.search.limit == 42
|
||||
|
||||
def test_explicit_path_not_found(self, tmp_path: Path) -> None:
|
||||
with pytest.raises(typer.BadParameter, match="Config file not found"):
|
||||
_load_config(tmp_path / "nonexistent.yaml")
|
||||
|
||||
def test_none_falls_back_to_find_config(self, tmp_path: Path) -> None:
|
||||
config_file = tmp_path / "haiku.rag.yaml"
|
||||
config_file.write_text("search:\n limit: 99\n")
|
||||
with patch("evaluations.benchmark.find_config_file", return_value=config_file):
|
||||
config = _load_config(None)
|
||||
assert config.search.limit == 99
|
||||
|
||||
def test_none_no_config_uses_defaults(self) -> None:
|
||||
with patch("evaluations.benchmark.find_config_file", return_value=None):
|
||||
config = _load_config(None)
|
||||
assert config == AppConfig()
|
||||
113
evaluations/tests/test_config.py
Normal file
113
evaluations/tests/test_config.py
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from evaluations.config import DatasetSpec, DocumentPayload, RetrievalSample
|
||||
from haiku.rag.config.models import AppConfig
|
||||
|
||||
|
||||
def _make_spec(**kwargs: object) -> DatasetSpec:
|
||||
defaults: dict[str, object] = {
|
||||
"key": "test",
|
||||
"db_filename": "test.lancedb",
|
||||
"document_loader": lambda: None,
|
||||
"document_mapper": lambda doc: None,
|
||||
"qa_loader": lambda: None,
|
||||
"qa_case_builder": lambda idx, doc: None,
|
||||
}
|
||||
defaults.update(kwargs)
|
||||
return DatasetSpec(**defaults) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestDatasetSpecDbPath:
|
||||
def test_override_path_takes_precedence(self) -> None:
|
||||
spec = _make_spec()
|
||||
override = Path("/tmp/custom.lancedb")
|
||||
assert spec.db_path(override) == override
|
||||
|
||||
def test_default_uses_data_dir(self) -> None:
|
||||
spec = _make_spec(db_filename="mydb.lancedb")
|
||||
with patch(
|
||||
"haiku.rag.utils.get_default_data_dir",
|
||||
return_value=Path("/home/user/.local/share/haiku.rag"),
|
||||
):
|
||||
result = spec.db_path()
|
||||
assert result == Path(
|
||||
"/home/user/.local/share/haiku.rag/evaluations/dbs/mydb.lancedb"
|
||||
)
|
||||
|
||||
def test_none_override_uses_default(self) -> None:
|
||||
spec = _make_spec(db_filename="other.lancedb")
|
||||
with patch(
|
||||
"haiku.rag.utils.get_default_data_dir",
|
||||
return_value=Path("/data"),
|
||||
):
|
||||
result = spec.db_path(None)
|
||||
assert result == Path("/data/evaluations/dbs/other.lancedb")
|
||||
|
||||
|
||||
class TestDatasetSpecDefaults:
|
||||
def test_optional_fields_default_to_none(self) -> None:
|
||||
spec = _make_spec()
|
||||
assert spec.retrieval_loader is None
|
||||
assert spec.retrieval_mapper is None
|
||||
assert spec.retrieval_evaluator is None
|
||||
assert spec.document_limit is None
|
||||
assert spec.system_prompt is None
|
||||
|
||||
|
||||
class TestResolveSystemPrompt:
|
||||
def test_config_prompt_overrides_spec_prompt(self) -> None:
|
||||
spec = _make_spec(system_prompt="spec prompt")
|
||||
config = AppConfig()
|
||||
config.prompts.qa = "config prompt"
|
||||
assert spec.resolve_system_prompt(config) == "config prompt"
|
||||
|
||||
def test_spec_prompt_used_when_config_unset(self) -> None:
|
||||
spec = _make_spec(system_prompt="spec prompt")
|
||||
config = AppConfig()
|
||||
assert spec.resolve_system_prompt(config) == "spec prompt"
|
||||
|
||||
def test_returns_none_when_both_unset(self) -> None:
|
||||
spec = _make_spec()
|
||||
config = AppConfig()
|
||||
assert spec.resolve_system_prompt(config) is None
|
||||
|
||||
|
||||
class TestDocumentPayload:
|
||||
def test_defaults(self) -> None:
|
||||
payload = DocumentPayload(uri="test://doc")
|
||||
assert payload.content is None
|
||||
assert payload.title is None
|
||||
assert payload.metadata is None
|
||||
assert payload.format == "md"
|
||||
assert payload.source_path is None
|
||||
|
||||
def test_all_fields(self) -> None:
|
||||
payload = DocumentPayload(
|
||||
uri="test://doc",
|
||||
content="hello",
|
||||
title="Title",
|
||||
metadata={"k": "v"},
|
||||
format="html",
|
||||
source_path=Path("/tmp/doc.pdf"),
|
||||
)
|
||||
assert payload.uri == "test://doc"
|
||||
assert payload.content == "hello"
|
||||
assert payload.source_path == Path("/tmp/doc.pdf")
|
||||
|
||||
|
||||
class TestRetrievalSample:
|
||||
def test_defaults(self) -> None:
|
||||
sample = RetrievalSample(question="q?", expected_uris=("u1",))
|
||||
assert sample.skip is False
|
||||
assert sample.source_type is None
|
||||
|
||||
def test_all_fields(self) -> None:
|
||||
sample = RetrievalSample(
|
||||
question="q?",
|
||||
expected_uris=("u1", "u2"),
|
||||
skip=True,
|
||||
source_type="image",
|
||||
)
|
||||
assert sample.skip is True
|
||||
assert sample.source_type == "image"
|
||||
287
evaluations/tests/test_datasets.py
Normal file
287
evaluations/tests/test_datasets.py
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
from pathlib import Path
|
||||
|
||||
from evaluations.datasets.hotpotqa import (
|
||||
build_hotpotqa_case,
|
||||
extract_unique_documents,
|
||||
map_hotpotqa_document,
|
||||
map_hotpotqa_retrieval,
|
||||
)
|
||||
from evaluations.datasets.open_rag_bench import (
|
||||
build_orb_case,
|
||||
download_pdf,
|
||||
is_multimodal_query,
|
||||
map_orb_document,
|
||||
map_orb_retrieval,
|
||||
)
|
||||
from evaluations.datasets.repliqa import (
|
||||
build_repliqa_case,
|
||||
map_repliqa_document,
|
||||
map_repliqa_retrieval,
|
||||
)
|
||||
from evaluations.datasets.wix import (
|
||||
build_wix_case,
|
||||
map_wix_document,
|
||||
map_wix_retrieval,
|
||||
)
|
||||
|
||||
|
||||
class TestRepliqa:
|
||||
def test_map_document(self) -> None:
|
||||
doc = {"document_id": "doc-42", "document_extracted": "Some content here."}
|
||||
payload = map_repliqa_document(doc)
|
||||
assert payload.uri == "doc-42"
|
||||
assert payload.content == "Some content here."
|
||||
|
||||
def test_map_retrieval(self) -> None:
|
||||
doc = {
|
||||
"question": "What happened?",
|
||||
"answer": "Something happened.",
|
||||
"document_id": "doc-42",
|
||||
}
|
||||
sample = map_repliqa_retrieval(doc)
|
||||
assert sample is not None
|
||||
assert sample.question == "What happened?"
|
||||
assert sample.expected_uris == ("doc-42",)
|
||||
|
||||
def test_map_retrieval_skips_unanswerable(self) -> None:
|
||||
doc = {
|
||||
"question": "What?",
|
||||
"answer": "The answer is not found in the document.",
|
||||
"document_id": "doc-1",
|
||||
}
|
||||
assert map_repliqa_retrieval(doc) is None
|
||||
|
||||
def test_build_case(self) -> None:
|
||||
doc = {
|
||||
"document_id": "doc-7",
|
||||
"question": "Why?",
|
||||
"answer": "Because.",
|
||||
}
|
||||
case = build_repliqa_case(3, doc)
|
||||
assert case.name == "3_doc-7"
|
||||
assert case.inputs == "Why?"
|
||||
assert case.expected_output == "Because."
|
||||
assert case.metadata == {"document_id": "doc-7", "case_index": "3"}
|
||||
|
||||
def test_build_case_none_document_id(self) -> None:
|
||||
doc = {"document_id": None, "question": "Q?", "answer": "A."}
|
||||
case = build_repliqa_case(1, doc)
|
||||
assert case.name == "case_1"
|
||||
|
||||
|
||||
class TestWix:
|
||||
def test_map_document_with_all_fields(self) -> None:
|
||||
doc = {
|
||||
"id": 123,
|
||||
"url": "https://wix.com/article",
|
||||
"html_content": "<p>Content</p>",
|
||||
"title": "My Article",
|
||||
}
|
||||
payload = map_wix_document(doc)
|
||||
assert payload.uri == "123"
|
||||
assert payload.content == "<p>Content</p>"
|
||||
assert payload.title == "My Article"
|
||||
assert payload.format == "html"
|
||||
assert payload.metadata == {
|
||||
"article_id": "123",
|
||||
"url": "https://wix.com/article",
|
||||
}
|
||||
|
||||
def test_map_document_no_id(self) -> None:
|
||||
doc = {
|
||||
"id": None,
|
||||
"url": "https://wix.com/page",
|
||||
"html_content": "<p>Text</p>",
|
||||
"title": None,
|
||||
}
|
||||
payload = map_wix_document(doc)
|
||||
assert payload.uri == "https://wix.com/page"
|
||||
|
||||
def test_map_document_no_metadata(self) -> None:
|
||||
doc = {"id": None, "url": None, "html_content": "<p>X</p>", "title": None}
|
||||
payload = map_wix_document(doc)
|
||||
assert payload.metadata is None
|
||||
|
||||
def test_map_retrieval(self) -> None:
|
||||
doc = {"question": "How to add a page?", "article_ids": [10, 20]}
|
||||
sample = map_wix_retrieval(doc)
|
||||
assert sample is not None
|
||||
assert sample.question == "How to add a page?"
|
||||
assert sample.expected_uris == ("10", "20")
|
||||
|
||||
def test_map_retrieval_no_article_ids(self) -> None:
|
||||
doc = {"question": "Q?", "article_ids": None}
|
||||
assert map_wix_retrieval(doc) is None
|
||||
|
||||
def test_map_retrieval_empty_article_ids(self) -> None:
|
||||
doc = {"question": "Q?", "article_ids": []}
|
||||
assert map_wix_retrieval(doc) is None
|
||||
|
||||
def test_build_case(self) -> None:
|
||||
doc = {
|
||||
"question": "How?",
|
||||
"answer": "Like this.",
|
||||
"article_ids": [5, 10],
|
||||
}
|
||||
case = build_wix_case(2, doc)
|
||||
assert case.name == "2_5-10"
|
||||
assert case.inputs == "How?"
|
||||
assert case.expected_output == "Like this."
|
||||
assert case.metadata is not None
|
||||
assert case.metadata["case_index"] == "2"
|
||||
|
||||
def test_build_case_no_article_ids(self) -> None:
|
||||
doc = {"question": "Q?", "answer": "A.", "article_ids": None}
|
||||
case = build_wix_case(1, doc)
|
||||
assert case.name == "case_1"
|
||||
|
||||
|
||||
class TestHotpotQA:
|
||||
def test_map_document(self) -> None:
|
||||
doc = {"title": "Albert Einstein", "content": "Was a physicist."}
|
||||
payload = map_hotpotqa_document(doc)
|
||||
assert payload.uri == "Albert Einstein"
|
||||
assert payload.content == "Was a physicist."
|
||||
assert payload.title == "Albert Einstein"
|
||||
|
||||
def test_map_retrieval(self) -> None:
|
||||
doc = {
|
||||
"question": "Who was Einstein?",
|
||||
"supporting_facts": {"title": ["Albert Einstein", "Physics"]},
|
||||
}
|
||||
sample = map_hotpotqa_retrieval(doc)
|
||||
assert sample is not None
|
||||
assert sample.expected_uris == ("Albert Einstein", "Physics")
|
||||
|
||||
def test_map_retrieval_deduplicates_titles(self) -> None:
|
||||
doc = {
|
||||
"question": "Q?",
|
||||
"supporting_facts": {"title": ["A", "B", "A"]},
|
||||
}
|
||||
sample = map_hotpotqa_retrieval(doc)
|
||||
assert sample is not None
|
||||
assert sample.expected_uris == ("A", "B")
|
||||
|
||||
def test_map_retrieval_no_titles(self) -> None:
|
||||
doc = {"question": "Q?", "supporting_facts": {"title": []}}
|
||||
assert map_hotpotqa_retrieval(doc) is None
|
||||
|
||||
def test_build_case(self) -> None:
|
||||
doc = {
|
||||
"id": "abc123",
|
||||
"question": "What is X?",
|
||||
"answer": "X is Y.",
|
||||
"type": "comparison",
|
||||
"level": "hard",
|
||||
}
|
||||
case = build_hotpotqa_case(5, doc)
|
||||
assert case.name == "5_abc123"
|
||||
assert case.inputs == "What is X?"
|
||||
assert case.expected_output == "X is Y."
|
||||
assert case.metadata == {
|
||||
"question_id": "abc123",
|
||||
"type": "comparison",
|
||||
"level": "hard",
|
||||
"case_index": "5",
|
||||
}
|
||||
|
||||
def test_extract_unique_documents(self) -> None:
|
||||
# Simulate a minimal dataset with context
|
||||
dataset = [
|
||||
{
|
||||
"context": {
|
||||
"title": ["Doc A", "Doc B"],
|
||||
"sentences": [["Sentence 1."], ["Sentence 2.", " More."]],
|
||||
}
|
||||
},
|
||||
{
|
||||
"context": {
|
||||
"title": ["Doc A", "Doc C"],
|
||||
"sentences": [["Dupe."], ["Sentence 3."]],
|
||||
}
|
||||
},
|
||||
]
|
||||
docs = extract_unique_documents(dataset) # type: ignore[arg-type]
|
||||
assert len(docs) == 3
|
||||
titles = [d["title"] for d in docs]
|
||||
assert titles == ["Doc A", "Doc B", "Doc C"]
|
||||
assert docs[1]["content"] == "Sentence 2. More."
|
||||
|
||||
|
||||
class TestOpenRAGBench:
|
||||
def test_map_document(self, tmp_path: Path) -> None:
|
||||
# Pre-create a cached PDF
|
||||
cache_dir = tmp_path / "pdfs"
|
||||
cache_dir.mkdir()
|
||||
pdf_path = cache_dir / "paper1.pdf"
|
||||
pdf_path.write_bytes(b"%PDF-fake")
|
||||
|
||||
doc = {"paper_id": "paper1", "pdf_url": "https://example.com/paper1.pdf"}
|
||||
# Patch get_cache_dir to use our tmp_path
|
||||
from unittest.mock import patch
|
||||
|
||||
with patch(
|
||||
"evaluations.datasets.open_rag_bench.get_cache_dir", return_value=cache_dir
|
||||
):
|
||||
payload = map_orb_document(doc)
|
||||
|
||||
assert payload is not None
|
||||
assert payload.uri == "paper1"
|
||||
assert payload.title == "paper1"
|
||||
assert payload.source_path == pdf_path
|
||||
assert payload.metadata == {"arxiv_id": "paper1"}
|
||||
|
||||
def test_map_document_download_fails(self, tmp_path: Path) -> None:
|
||||
cache_dir = tmp_path / "pdfs"
|
||||
cache_dir.mkdir()
|
||||
|
||||
doc = {"paper_id": "missing", "pdf_url": "https://example.com/missing.pdf"}
|
||||
from unittest.mock import patch
|
||||
|
||||
with patch(
|
||||
"evaluations.datasets.open_rag_bench.get_cache_dir", return_value=cache_dir
|
||||
):
|
||||
with patch(
|
||||
"evaluations.datasets.open_rag_bench.download_pdf", return_value=None
|
||||
):
|
||||
payload = map_orb_document(doc)
|
||||
|
||||
assert payload is None
|
||||
|
||||
def test_map_retrieval(self) -> None:
|
||||
doc = {
|
||||
"query": "What is attention?",
|
||||
"doc_id": "1706.03762",
|
||||
"source": "text",
|
||||
}
|
||||
sample = map_orb_retrieval(doc)
|
||||
assert sample is not None
|
||||
assert sample.question == "What is attention?"
|
||||
assert sample.expected_uris == ("1706.03762",)
|
||||
assert sample.source_type == "text"
|
||||
|
||||
def test_build_case(self) -> None:
|
||||
doc = {
|
||||
"query_id": "q_abcdef12",
|
||||
"query": "Explain transformers.",
|
||||
"answer": "Transformers are...",
|
||||
"type": "factual",
|
||||
"source": "text",
|
||||
}
|
||||
case = build_orb_case(1, doc)
|
||||
assert case.name == "1_q_abcdef"
|
||||
assert case.inputs == "Explain transformers."
|
||||
assert case.expected_output == "Transformers are..."
|
||||
assert case.metadata is not None
|
||||
assert case.metadata["query_id"] == "q_abcdef12"
|
||||
|
||||
def test_download_pdf_uses_cache(self, tmp_path: Path) -> None:
|
||||
pdf_path = tmp_path / "cached.pdf"
|
||||
pdf_path.write_bytes(b"%PDF-cached")
|
||||
result = download_pdf("cached", "https://example.com/cached.pdf", tmp_path)
|
||||
assert result == pdf_path
|
||||
|
||||
def test_is_multimodal_query(self) -> None:
|
||||
assert is_multimodal_query("image") is True
|
||||
assert is_multimodal_query("image_table") is True
|
||||
assert is_multimodal_query("text") is False
|
||||
104
evaluations/tests/test_evaluators.py
Normal file
104
evaluations/tests/test_evaluators.py
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from evaluations.evaluators.map import MAPEvaluator
|
||||
from evaluations.evaluators.mrr import MRREvaluator
|
||||
|
||||
|
||||
class TestMRREvaluator:
|
||||
def setup_method(self) -> None:
|
||||
self.evaluator = MRREvaluator()
|
||||
|
||||
def _make_ctx(
|
||||
self, relevant_uris: list[str], retrieved_uris: list[str]
|
||||
) -> MagicMock:
|
||||
ctx = MagicMock()
|
||||
ctx.metadata = {"relevant_uris": relevant_uris}
|
||||
ctx.output = retrieved_uris
|
||||
return ctx
|
||||
|
||||
def test_first_result_relevant(self) -> None:
|
||||
ctx = self._make_ctx(["doc1"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 1.0
|
||||
|
||||
def test_second_result_relevant(self) -> None:
|
||||
ctx = self._make_ctx(["doc2"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 0.5
|
||||
|
||||
def test_third_result_relevant(self) -> None:
|
||||
ctx = self._make_ctx(["doc3"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == pytest.approx(1 / 3)
|
||||
|
||||
def test_no_relevant_found(self) -> None:
|
||||
ctx = self._make_ctx(["doc_x"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
|
||||
def test_empty_retrieved(self) -> None:
|
||||
ctx = self._make_ctx(["doc1"], [])
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
|
||||
def test_multiple_relevant_returns_first_match(self) -> None:
|
||||
ctx = self._make_ctx(["doc2", "doc3"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 0.5
|
||||
|
||||
def test_none_metadata(self) -> None:
|
||||
ctx = MagicMock()
|
||||
ctx.metadata = None
|
||||
ctx.output = ["doc1"]
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
|
||||
def test_empty_relevant_uris(self) -> None:
|
||||
ctx = self._make_ctx([], ["doc1", "doc2"])
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
|
||||
|
||||
class TestMAPEvaluator:
|
||||
def setup_method(self) -> None:
|
||||
self.evaluator = MAPEvaluator()
|
||||
|
||||
def _make_ctx(
|
||||
self, relevant_uris: list[str], retrieved_uris: list[str]
|
||||
) -> MagicMock:
|
||||
ctx = MagicMock()
|
||||
ctx.metadata = {"relevant_uris": relevant_uris}
|
||||
ctx.output = retrieved_uris
|
||||
return ctx
|
||||
|
||||
def test_perfect_single_doc(self) -> None:
|
||||
ctx = self._make_ctx(["doc1"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 1.0
|
||||
|
||||
def test_perfect_two_docs(self) -> None:
|
||||
# Both relevant at positions 1 and 2: P@1=1/1, P@2=2/2 → AP = (1+1)/2 = 1.0
|
||||
ctx = self._make_ctx(["doc1", "doc2"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 1.0
|
||||
|
||||
def test_one_relevant_at_second_position(self) -> None:
|
||||
# 1 relevant doc at position 2: P@2=1/2 → AP = 0.5/1 = 0.5
|
||||
ctx = self._make_ctx(["doc2"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 0.5
|
||||
|
||||
def test_two_relevant_with_gap(self) -> None:
|
||||
# Relevant at positions 1 and 3: P@1=1/1, P@3=2/3 → AP = (1 + 2/3)/2
|
||||
ctx = self._make_ctx(["doc1", "doc3"], ["doc1", "doc2", "doc3"])
|
||||
expected = (1.0 + 2 / 3) / 2
|
||||
assert self.evaluator.evaluate(ctx) == pytest.approx(expected)
|
||||
|
||||
def test_no_relevant_found(self) -> None:
|
||||
ctx = self._make_ctx(["doc_x"], ["doc1", "doc2", "doc3"])
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
|
||||
def test_empty_retrieved(self) -> None:
|
||||
ctx = self._make_ctx(["doc1"], [])
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
|
||||
def test_none_metadata(self) -> None:
|
||||
ctx = MagicMock()
|
||||
ctx.metadata = None
|
||||
ctx.output = ["doc1"]
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
|
||||
def test_empty_relevant_uris(self) -> None:
|
||||
ctx = self._make_ctx([], ["doc1", "doc2"])
|
||||
assert self.evaluator.evaluate(ctx) == 0.0
|
||||
415
evaluations/tests/test_optimization.py
Normal file
415
evaluations/tests/test_optimization.py
Normal file
|
|
@ -0,0 +1,415 @@
|
|||
from pathlib import Path
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from pydantic_ai.models.test import TestModel
|
||||
from pydantic_evals import Case
|
||||
|
||||
from gepa.core.adapter import EvaluationBatch
|
||||
|
||||
from evaluations.config import DatasetSpec
|
||||
from evaluations.optimization import (
|
||||
EvalTrajectory,
|
||||
QAPromptAdapter,
|
||||
ReflectionLM,
|
||||
run_optimization,
|
||||
)
|
||||
from haiku.rag.config.models import AppConfig
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_cases() -> list[Case[str, str, dict[str, str]]]:
|
||||
return [
|
||||
Case(
|
||||
name="q1",
|
||||
inputs="What is X?",
|
||||
expected_output="X is a thing.",
|
||||
metadata={"case_index": "1"},
|
||||
),
|
||||
Case(
|
||||
name="q2",
|
||||
inputs="How does Y work?",
|
||||
expected_output="Y works by Z.",
|
||||
metadata={"case_index": "2"},
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def adapter(tmp_path: Path) -> QAPromptAdapter:
|
||||
return QAPromptAdapter(
|
||||
config=AppConfig(),
|
||||
db_path=tmp_path / "test.lancedb",
|
||||
judge_model=MagicMock(),
|
||||
)
|
||||
|
||||
|
||||
class TestMakeReflectiveDataset:
|
||||
def test_builds_records_from_trajectories(self, adapter: QAPromptAdapter) -> None:
|
||||
trajectories = [
|
||||
EvalTrajectory(
|
||||
question="What is X?",
|
||||
expected_answer="X is a thing.",
|
||||
actual_answer="X is wrong.",
|
||||
score=0.2,
|
||||
judge_reason="Factually incorrect",
|
||||
),
|
||||
EvalTrajectory(
|
||||
question="How does Y?",
|
||||
expected_answer="Y works by Z.",
|
||||
actual_answer="Y works by Z.",
|
||||
score=1.0,
|
||||
judge_reason=None,
|
||||
),
|
||||
]
|
||||
eval_batch: EvaluationBatch[EvalTrajectory, str | None] = EvaluationBatch(
|
||||
outputs=["X is wrong.", "Y works by Z."],
|
||||
scores=[0.2, 1.0],
|
||||
trajectories=trajectories,
|
||||
)
|
||||
|
||||
result = adapter.make_reflective_dataset(
|
||||
{"instructions": "test"}, eval_batch, ["instructions"]
|
||||
)
|
||||
|
||||
assert "instructions" in result
|
||||
records = result["instructions"]
|
||||
assert len(records) == 2
|
||||
|
||||
assert records[0]["Inputs"]["question"] == "What is X?"
|
||||
assert records[0]["Generated Outputs"]["answer"] == "X is wrong."
|
||||
assert "Expected answer: X is a thing." in records[0]["Feedback"]
|
||||
assert "Score: 0.20" in records[0]["Feedback"]
|
||||
assert "Factually incorrect" in records[0]["Feedback"]
|
||||
|
||||
assert records[1]["Inputs"]["question"] == "How does Y?"
|
||||
assert records[1]["Generated Outputs"]["answer"] == "Y works by Z."
|
||||
assert "Score: 1.00" in records[1]["Feedback"]
|
||||
assert "N/A" in records[1]["Feedback"]
|
||||
|
||||
def test_returns_empty_when_no_trajectories(self, adapter: QAPromptAdapter) -> None:
|
||||
eval_batch: EvaluationBatch[EvalTrajectory, str | None] = EvaluationBatch(
|
||||
outputs=[], scores=[], trajectories=None
|
||||
)
|
||||
|
||||
result = adapter.make_reflective_dataset(
|
||||
{"instructions": "test"}, eval_batch, ["instructions"]
|
||||
)
|
||||
|
||||
assert result == {}
|
||||
|
||||
def test_none_answer_becomes_no_answer(self, adapter: QAPromptAdapter) -> None:
|
||||
trajectories = [
|
||||
EvalTrajectory(
|
||||
question="What?",
|
||||
expected_answer="Answer.",
|
||||
actual_answer=None,
|
||||
score=0.0,
|
||||
judge_reason="Failed",
|
||||
),
|
||||
]
|
||||
eval_batch: EvaluationBatch[EvalTrajectory, str | None] = EvaluationBatch(
|
||||
outputs=[None],
|
||||
scores=[0.0],
|
||||
trajectories=trajectories,
|
||||
)
|
||||
|
||||
result = adapter.make_reflective_dataset(
|
||||
{"instructions": "test"}, eval_batch, ["instructions"]
|
||||
)
|
||||
|
||||
assert result["instructions"][0]["Generated Outputs"]["answer"] == "(no answer)"
|
||||
|
||||
|
||||
class TestEvaluateAsync:
|
||||
@pytest.mark.asyncio
|
||||
async def test_returns_scores_and_outputs(
|
||||
self,
|
||||
adapter: QAPromptAdapter,
|
||||
sample_cases: list[Case[str, str, dict[str, str]]],
|
||||
) -> None:
|
||||
stub_qa = AsyncMock()
|
||||
stub_qa.answer = AsyncMock(return_value=("X is a thing.", []))
|
||||
adapter._judge = AsyncMock(return_value=(0.85, "Good answer")) # type: ignore[method-assign]
|
||||
|
||||
result = await adapter._evaluate_async(
|
||||
sample_cases, stub_qa, capture_traces=False
|
||||
)
|
||||
|
||||
assert len(result.outputs) == 2
|
||||
assert len(result.scores) == 2
|
||||
assert all(o == "X is a thing." for o in result.outputs)
|
||||
assert all(s == 0.85 for s in result.scores)
|
||||
assert result.trajectories is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_populates_trajectories_when_captured(
|
||||
self,
|
||||
adapter: QAPromptAdapter,
|
||||
sample_cases: list[Case[str, str, dict[str, str]]],
|
||||
) -> None:
|
||||
stub_qa = AsyncMock()
|
||||
stub_qa.answer = AsyncMock(return_value=("An answer.", []))
|
||||
adapter._judge = AsyncMock(return_value=(0.9, "Almost perfect")) # type: ignore[method-assign]
|
||||
|
||||
result = await adapter._evaluate_async(
|
||||
sample_cases, stub_qa, capture_traces=True
|
||||
)
|
||||
|
||||
assert result.trajectories is not None
|
||||
assert len(result.trajectories) == 2
|
||||
|
||||
traj = result.trajectories[0]
|
||||
assert traj.question == "What is X?"
|
||||
assert traj.expected_answer == "X is a thing."
|
||||
assert traj.actual_answer == "An answer."
|
||||
assert traj.score == 0.9
|
||||
assert traj.judge_reason == "Almost perfect"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handles_qa_failure(
|
||||
self,
|
||||
adapter: QAPromptAdapter,
|
||||
sample_cases: list[Case[str, str, dict[str, str]]],
|
||||
) -> None:
|
||||
stub_qa = AsyncMock()
|
||||
stub_qa.answer = AsyncMock(side_effect=RuntimeError("LLM down"))
|
||||
|
||||
result = await adapter._evaluate_async(
|
||||
sample_cases, stub_qa, capture_traces=True
|
||||
)
|
||||
|
||||
assert all(o is None for o in result.outputs)
|
||||
assert all(s == 0.0 for s in result.scores)
|
||||
assert result.trajectories is not None
|
||||
assert all(t.actual_answer is None for t in result.trajectories)
|
||||
assert all(
|
||||
t.judge_reason == "QA agent failed to produce an answer"
|
||||
for t in result.trajectories
|
||||
)
|
||||
|
||||
|
||||
class TestReflectionLM:
|
||||
def test_handles_string_prompt(self) -> None:
|
||||
test_model = TestModel(custom_output_text="Reflected response")
|
||||
|
||||
with patch("evaluations.optimization.get_model", return_value=test_model):
|
||||
lm = ReflectionLM(model_config=AppConfig().qa.model, config=AppConfig())
|
||||
|
||||
result = lm("test prompt")
|
||||
|
||||
assert result == "Reflected response"
|
||||
|
||||
def test_formats_chat_messages_into_string(self) -> None:
|
||||
test_model = TestModel(custom_output_text="Chat response")
|
||||
prompts_received: list[str] = []
|
||||
|
||||
with patch("evaluations.optimization.get_model", return_value=test_model):
|
||||
lm = ReflectionLM(model_config=AppConfig().qa.model, config=AppConfig())
|
||||
|
||||
original_run_sync = lm._agent.run_sync
|
||||
|
||||
def capturing_run_sync(prompt: str, **kwargs: Any) -> Any:
|
||||
prompts_received.append(prompt)
|
||||
return original_run_sync(prompt, **kwargs)
|
||||
|
||||
lm._agent.run_sync = capturing_run_sync # type: ignore[method-assign]
|
||||
|
||||
messages: list[dict[str, Any]] = [
|
||||
{"role": "system", "content": "You are helpful."},
|
||||
{"role": "user", "content": "Hello"},
|
||||
]
|
||||
result = lm(messages)
|
||||
|
||||
assert result == "Chat response"
|
||||
assert len(prompts_received) == 1
|
||||
assert "system: You are helpful." in prompts_received[0]
|
||||
assert "user: Hello" in prompts_received[0]
|
||||
|
||||
|
||||
class TestEvaluateSync:
|
||||
def test_delegates_to_evaluate_async(
|
||||
self,
|
||||
adapter: QAPromptAdapter,
|
||||
sample_cases: list[Case[str, str, dict[str, str]]],
|
||||
) -> None:
|
||||
expected_batch: EvaluationBatch[EvalTrajectory, str | None] = EvaluationBatch(
|
||||
outputs=["answer1", "answer2"],
|
||||
scores=[0.9, 0.8],
|
||||
trajectories=None,
|
||||
)
|
||||
|
||||
with patch.object(
|
||||
adapter,
|
||||
"_evaluate_with_setup",
|
||||
new_callable=AsyncMock,
|
||||
return_value=expected_batch,
|
||||
) as mock_eval:
|
||||
result = adapter.evaluate(
|
||||
sample_cases, {"instructions": "my prompt"}, capture_traces=True
|
||||
)
|
||||
|
||||
mock_eval.assert_called_once_with(sample_cases, "my prompt", True)
|
||||
assert result is expected_batch
|
||||
|
||||
|
||||
class TestProposalAttribute:
|
||||
def test_propose_new_texts_is_none(self, adapter: QAPromptAdapter) -> None:
|
||||
assert adapter.propose_new_texts is None
|
||||
|
||||
|
||||
def _make_cases(n: int) -> list[Case[str, str, dict[str, str]]]:
|
||||
return [
|
||||
Case(
|
||||
name=f"q{i}",
|
||||
inputs=f"Question {i}?",
|
||||
expected_output=f"Answer {i}.",
|
||||
metadata={"case_index": str(i)},
|
||||
)
|
||||
for i in range(1, n + 1)
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gepa_mock_result() -> MagicMock:
|
||||
mock_result = MagicMock()
|
||||
mock_result.best_idx = 0
|
||||
mock_result.val_aggregate_scores = [0.95]
|
||||
mock_result.best_candidate = {"instructions": "optimized prompt"}
|
||||
mock_result.total_metric_calls = 10
|
||||
mock_result.num_candidates = 3
|
||||
return mock_result
|
||||
|
||||
|
||||
class TestRunOptimization:
|
||||
def _make_spec(self, db_path: Path) -> DatasetSpec:
|
||||
return DatasetSpec(
|
||||
key="test",
|
||||
db_filename="test.lancedb",
|
||||
document_loader=lambda: None, # type: ignore[return-value]
|
||||
document_mapper=lambda doc: None,
|
||||
qa_loader=lambda: None, # type: ignore[return-value]
|
||||
qa_case_builder=lambda idx, doc: None, # type: ignore[return-value]
|
||||
system_prompt="You are a test assistant.",
|
||||
)
|
||||
|
||||
def test_returns_results(self, tmp_path: Path, gepa_mock_result: MagicMock) -> None:
|
||||
spec = self._make_spec(tmp_path / "test.lancedb")
|
||||
cases = _make_cases(4)
|
||||
|
||||
with (
|
||||
patch("evaluations.optimization.get_model"),
|
||||
patch("evaluations.optimization.ReflectionLM"),
|
||||
patch("gepa.optimize", return_value=gepa_mock_result),
|
||||
):
|
||||
result = run_optimization(
|
||||
spec=spec,
|
||||
config=AppConfig(),
|
||||
cases=cases,
|
||||
num_candidates=10,
|
||||
db_path=tmp_path / "test.lancedb",
|
||||
)
|
||||
|
||||
assert result["best_score"] == 0.95
|
||||
assert result["best_prompt"] == "optimized prompt"
|
||||
assert result["total_calls"] == 10
|
||||
assert result["num_candidates"] == 3
|
||||
|
||||
def test_saves_output_file(
|
||||
self, tmp_path: Path, gepa_mock_result: MagicMock
|
||||
) -> None:
|
||||
spec = self._make_spec(tmp_path / "test.lancedb")
|
||||
cases = _make_cases(4)
|
||||
output_path = tmp_path / "prompt.txt"
|
||||
|
||||
gepa_mock_result.val_aggregate_scores = [0.85]
|
||||
gepa_mock_result.best_candidate = {"instructions": "saved prompt"}
|
||||
gepa_mock_result.total_metric_calls = 5
|
||||
gepa_mock_result.num_candidates = 2
|
||||
|
||||
with (
|
||||
patch("evaluations.optimization.get_model"),
|
||||
patch("evaluations.optimization.ReflectionLM"),
|
||||
patch("gepa.optimize", return_value=gepa_mock_result),
|
||||
):
|
||||
run_optimization(
|
||||
spec=spec,
|
||||
config=AppConfig(),
|
||||
cases=cases,
|
||||
num_candidates=5,
|
||||
db_path=tmp_path / "test.lancedb",
|
||||
output=output_path,
|
||||
)
|
||||
|
||||
assert output_path.read_text() == "saved prompt"
|
||||
|
||||
def test_uses_default_prompt_when_spec_has_none(self, tmp_path: Path) -> None:
|
||||
spec = DatasetSpec(
|
||||
key="test",
|
||||
db_filename="test.lancedb",
|
||||
document_loader=lambda: None, # type: ignore[return-value]
|
||||
document_mapper=lambda doc: None,
|
||||
qa_loader=lambda: None, # type: ignore[return-value]
|
||||
qa_case_builder=lambda idx, doc: None, # type: ignore[return-value]
|
||||
)
|
||||
cases = _make_cases(4)
|
||||
|
||||
mock_result = MagicMock()
|
||||
mock_result.best_idx = 0
|
||||
mock_result.val_aggregate_scores = [0.5]
|
||||
mock_result.best_candidate = "fallback prompt"
|
||||
mock_result.total_metric_calls = 1
|
||||
mock_result.num_candidates = 1
|
||||
|
||||
with (
|
||||
patch("evaluations.optimization.get_model"),
|
||||
patch("evaluations.optimization.ReflectionLM"),
|
||||
patch("gepa.optimize", return_value=mock_result) as mock_gepa,
|
||||
):
|
||||
result = run_optimization(
|
||||
spec=spec,
|
||||
config=AppConfig(),
|
||||
cases=cases,
|
||||
num_candidates=1,
|
||||
db_path=tmp_path / "test.lancedb",
|
||||
)
|
||||
|
||||
# When best_candidate is a string (not dict), it should be used directly
|
||||
assert result["best_prompt"] == "fallback prompt"
|
||||
|
||||
# Verify seed_candidate used QA_SYSTEM_PROMPT (not None)
|
||||
call_kwargs = mock_gepa.call_args[1]
|
||||
seed = call_kwargs["seed_candidate"]
|
||||
assert seed["instructions"] is not None
|
||||
assert len(seed["instructions"]) > 0
|
||||
|
||||
def test_splits_cases_into_train_and_val(self, tmp_path: Path) -> None:
|
||||
spec = self._make_spec(tmp_path / "test.lancedb")
|
||||
cases = _make_cases(10)
|
||||
|
||||
mock_result = MagicMock()
|
||||
mock_result.best_idx = 0
|
||||
mock_result.val_aggregate_scores = [0.7]
|
||||
mock_result.best_candidate = {"instructions": "prompt"}
|
||||
mock_result.total_metric_calls = 50
|
||||
mock_result.num_candidates = 1
|
||||
|
||||
with (
|
||||
patch("evaluations.optimization.get_model"),
|
||||
patch("evaluations.optimization.ReflectionLM"),
|
||||
patch("gepa.optimize", return_value=mock_result) as mock_gepa,
|
||||
):
|
||||
run_optimization(
|
||||
spec=spec,
|
||||
config=AppConfig(),
|
||||
cases=cases,
|
||||
num_candidates=5,
|
||||
db_path=tmp_path / "test.lancedb",
|
||||
)
|
||||
|
||||
call_kwargs = mock_gepa.call_args[1]
|
||||
assert len(call_kwargs["trainset"]) == 5
|
||||
assert len(call_kwargs["valset"]) == 5
|
||||
# Budget = valset_size + num_candidates * (2*minibatch + valset_size)
|
||||
assert call_kwargs["max_metric_calls"] == 5 + 5 * (2 * 3 + 5)
|
||||
49
uv.lock
49
uv.lock
|
|
@ -1252,6 +1252,15 @@ wheels = [
|
|||
{ url = "https://files.pythonhosted.org/packages/ae/1c/d544657381270fe21afc2a4dc560e1607e28d7115555cef148271cb25522/genai_prices-0.0.53-py3-none-any.whl", hash = "sha256:5a5dfd92089e9e8a174f7097a1521e36f4e75c74cfbdfb1ec56283bae3c0c96e", size = 61850, upload-time = "2026-02-11T20:47:16.774Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "gepa"
|
||||
version = "0.1.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f5/30/511e52916956508f56eca721260fcd524cfffd580e57782dd471be925f7e/gepa-0.1.0.tar.gz", hash = "sha256:f8b3d7918d4cdcf8593f39ef1cc757c4ba1a4e6793e3ffb622e6c0bc60a1efd9", size = 226064, upload-time = "2026-02-19T19:43:08.272Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/1f/32/fe8afb3d2a6605a6bcbc8f119f0a2adae96e9e5d57ebed055490219956a8/gepa-0.1.0-py3-none-any.whl", hash = "sha256:4e3f8fe8ca20169e60518b2e9d416e8c4a579459848adffdcad12223fbf9643e", size = 191392, upload-time = "2026-02-19T19:43:07.065Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ghp-import"
|
||||
version = "2.1.0"
|
||||
|
|
@ -1413,6 +1422,7 @@ version = "0.33.2"
|
|||
source = { editable = "evaluations" }
|
||||
dependencies = [
|
||||
{ name = "datasets" },
|
||||
{ name = "gepa" },
|
||||
{ name = "haiku-rag-slim" },
|
||||
{ name = "huggingface-hub" },
|
||||
{ name = "pydantic-ai-slim", extra = ["evals", "logfire"] },
|
||||
|
|
@ -1423,6 +1433,7 @@ dependencies = [
|
|||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "datasets", specifier = ">=4.6.1" },
|
||||
{ name = "gepa", specifier = ">=0.1.0" },
|
||||
{ name = "haiku-rag-slim", editable = "haiku_rag_slim" },
|
||||
{ name = "huggingface-hub", specifier = ">=0.20.0" },
|
||||
{ name = "pydantic-ai-slim", extras = ["evals", "logfire"], specifier = ">=1.66.0" },
|
||||
|
|
@ -4543,27 +4554,27 @@ wheels = [
|
|||
|
||||
[[package]]
|
||||
name = "ruff"
|
||||
version = "0.15.4"
|
||||
version = "0.15.5"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/da/31/d6e536cdebb6568ae75a7f00e4b4819ae0ad2640c3604c305a0428680b0c/ruff-0.15.4.tar.gz", hash = "sha256:3412195319e42d634470cc97aa9803d07e9d5c9223b99bcb1518f0c725f26ae1", size = 4569550, upload-time = "2026-02-26T20:04:14.959Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/77/9b/840e0039e65fcf12758adf684d2289024d6140cde9268cc59887dc55189c/ruff-0.15.5.tar.gz", hash = "sha256:7c3601d3b6d76dce18c5c824fc8d06f4eef33d6df0c21ec7799510cde0f159a2", size = 4574214, upload-time = "2026-03-05T20:06:34.946Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f2/82/c11a03cfec3a4d26a0ea1e571f0f44be5993b923f905eeddfc397c13d360/ruff-0.15.4-py3-none-linux_armv6l.whl", hash = "sha256:a1810931c41606c686bae8b5b9a8072adac2f611bb433c0ba476acba17a332e0", size = 10453333, upload-time = "2026-02-26T20:04:20.093Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ce/5d/6a1f271f6e31dffb31855996493641edc3eef8077b883eaf007a2f1c2976/ruff-0.15.4-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:5a1632c66672b8b4d3e1d1782859e98d6e0b4e70829530666644286600a33992", size = 10853356, upload-time = "2026-02-26T20:04:05.808Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b1/d8/0fab9f8842b83b1a9c2bf81b85063f65e93fb512e60effa95b0be49bfc54/ruff-0.15.4-py3-none-macosx_11_0_arm64.whl", hash = "sha256:a4386ba2cd6c0f4ff75252845906acc7c7c8e1ac567b7bc3d373686ac8c222ba", size = 10187434, upload-time = "2026-02-26T20:03:54.656Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/85/cc/cc220fd9394eff5db8d94dec199eec56dd6c9f3651d8869d024867a91030/ruff-0.15.4-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b2496488bdfd3732747558b6f95ae427ff066d1fcd054daf75f5a50674411e75", size = 10535456, upload-time = "2026-02-26T20:03:52.738Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fa/0f/bced38fa5cf24373ec767713c8e4cadc90247f3863605fb030e597878661/ruff-0.15.4-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3f1c4893841ff2d54cbda1b2860fa3260173df5ddd7b95d370186f8a5e66a4ac", size = 10287772, upload-time = "2026-02-26T20:04:08.138Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2b/90/58a1802d84fed15f8f281925b21ab3cecd813bde52a8ca033a4de8ab0e7a/ruff-0.15.4-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:820b8766bd65503b6c30aaa6331e8ef3a6e564f7999c844e9a547c40179e440a", size = 11049051, upload-time = "2026-02-26T20:04:03.53Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d2/ac/b7ad36703c35f3866584564dc15f12f91cb1a26a897dc2fd13d7cb3ae1af/ruff-0.15.4-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c9fb74bab47139c1751f900f857fa503987253c3ef89129b24ed375e72873e85", size = 11890494, upload-time = "2026-02-26T20:04:10.497Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/93/3d/3eb2f47a39a8b0da99faf9c54d3eb24720add1e886a5309d4d1be73a6380/ruff-0.15.4-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f80c98765949c518142b3a50a5db89343aa90f2c2bf7799de9986498ae6176db", size = 11326221, upload-time = "2026-02-26T20:04:12.84Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ff/90/bf134f4c1e5243e62690e09d63c55df948a74084c8ac3e48a88468314da6/ruff-0.15.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:451a2e224151729b3b6c9ffb36aed9091b2996fe4bdbd11f47e27d8f2e8888ec", size = 11168459, upload-time = "2026-02-26T20:04:00.969Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b5/e5/a64d27688789b06b5d55162aafc32059bb8c989c61a5139a36e1368285eb/ruff-0.15.4-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:a8f157f2e583c513c4f5f896163a93198297371f34c04220daf40d133fdd4f7f", size = 11104366, upload-time = "2026-02-26T20:03:48.099Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f1/f6/32d1dcb66a2559763fc3027bdd65836cad9eb09d90f2ed6a63d8e9252b02/ruff-0.15.4-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:917cc68503357021f541e69b35361c99387cdbbf99bd0ea4aa6f28ca99ff5338", size = 10510887, upload-time = "2026-02-26T20:03:45.771Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ff/92/22d1ced50971c5b6433aed166fcef8c9343f567a94cf2b9d9089f6aa80fe/ruff-0.15.4-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:e9737c8161da79fd7cfec19f1e35620375bd8b2a50c3e77fa3d2c16f574105cc", size = 10285939, upload-time = "2026-02-26T20:04:22.42Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e6/f4/7c20aec3143837641a02509a4668fb146a642fd1211846634edc17eb5563/ruff-0.15.4-py3-none-musllinux_1_2_i686.whl", hash = "sha256:291258c917539e18f6ba40482fe31d6f5ac023994ee11d7bdafd716f2aab8a68", size = 10765471, upload-time = "2026-02-26T20:03:58.924Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d0/09/6d2f7586f09a16120aebdff8f64d962d7c4348313c77ebb29c566cefc357/ruff-0.15.4-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:3f83c45911da6f2cd5936c436cf86b9f09f09165f033a99dcf7477e34041cbc3", size = 11263382, upload-time = "2026-02-26T20:04:24.424Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/1b/fa/2ef715a1cd329ef47c1a050e10dee91a9054b7ce2fcfdd6a06d139afb7ec/ruff-0.15.4-py3-none-win32.whl", hash = "sha256:65594a2d557d4ee9f02834fcdf0a28daa8b3b9f6cb2cb93846025a36db47ef22", size = 10506664, upload-time = "2026-02-26T20:03:50.56Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d0/a8/c688ef7e29983976820d18710f955751d9f4d4eb69df658af3d006e2ba3e/ruff-0.15.4-py3-none-win_amd64.whl", hash = "sha256:04196ad44f0df220c2ece5b0e959c2f37c777375ec744397d21d15b50a75264f", size = 11651048, upload-time = "2026-02-26T20:04:17.191Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3e/0a/9e1be9035b37448ce2e68c978f0591da94389ade5a5abafa4cf99985d1b2/ruff-0.15.4-py3-none-win_arm64.whl", hash = "sha256:60d5177e8cfc70e51b9c5fad936c634872a74209f934c1e79107d11787ad5453", size = 10966776, upload-time = "2026-02-26T20:03:56.908Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/47/20/5369c3ce21588c708bcbe517a8fbe1a8dfdb5dfd5137e14790b1da71612c/ruff-0.15.5-py3-none-linux_armv6l.whl", hash = "sha256:4ae44c42281f42e3b06b988e442d344a5b9b72450ff3c892e30d11b29a96a57c", size = 10478185, upload-time = "2026-03-05T20:06:29.093Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/44/ed/e81dd668547da281e5dce710cf0bc60193f8d3d43833e8241d006720e42b/ruff-0.15.5-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:6edd3792d408ebcf61adabc01822da687579a1a023f297618ac27a5b51ef0080", size = 10859201, upload-time = "2026-03-05T20:06:32.632Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c4/8f/533075f00aaf19b07c5cd6aa6e5d89424b06b3b3f4583bfa9c640a079059/ruff-0.15.5-py3-none-macosx_11_0_arm64.whl", hash = "sha256:89f463f7c8205a9f8dea9d658d59eff49db05f88f89cc3047fb1a02d9f344010", size = 10184752, upload-time = "2026-03-05T20:06:40.312Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/66/0e/ba49e2c3fa0395b3152bad634c7432f7edfc509c133b8f4529053ff024fb/ruff-0.15.5-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ba786a8295c6574c1116704cf0b9e6563de3432ac888d8f83685654fe528fd65", size = 10534857, upload-time = "2026-03-05T20:06:19.581Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/59/71/39234440f27a226475a0659561adb0d784b4d247dfe7f43ffc12dd02e288/ruff-0.15.5-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fd4b801e57955fe9f02b31d20375ab3a5c4415f2e5105b79fb94cf2642c91440", size = 10309120, upload-time = "2026-03-05T20:06:00.435Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f5/87/4140aa86a93df032156982b726f4952aaec4a883bb98cb6ef73c347da253/ruff-0.15.5-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:391f7c73388f3d8c11b794dbbc2959a5b5afe66642c142a6effa90b45f6f5204", size = 11047428, upload-time = "2026-03-05T20:05:51.867Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5a/f7/4953e7e3287676f78fbe85e3a0ca414c5ca81237b7575bdadc00229ac240/ruff-0.15.5-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8dc18f30302e379fe1e998548b0f5e9f4dff907f52f73ad6da419ea9c19d66c8", size = 11914251, upload-time = "2026-03-05T20:06:22.887Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/77/46/0f7c865c10cf896ccf5a939c3e84e1cfaeed608ff5249584799a74d33835/ruff-0.15.5-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1cc6e7f90087e2d27f98dc34ed1b3ab7c8f0d273cc5431415454e22c0bd2a681", size = 11333801, upload-time = "2026-03-05T20:05:57.168Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d3/01/a10fe54b653061585e655f5286c2662ebddb68831ed3eaebfb0eb08c0a16/ruff-0.15.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c1cb7169f53c1ddb06e71a9aebd7e98fc0fea936b39afb36d8e86d36ecc2636a", size = 11206821, upload-time = "2026-03-05T20:06:03.441Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7a/0d/2132ceaf20c5e8699aa83da2706ecb5c5dcdf78b453f77edca7fb70f8a93/ruff-0.15.5-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:9b037924500a31ee17389b5c8c4d88874cc6ea8e42f12e9c61a3d754ff72f1ca", size = 11133326, upload-time = "2026-03-05T20:06:25.655Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/72/cb/2e5259a7eb2a0f87c08c0fe5bf5825a1e4b90883a52685524596bfc93072/ruff-0.15.5-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:65bb414e5b4eadd95a8c1e4804f6772bbe8995889f203a01f77ddf2d790929dd", size = 10510820, upload-time = "2026-03-05T20:06:37.79Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ff/20/b67ce78f9e6c59ffbdb5b4503d0090e749b5f2d31b599b554698a80d861c/ruff-0.15.5-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:d20aa469ae3b57033519c559e9bc9cd9e782842e39be05b50e852c7c981fa01d", size = 10302395, upload-time = "2026-03-05T20:05:54.504Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5f/e5/719f1acccd31b720d477751558ed74e9c88134adcc377e5e886af89d3072/ruff-0.15.5-py3-none-musllinux_1_2_i686.whl", hash = "sha256:15388dd28c9161cdb8eda68993533acc870aa4e646a0a277aa166de9ad5a8752", size = 10754069, upload-time = "2026-03-05T20:06:06.422Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c3/9c/d1db14469e32d98f3ca27079dbd30b7b44dbb5317d06ab36718dee3baf03/ruff-0.15.5-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:b30da330cbd03bed0c21420b6b953158f60c74c54c5f4c1dabbdf3a57bf355d2", size = 11304315, upload-time = "2026-03-05T20:06:10.867Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/28/3a/950367aee7c69027f4f422059227b290ed780366b6aecee5de5039d50fa8/ruff-0.15.5-py3-none-win32.whl", hash = "sha256:732e5ee1f98ba5b3679029989a06ca39a950cced52143a0ea82a2102cb592b74", size = 10551676, upload-time = "2026-03-05T20:06:13.705Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b8/00/bf077a505b4e649bdd3c47ff8ec967735ce2544c8e4a43aba42ee9bf935d/ruff-0.15.5-py3-none-win_amd64.whl", hash = "sha256:821d41c5fa9e19117616c35eaa3f4b75046ec76c65e7ae20a333e9a8696bc7fe", size = 11678972, upload-time = "2026-03-05T20:06:45.379Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fe/4e/cd76eca6db6115604b7626668e891c9dd03330384082e33662fb0f113614/ruff-0.15.5-py3-none-win_arm64.whl", hash = "sha256:b498d1c60d2fe5c10c45ec3f698901065772730b411f164ae270bb6bfcc4740b", size = 10965572, upload-time = "2026-03-05T20:06:16.984Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
|
|||
Loading…
Reference in a new issue