Merge pull request #162 from ggozad/feat/eval-dbs

Set default evaluation  dataset db location, create evaluation script

CHANGELOG.md

@@ -4,6 +4,13 @@
 ### Added
 
 - **Database Inspector**: New `inspect` CLI command launches interactive TUI for browsing documents and chunks & searching
+- **Evaluations**: Added `evaluations` CLI script for running benchmarks (replaces `python -m evaluations.benchmark`)
+- **Evaluations**: Added `--db` option to override evaluation database path
+  - Default database location moved to haiku.rag data directory:
+    - macOS: `~/Library/Application Support/haiku.rag/evaluations/dbs/`
+    - Linux: `~/.local/share/haiku.rag/evaluations/dbs/`
+    - Windows: `C:/Users/<USER>/AppData/Roaming/haiku.rag/evaluations/dbs/`
+  - Previously stored in `evaluations/data/` within the repository
 - **Evaluations**: Added comprehensive experiment metadata tracking for better reproducibility
   - Records dataset name, test case count, and all model configurations
   - Tracks embedder settings: provider, model, and vector dimensions

docs/benchmarks.md

@@ -2,32 +2,38 @@
 
 We use the [repliqa](https://huggingface.co/datasets/ServiceNow/repliqa) dataset for the evaluation of `haiku.rag`.
 
-You can perform your own evaluations with the Typer CLI in
-`evaluations/evaluations/benchmark.py`, for example `python -m evaluations.benchmark repliqa`.
+You can perform your own evaluations with the `evaluations` CLI command:
+
+```bash
+evaluations repliqa
+```
+
 The evaluation flow is orchestrated with
 [`pydantic-evals`](https://github.com/pydantic/pydantic-ai/tree/main/libs/pydantic-evals),
 which we leverage for dataset management, scoring, and report generation.
 
 ## Configuration
 
-The benchmark script accepts a `--config` option to specify a custom `haiku.rag.yaml` configuration file:
+The benchmark script accepts several options:
 
 ```bash
-python -m evaluations.benchmark repliqa --config /path/to/haiku.rag.yaml
+evaluations repliqa --config /path/to/haiku.rag.yaml --db /path/to/custom.lancedb
 ```
 
-If no config file is specified, the script will search for a config file in the standard locations:
-1. `./haiku.rag.yaml` (current directory)
-2. User config directory
-3. Falls back to default configuration
-
-You can also use command-line options:
+**Configuration options:**
+- `--config PATH` - Specify a custom `haiku.rag.yaml` configuration file
+- `--db PATH` - Override the database path (default: `~/.local/share/haiku.rag/evaluations/dbs/{dataset}.lancedb` on Linux, `~/Library/Application Support/haiku.rag/evaluations/dbs/{dataset}.lancedb` on macOS)
 - `--skip-db` - Skip updating the evaluation database
 - `--skip-retrieval` - Skip retrieval benchmark
 - `--skip-qa` - Skip QA benchmark
 - `--limit N` - Limit number of test cases for both retrieval and QA
 - `--name NAME` - Override the evaluation name (defaults to `{dataset}_retrieval_evaluation` or `{dataset}_qa_evaluation`)
 
+If no config file is specified, the script will search for a config file in the standard locations:
+1. `./haiku.rag.yaml` (current directory)
+2. User config directory
+3. Falls back to default configuration
+
 ## RepliQA Retrieval
 
 We use the [RepliQA](https://huggingface.co/datasets/ServiceNow/repliqa) dataset to evaluate retrieval performance. We load the `News Stories` from `repliqa_3` (1035 documents) and index them. Subsequently, we run a search over the `question` field for each row of the dataset and check whether we match the document that answers the question. Questions for which the answer cannot be found in the documents are ignored.

evaluations/README.md

@@ -9,3 +9,33 @@ This package is not published to PyPI and is only used for development and testi
 Contains evaluation scripts for benchmarking RAG performance using datasets like:
 - RepliQA
 - WiX
+
+## Usage
+
+After installing the package, you can run evaluations using the `evaluations` command:
+
+```bash
+# Run evaluations with default settings
+evaluations repliqa
+
+# Use a custom config file
+evaluations repliqa --config /path/to/haiku.rag.yaml
+
+# Override the database path
+evaluations repliqa --db /path/to/custom.lancedb
+
+# Skip database population and run only benchmarks
+evaluations repliqa --skip-db
+
+# Limit the number of test cases
+evaluations repliqa --limit 100
+```
+
+## Database Storage
+
+By default, evaluation databases are stored in the haiku.rag data directory:
+- **Linux**: `~/.local/share/haiku.rag/evaluations/dbs/`
+- **macOS**: `~/Library/Application Support/haiku.rag/evaluations/dbs/`
+- **Windows**: `C:/Users/<USER>/AppData/Roaming/haiku.rag/evaluations/dbs/`
+
+You can override this with the `--db` option.

evaluations/evaluations/benchmark.py

@@ -57,15 +57,18 @@ def build_experiment_metadata(
     }
 
 
-async def populate_db(spec: DatasetSpec, config: AppConfig) -> None:
-    spec.db_path.parent.mkdir(parents=True, exist_ok=True)
+async def populate_db(
+    spec: DatasetSpec, config: AppConfig, db_path: Path | None = None
+) -> None:
+    db = spec.db_path(db_path)
+    db.parent.mkdir(parents=True, exist_ok=True)
     corpus = spec.document_loader()
     if spec.document_limit is not None:
         corpus = corpus.select(range(min(spec.document_limit, len(corpus))))
 
     with Progress() as progress:
         task = progress.add_task("[green]Populating database...", total=len(corpus))
-        async with HaikuRAG(spec.db_path, config=config) as rag:
+        async with HaikuRAG(db, config=config) as rag:
             for doc in corpus:
                 doc_mapping = cast(Mapping[str, Any], doc)
                 payload = spec.document_mapper(doc_mapping)
@@ -96,6 +99,7 @@ async def run_retrieval_benchmark(
     config: AppConfig,
     limit: int | None = None,
     name: str | None = None,
+    db_path: Path | None = None,
 ) -> dict[str, float] | None:
     if spec.retrieval_loader is None or spec.retrieval_mapper is None:
         console.print("Skipping retrieval benchmark; no retrieval config.")
@@ -137,7 +141,8 @@ async def run_retrieval_benchmark(
         evaluators=[evaluator],
     )
 
-    async with HaikuRAG(spec.db_path, config=config) as rag:
+    db = spec.db_path(db_path)
+    async with HaikuRAG(db, config=config) as rag:
 
         async def retrieval_target(question: str) -> list[str]:
             chunks = await rag.search(query=question, limit=5)
@@ -197,6 +202,7 @@ async def run_qa_benchmark(
     config: AppConfig,
     limit: int | None = None,
     name: str | None = None,
+    db_path: Path | None = None,
 ) -> ReportCaseFailure[str, str, dict[str, str]] | None:
     corpus = spec.qa_loader()
     if limit is not None:
@@ -229,7 +235,8 @@ async def run_qa_benchmark(
         ],
     )
 
-    async with HaikuRAG(spec.db_path, config=config) as rag:
+    db = spec.db_path(db_path)
+    async with HaikuRAG(db, config=config) as rag:
         system_prompt = WIX_SUPPORT_PROMPT if spec.key == "wix" else None
         qa = get_qa_agent(rag, system_prompt=system_prompt)
 
@@ -289,18 +296,21 @@ async def evaluate_dataset(
     skip_qa: bool,
     limit: int | None,
     name: str | None,
+    db_path: Path | None,
 ) -> None:
     if not skip_db:
         console.print(f"Using dataset: {spec.key}", style="bold magenta")
-        await populate_db(spec, config)
+        await populate_db(spec, config, db_path=db_path)
 
     if not skip_retrieval:
         console.print("Running retrieval benchmarks...", style="bold blue")
-        await run_retrieval_benchmark(spec, config, limit=limit, name=name)
+        await run_retrieval_benchmark(
+            spec, config, limit=limit, name=name, db_path=db_path
+        )
 
     if not skip_qa:
         console.print("\nRunning QA benchmarks...", style="bold yellow")
-        await run_qa_benchmark(spec, config, limit=limit, name=name)
+        await run_qa_benchmark(spec, config, limit=limit, name=name, db_path=db_path)
 
 
 app = typer.Typer(help="Run retrieval and QA benchmarks for configured datasets.")
@@ -312,6 +322,7 @@ def run(
     config: Path | None = typer.Option(
         None, "--config", help="Path to haiku.rag YAML config file."
     ),
+    db: Path | None = typer.Option(None, "--db", help="Override the database path."),
     skip_db: bool = typer.Option(
         False, "--skip-db", help="Skip updateing the evaluation db."
     ),
@@ -358,6 +369,7 @@ def run(
             skip_qa=skip_qa,
             limit=limit,
             name=name,
+            db_path=db,
         )
     )
 

evaluations/evaluations/config.py

@@ -43,6 +43,19 @@ class DatasetSpec:
     retrieval_evaluator: Evaluator | None = None
     document_limit: int | None = None
 
-    @property
-    def db_path(self) -> Path:
-        return Path(__file__).parent / "data" / self.db_filename
+    def db_path(self, override_path: Path | None = None) -> Path:
+        """Get the database path.
+
+        Args:
+            override_path: Optional path to override the default database location.
+
+        Returns:
+            The database path to use.
+        """
+        if override_path is not None:
+            return override_path
+
+        from haiku.rag.utils import get_default_data_dir
+
+        data_dir = get_default_data_dir()
+        return data_dir / "evaluations" / "dbs" / self.db_filename

evaluations/pyproject.toml

@@ -15,6 +15,9 @@ dependencies = [
     "python-dotenv>=1.2.1",
 ]
 
+[project.scripts]
+evaluations = "evaluations.benchmark:app"
+
 [build-system]
 requires = ["hatchling"]
 build-backend = "hatchling.build"