Merge pull request #552 from ggozad/worktree-visibility-metadata
Fix the MCP registry entry and sharpen discovery metadata
This commit is contained in:
Yiorgis Gozadinos
GitHub
commit
c5a8e5571d
10 changed files with 232 additions and 163 deletions
11
CHANGELOG.md
11
CHANGELOG.md
|
|
@ -12,15 +12,18 @@
|
|||
|
||||
- Eval judge pinned to `qwen3.8`: `DEFAULT_JUDGE_MODEL` is `ollama:qwen3.8`, and the reference configs use `Inferact/Qwen3.8-27B-NVFP4` with `extra_body.chat_template_kwargs.reasoning_effort: low`. Results in `docs/benchmarks.md` were judged by `Qwen3.6-35B-A3B-NVFP4` and are not re-judged.
|
||||
- `import_documents` embeds chunks across the whole batch in one pass instead of per document.
|
||||
|
||||
### Fixed
|
||||
|
||||
- `DocumentRepository.delete_all` recreated `document_items` from `DocumentItemRecord` instead of `get_document_items_arrow_schema()`, returning `picture_data` as `binary` rather than `large_binary`.
|
||||
- `haiku.rag` and `haiku.rag-slim` summaries and keywords; both packages now publish `[project.urls]`.
|
||||
- `server.json` declares `title` and `websiteUrl`, and drops the `keywords` and `license` keys, which are not in the server schema.
|
||||
|
||||
### Removed
|
||||
|
||||
- `wix` evaluation dataset and its reference config `evaluations/configs/wix.yaml`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- `DocumentRepository.delete_all` recreated `document_items` from `DocumentItemRecord` instead of `get_document_items_arrow_schema()`, returning `picture_data` as `binary` rather than `large_binary`.
|
||||
- `server.json` runtime arguments are `mcp --stdio`, was `serve --mcp`.
|
||||
|
||||
## [0.74.0] - 2026-08-13
|
||||
|
||||
### Added
|
||||
|
|
|
|||
10
README.md
10
README.md
|
|
@ -1,9 +1,15 @@
|
|||
# Haiku RAG
|
||||
# haiku.rag
|
||||
|
||||
[](https://pypi.org/project/haiku.rag/)
|
||||
[](https://pypi.org/project/haiku.rag/)
|
||||
[](https://pepy.tech/projects/haiku-rag-slim)
|
||||
[](https://ggozad.github.io/haiku.rag/)
|
||||
[](https://github.com/ggozad/haiku.rag/actions/workflows/test.yml)
|
||||
[](https://codecov.io/gh/ggozad/haiku.rag)
|
||||
|
||||
Agentic RAG built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/), and [Docling](https://docling-project.github.io/docling/).
|
||||
Agentic RAG that answers questions about your own documents with citations to page numbers and section headings. Runs locally on an embedded database, no server required.
|
||||
|
||||
Built on [LanceDB](https://lancedb.com/), [Pydantic AI](https://ai.pydantic.dev/), and [Docling](https://docling-project.github.io/docling/). Full documentation at [ggozad.github.io/haiku.rag](https://ggozad.github.io/haiku.rag/).
|
||||
|
||||
> **New: vision and multimodal search.** Picture-aware ingestion captures embedded figure bytes; vision-capable QA models receive them alongside text. Multimodal embedders put picture vectors in the same space as text, enabling text-as-query → figure hits and image-as-query retrieval.
|
||||
|
||||
|
|
|
|||
|
|
@ -2,139 +2,6 @@
|
|||
|
||||
We evaluate `haiku.rag` on a small set of datasets that exercise different parts of the pipeline. OpenRAG Bench (ORB), T²-RAGBench, HotpotQA, and MTRAG are the datasets we currently track. Retrieval, QA accuracy, and citation retrieval are scored end-to-end through the RAG and analysis capabilities.
|
||||
|
||||
## Running Evaluations
|
||||
|
||||
You can run evaluations with the `evaluations` CLI:
|
||||
|
||||
```bash
|
||||
evaluations run hotpotqa
|
||||
evaluations run orb_text
|
||||
```
|
||||
|
||||
The evaluation flow is orchestrated with [`pydantic-evals`](https://github.com/pydantic/pydantic-ai/tree/main/libs/pydantic-evals), which we leverage for dataset management, scoring, and report generation.
|
||||
|
||||
### Pre-built Databases
|
||||
|
||||
Building evaluation databases from scratch can take a long time, especially for large datasets like OpenRAG Bench. Pre-built databases are available on HuggingFace:
|
||||
|
||||
```bash
|
||||
# Download a specific dataset
|
||||
evaluations download hotpotqa
|
||||
|
||||
# Download all datasets
|
||||
evaluations download all
|
||||
|
||||
# Force re-download (overwrite existing)
|
||||
evaluations download hotpotqa --force
|
||||
```
|
||||
|
||||
Active datasets:
|
||||
|
||||
| Dataset | Size |
|
||||
|---------|------|
|
||||
| `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~18 GB |
|
||||
| `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB |
|
||||
| `orb_multimodal_nemotron` — OpenRAG Bench, multimodal embedder (`nvidia/llama-nemotron-embed-vl-1b-v2`), the embedder behind the published headline results | ~16 GB |
|
||||
| `t2_finqa` — T²-RAGBench (FinQA) financial QA, text embedder (`qwen3-embedding:4b`); scored by exact numeric match, run with `--target analysis-capability` | ~2 GB |
|
||||
| `hotpotqa` — HotpotQA multi-hop QA over Wikipedia paragraphs, text embedder (`qwen3-embedding:4b`) | ~1.5 GB |
|
||||
| `mtrag_clapnq` — MTRAG multi-turn RAG, ClapNQ (Wikipedia) passages, text embedder (`qwen3-embedding:4b`); also serves the `mtrag_clapnq_rewrite`, `mtrag_clapnq_live` and `mtrag_clapnq_live_uncompacted` keys | ~2.8 GB |
|
||||
|
||||
After downloading, run benchmarks with `--skip-db`. Each database is built with a specific embedder, so pass its reference config from `evaluations/configs/` (a database only opens against a config whose embedder matches):
|
||||
|
||||
```bash
|
||||
evaluations run orb_multimodal_nemotron --skip-db --config configs/orb_multimodal_nemotron.yaml
|
||||
```
|
||||
|
||||
The configs use `vllm` as the model host. Point `base_url` at your own OpenAI-compatible endpoints to reproduce the numbers.
|
||||
|
||||
### Configuration
|
||||
|
||||
The benchmark script accepts several options:
|
||||
|
||||
```bash
|
||||
evaluations run hotpotqa --config /path/to/haiku.rag.yaml --db /path/to/custom.lancedb
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
- `--config PATH` - Specify a custom `haiku.rag.yaml` configuration file
|
||||
- `--db PATH` - Override the database path (default: platform-specific user data directory)
|
||||
- `--skip-db` - Skip updating the evaluation database
|
||||
- `--skip-retrieval` - Skip retrieval benchmark
|
||||
- `--skip-qa` - Skip QA benchmark
|
||||
- `--limit N` - Limit number of test cases
|
||||
- `--name NAME` - Override the evaluation name
|
||||
- `--target {rag-capability,analysis-capability}` - Choose which [capability](capabilities/index.md) to benchmark end-to-end (default: `rag-capability`). The target names remain stable dataset identifiers.
|
||||
- `--capability-model PROVIDER:NAME` - Override the capability model independently from the judge (default: `config.qa.model`, or `config.analysis.model` when set for `--target analysis-capability`).
|
||||
- `--filter CLAUSE` / `-f CLAUSE` - Restrict every benchmark search to a subset of the database (see [Restricting the corpus](#restricting-the-corpus)).
|
||||
|
||||
If no config file is specified, the script searches standard locations: `./haiku.rag.yaml`, user config directory, then falls back to defaults.
|
||||
|
||||
To pin the LLM judge in YAML (rather than the default `ollama:qwen3.8`). These are the recommended settings:
|
||||
|
||||
```yaml
|
||||
evaluations:
|
||||
judge:
|
||||
provider: openai
|
||||
name: Inferact/Qwen3.8-27B-NVFP4
|
||||
base_url: http://localhost:8000/v1 # optional, for OpenAI-compatible servers (vLLM, LM Studio, etc.)
|
||||
temperature: 0.6
|
||||
max_tokens: 16384
|
||||
extra_body:
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
min_p: 0
|
||||
chat_template_kwargs:
|
||||
reasoning_effort: low # qwen3.8: low | medium | xhigh (default)
|
||||
```
|
||||
|
||||
### Restricting the corpus
|
||||
|
||||
When a database holds documents from several corpora — only some of which a dataset's questions are drawn from — `--filter` restricts every benchmark search to a subset. It takes the same SQL `WHERE` clause as `haiku-rag search --filter`, over document columns (`id`, `uri`, `title`, `created_at`, `updated_at`, `metadata`). Each dataset writes its own URIs: `orb_text` uses bare arXiv ids such as `2407.01528v3`, `hotpotqa` uses page titles.
|
||||
|
||||
```bash
|
||||
evaluations run orb_text --skip-db --config haiku.rag.s3.yaml \
|
||||
--filter "uri LIKE '2407%'"
|
||||
```
|
||||
|
||||
If the corpora are distinguished by a tag rather than by URI, attach it at ingest time as document metadata and match it with `LIKE`. `metadata` is stored as a `json.dumps` string, so there is no JSON subfield access — match the serialized key/value, including the space after the colon:
|
||||
|
||||
```bash
|
||||
evaluations run orb_text --skip-db --filter "metadata LIKE '%\"corpus\": \"orb_text\"%'"
|
||||
```
|
||||
|
||||
The clause applies to both benchmark phases — the retrieval benchmark's searches and every search the capability runs during QA — so the two score the same subset. It is recorded as `document_filter` in the run's experiment metadata, so a filtered run is never mistaken for an unfiltered one when comparing results.
|
||||
|
||||
Filtering affects searches only — a run without `--skip-db` still populates the database with the dataset's full corpus.
|
||||
|
||||
## Methodology
|
||||
|
||||
### Retrieval Metrics
|
||||
|
||||
**Mean Average Precision (MAP)** scores ranked retrieval results against the gold `expected_uris`.
|
||||
|
||||
- For each relevant document at position k, calculate precision@k = (relevant docs in top k) / k
|
||||
- Average Precision (AP) = sum of these precision values / total relevant documents
|
||||
- MAP is the mean of AP scores across all queries
|
||||
- Range: 0 to 1. Rewards ranking relevant documents higher
|
||||
- For single-doc queries this collapses to `1/rank` (i.e. reciprocal rank)
|
||||
|
||||
### QA Accuracy
|
||||
|
||||
`pydantic-evals` coordinates an LLM judge to determine whether the capability's answer is correct. The default judge is `ollama:qwen3.8`, pinned so changes to the capability model don't change the judge underneath. Set `evaluations.judge` in `haiku.rag.yaml` to override (including a custom `base_url` for any OpenAI-compatible endpoint). Accuracy is the fraction of correctly answered questions.
|
||||
|
||||
A dataset that brings its own deterministic evaluator is scored by that evaluator instead, and no judge runs. T²-RAGBench is the only such dataset today, scored by `NumberMatchEvaluator`.
|
||||
|
||||
`qwen3.8` replaced `qwen3.6` after a 120-case calibration on ORB, stratified 60 pass / 60 fail: agreement 0.950, Cohen's κ 0.900, and in all 6 disagreements it matched or beat `qwen3.6` (4 were `qwen3.6` failing answers that were equivalent in different notation). It emits no reasoning content, so it avoids the thinking spirals that made `qwen3.6` exceed its output budget and drop verdicts. `reasoning_effort` changes its verdicts in 1 case per 120, so the cheaper `low` is pinned.
|
||||
|
||||
Before that, we picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibration (gpt-oss / qwen3.6 as both answerer and judge, with Claude Opus 4.7 as a reference). `qwen3.6` had κ ≥ 0.66 vs the reference on both same-family and cross-family answerers (vs ~0.39–0.55 for `gpt-oss`) and showed no measurable self-preference bias, while `gpt-oss` was ~10 pp more lenient on its own outputs.
|
||||
|
||||
### Citation Retrieval
|
||||
|
||||
Alongside QA accuracy, a second metric scores the URIs the capability registered via the `cite` tool against each dataset's gold `expected_uris`, using the same MAP math as raw retrieval. The score key is `cited_map`. Console output also includes the cite rate (% of cases with at least one citation) and the mean number of citations per case.
|
||||
|
||||
This is computed alongside QA accuracy from the same capability run, no extra invocations. The signal complements raw retrieval: where raw retrieval measures whether the retriever surfaced the gold document at any rank, citation retrieval measures whether the capability grounded its answer on it.
|
||||
|
||||
## Current results
|
||||
|
||||
Numbers below were measured under `Qwen3.6-35B-A3B-NVFP4` as judge, on a recent `haiku.rag` version. The pinned judge is now `qwen3.8`; rows are not re-judged, so compare rows to each other rather than to runs judged by `qwen3.8`.
|
||||
|
|
@ -262,3 +129,136 @@ The two live arms replay the same 29 conversations (224 turns) and differ only i
|
|||
- Answer pass rate: 185/224 vs 175/224 turns. Of the 18 turns where the arms disagree, 14 pass only compacted and 4 only uncompacted. McNemar exact two-sided p = 0.031. The paired difference is +4.5pp with a Wald 95% CI of +0.8 to +8.1pp, so the honest claim is an improvement of roughly 1 to 8 points, not the point estimate.
|
||||
- Citation MAP, macro-averaged over conversations with 208 of 224 turns eligible (turns with gold passages) in each arm: 0.4174 compacted vs 0.4230 uncompacted. The gold-prefix 0.35 is over 208 of 224 eligible cases.
|
||||
- Refusal precision and recall against the answerability labels (16 UNANSWERABLE turns per arm): compacted 0.33 precision and 0.44 recall (21 refusals), uncompacted 0.23 and 0.31 (22 refusals). Gold-prefix: 0.24 and 0.44 (29 refusals).
|
||||
|
||||
## Methodology
|
||||
|
||||
### Retrieval Metrics
|
||||
|
||||
**Mean Average Precision (MAP)** scores ranked retrieval results against the gold `expected_uris`.
|
||||
|
||||
- For each relevant document at position k, calculate precision@k = (relevant docs in top k) / k
|
||||
- Average Precision (AP) = sum of these precision values / total relevant documents
|
||||
- MAP is the mean of AP scores across all queries
|
||||
- Range: 0 to 1. Rewards ranking relevant documents higher
|
||||
- For single-doc queries this collapses to `1/rank` (i.e. reciprocal rank)
|
||||
|
||||
### QA Accuracy
|
||||
|
||||
`pydantic-evals` coordinates an LLM judge to determine whether the capability's answer is correct. The default judge is `ollama:qwen3.8`, pinned so changes to the capability model don't change the judge underneath. Set `evaluations.judge` in `haiku.rag.yaml` to override (including a custom `base_url` for any OpenAI-compatible endpoint). Accuracy is the fraction of correctly answered questions.
|
||||
|
||||
A dataset that brings its own deterministic evaluator is scored by that evaluator instead, and no judge runs. T²-RAGBench is the only such dataset today, scored by `NumberMatchEvaluator`.
|
||||
|
||||
`qwen3.8` replaced `qwen3.6` after a 120-case calibration on ORB, stratified 60 pass / 60 fail: agreement 0.950, Cohen's κ 0.900, and in all 6 disagreements it matched or beat `qwen3.6` (4 were `qwen3.6` failing answers that were equivalent in different notation). It emits no reasoning content, so it avoids the thinking spirals that made `qwen3.6` exceed its output budget and drop verdicts. `reasoning_effort` changes its verdicts in 1 case per 120, so the cheaper `low` is pinned.
|
||||
|
||||
Before that, we picked `qwen3.6` over the previously-pinned `gpt-oss` after a 4-cell calibration (gpt-oss / qwen3.6 as both answerer and judge, with Claude Opus 4.7 as a reference). `qwen3.6` had κ ≥ 0.66 vs the reference on both same-family and cross-family answerers (vs ~0.39–0.55 for `gpt-oss`) and showed no measurable self-preference bias, while `gpt-oss` was ~10 pp more lenient on its own outputs.
|
||||
|
||||
### Citation Retrieval
|
||||
|
||||
Alongside QA accuracy, a second metric scores the URIs the capability registered via the `cite` tool against each dataset's gold `expected_uris`, using the same MAP math as raw retrieval. The score key is `cited_map`. Console output also includes the cite rate (% of cases with at least one citation) and the mean number of citations per case.
|
||||
|
||||
This is computed alongside QA accuracy from the same capability run, no extra invocations. The signal complements raw retrieval: where raw retrieval measures whether the retriever surfaced the gold document at any rank, citation retrieval measures whether the capability grounded its answer on it.
|
||||
|
||||
## Running Evaluations
|
||||
|
||||
You can run evaluations with the `evaluations` CLI:
|
||||
|
||||
```bash
|
||||
evaluations run hotpotqa
|
||||
evaluations run orb_text
|
||||
```
|
||||
|
||||
The evaluation flow is orchestrated with [`pydantic-evals`](https://github.com/pydantic/pydantic-ai/tree/main/libs/pydantic-evals), which we leverage for dataset management, scoring, and report generation.
|
||||
|
||||
### Pre-built Databases
|
||||
|
||||
Building evaluation databases from scratch can take a long time, especially for large datasets like OpenRAG Bench. Pre-built databases are available on HuggingFace:
|
||||
|
||||
```bash
|
||||
# Download a specific dataset
|
||||
evaluations download hotpotqa
|
||||
|
||||
# Download all datasets
|
||||
evaluations download all
|
||||
|
||||
# Force re-download (overwrite existing)
|
||||
evaluations download hotpotqa --force
|
||||
```
|
||||
|
||||
Active datasets:
|
||||
|
||||
| Dataset | Size |
|
||||
|---------|------|
|
||||
| `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~18 GB |
|
||||
| `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB |
|
||||
| `orb_multimodal_nemotron` — OpenRAG Bench, multimodal embedder (`nvidia/llama-nemotron-embed-vl-1b-v2`), the embedder behind the published headline results | ~16 GB |
|
||||
| `t2_finqa` — T²-RAGBench (FinQA) financial QA, text embedder (`qwen3-embedding:4b`); scored by exact numeric match, run with `--target analysis-capability` | ~2 GB |
|
||||
| `hotpotqa` — HotpotQA multi-hop QA over Wikipedia paragraphs, text embedder (`qwen3-embedding:4b`) | ~1.5 GB |
|
||||
| `mtrag_clapnq` — MTRAG multi-turn RAG, ClapNQ (Wikipedia) passages, text embedder (`qwen3-embedding:4b`); also serves the `mtrag_clapnq_rewrite`, `mtrag_clapnq_live` and `mtrag_clapnq_live_uncompacted` keys | ~2.8 GB |
|
||||
|
||||
After downloading, run benchmarks with `--skip-db`. Each database is built with a specific embedder, so pass its reference config from `evaluations/configs/` (a database only opens against a config whose embedder matches):
|
||||
|
||||
```bash
|
||||
evaluations run orb_multimodal_nemotron --skip-db --config configs/orb_multimodal_nemotron.yaml
|
||||
```
|
||||
|
||||
The configs use `vllm` as the model host. Point `base_url` at your own OpenAI-compatible endpoints to reproduce the numbers.
|
||||
|
||||
### Configuration
|
||||
|
||||
The benchmark script accepts several options:
|
||||
|
||||
```bash
|
||||
evaluations run hotpotqa --config /path/to/haiku.rag.yaml --db /path/to/custom.lancedb
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
- `--config PATH` - Specify a custom `haiku.rag.yaml` configuration file
|
||||
- `--db PATH` - Override the database path (default: platform-specific user data directory)
|
||||
- `--skip-db` - Skip updating the evaluation database
|
||||
- `--skip-retrieval` - Skip retrieval benchmark
|
||||
- `--skip-qa` - Skip QA benchmark
|
||||
- `--limit N` - Limit number of test cases
|
||||
- `--name NAME` - Override the evaluation name
|
||||
- `--target {rag-capability,analysis-capability}` - Choose which [capability](capabilities/index.md) to benchmark end-to-end (default: `rag-capability`). The target names remain stable dataset identifiers.
|
||||
- `--capability-model PROVIDER:NAME` - Override the capability model independently from the judge (default: `config.qa.model`, or `config.analysis.model` when set for `--target analysis-capability`).
|
||||
- `--filter CLAUSE` / `-f CLAUSE` - Restrict every benchmark search to a subset of the database (see [Restricting the corpus](#restricting-the-corpus)).
|
||||
|
||||
If no config file is specified, the script searches standard locations: `./haiku.rag.yaml`, user config directory, then falls back to defaults.
|
||||
|
||||
To pin the LLM judge in YAML (rather than the default `ollama:qwen3.8`). These are the recommended settings:
|
||||
|
||||
```yaml
|
||||
evaluations:
|
||||
judge:
|
||||
provider: openai
|
||||
name: Inferact/Qwen3.8-27B-NVFP4
|
||||
base_url: http://localhost:8000/v1 # optional, for OpenAI-compatible servers (vLLM, LM Studio, etc.)
|
||||
temperature: 0.6
|
||||
max_tokens: 16384
|
||||
extra_body:
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
min_p: 0
|
||||
chat_template_kwargs:
|
||||
reasoning_effort: low # qwen3.8: low | medium | xhigh (default)
|
||||
```
|
||||
|
||||
### Restricting the corpus
|
||||
|
||||
When a database holds documents from several corpora — only some of which a dataset's questions are drawn from — `--filter` restricts every benchmark search to a subset. It takes the same SQL `WHERE` clause as `haiku-rag search --filter`, over document columns (`id`, `uri`, `title`, `created_at`, `updated_at`, `metadata`). Each dataset writes its own URIs: `orb_text` uses bare arXiv ids such as `2407.01528v3`, `hotpotqa` uses page titles.
|
||||
|
||||
```bash
|
||||
evaluations run orb_text --skip-db --config haiku.rag.s3.yaml \
|
||||
--filter "uri LIKE '2407%'"
|
||||
```
|
||||
|
||||
If the corpora are distinguished by a tag rather than by URI, attach it at ingest time as document metadata and match it with `LIKE`. `metadata` is stored as a `json.dumps` string, so there is no JSON subfield access — match the serialized key/value, including the space after the colon:
|
||||
|
||||
```bash
|
||||
evaluations run orb_text --skip-db --filter "metadata LIKE '%\"corpus\": \"orb_text\"%'"
|
||||
```
|
||||
|
||||
The clause applies to both benchmark phases — the retrieval benchmark's searches and every search the capability runs during QA — so the two score the same subset. It is recorded as `document_filter` in the run's experiment metadata, so a filtered run is never mistaken for an unfiltered one when comparing results.
|
||||
|
||||
Filtering affects searches only — a run without `--skip-db` still populates the database with the dataset's full corpus.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,37 @@
|
|||
---
|
||||
title: haiku.rag
|
||||
description: Local-first agentic RAG. Index PDFs, web pages, and whole directories, then ask questions and get answers cited to page numbers and section headings. Hybrid search, reranking, and multimodal retrieval on embedded LanceDB.
|
||||
---
|
||||
|
||||
haiku.rag indexes PDFs, web pages, and whole directories, retrieves with hybrid search, and answers with citations down to the page number and section heading. It runs on an embedded database with open models, so your documents stay on your machine and there is no server to operate.
|
||||
|
||||
```bash
|
||||
uv pip install haiku.rag
|
||||
|
||||
haiku-rag init
|
||||
haiku-rag add-src ~/Documents/some-paper.pdf
|
||||
haiku-rag ask "what does it conclude?"
|
||||
```
|
||||
|
||||
[Quickstart](tutorial.md) covers provider setup and the first ingestion.
|
||||
|
||||
## Why haiku.rag
|
||||
|
||||
**Answers you can check.** Every answer carries citations with page numbers and section headings. Visual grounding shows the cited chunk highlighted on the original page image. Optional capabilities require an answer to declare what grounds it, including declaring that nothing does.
|
||||
|
||||
**Local-first, no server.** Embedded [LanceDB](https://lancedb.com/) and open models through [Ollama](https://ollama.com/) by default. No database to run and no API keys required. The same code runs against S3, GCS, Azure, LanceDB Cloud, or any provider Pydantic AI supports.
|
||||
|
||||
**Built for agents.** Native [Pydantic AI](https://ai.pydantic.dev/) capabilities compose into your own agents. An [MCP server](mcp.md) exposes the same database to Claude Desktop and other assistants. The analysis capability runs sandboxed Python across documents for questions that need computation rather than retrieval.
|
||||
|
||||
**Measured, not asserted.** Retrieval and answer quality are tracked against public benchmarks with runnable configs. See [Benchmarks](benchmarks.md).
|
||||
|
||||
## Start here
|
||||
|
||||
- [Quickstart](tutorial.md): install, index, chat.
|
||||
- [Overview](overview.md): what haiku.rag does, end to end.
|
||||
- [Capabilities](capabilities/index.md): native RAG and analysis capabilities for Pydantic AI agents.
|
||||
- [Python API](python.md): use haiku.rag from code.
|
||||
- [MCP server](mcp.md): expose haiku.rag to Claude Desktop or other AI assistants.
|
||||
- [Configuration](configuration/index.md): every setting.
|
||||
|
||||
MIT licensed. Source on [GitHub](https://github.com/ggozad/haiku.rag).
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Haiku RAG - Evaluations
|
||||
# haiku.rag - Evaluations
|
||||
|
||||
Internal benchmarking and evaluation scripts for haiku.rag.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +1,28 @@
|
|||
[project]
|
||||
|
||||
name = "haiku.rag-slim"
|
||||
description = "Opinionated agentic RAG powered by LanceDB, Pydantic AI, and Docling - Minimal dependencies"
|
||||
description = "Local-first agentic RAG with citations - hybrid search, reranking and multimodal retrieval over your own documents, no database server required - Minimal dependencies"
|
||||
version = "0.75.0"
|
||||
authors = [{ name = "Yiorgis Gozadinos", email = "ggozadinos@gmail.com" }]
|
||||
license = { text = "MIT" }
|
||||
readme = { file = "README.md", content-type = "text/markdown" }
|
||||
requires-python = ">=3.12"
|
||||
keywords = ["RAG", "lancedb", "vector-database", "ml", "mcp"]
|
||||
keywords = [
|
||||
"RAG",
|
||||
"agentic-rag",
|
||||
"lancedb",
|
||||
"vector-database",
|
||||
"hybrid-search",
|
||||
"reranking",
|
||||
"multimodal-rag",
|
||||
"embeddings",
|
||||
"citations",
|
||||
"document-ingestion",
|
||||
"mcp",
|
||||
"mcp-server",
|
||||
"pydantic-ai",
|
||||
"docling",
|
||||
]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Environment :: Console",
|
||||
|
|
@ -41,6 +56,13 @@ dependencies = [
|
|||
"zstandard>=0.23.0; python_version<'3.14'",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://ggozad.github.io/haiku.rag/"
|
||||
Documentation = "https://ggozad.github.io/haiku.rag/"
|
||||
Repository = "https://github.com/ggozad/haiku.rag"
|
||||
Issues = "https://github.com/ggozad/haiku.rag/issues"
|
||||
Changelog = "https://ggozad.github.io/haiku.rag/changelog/"
|
||||
|
||||
[project.optional-dependencies]
|
||||
# Document processing
|
||||
docling = ["docling>=2.102.2,<3.0.0", "opencv-python-headless>=4.6.0.66,<5.0.0.0"]
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
{% block extrahead %}
|
||||
<meta property="og:type" content="website">
|
||||
<meta property="og:title" content="{% if page.title and page.title != config.site_name %}{{ page.title }} – {{ config.site_name }}{% else %}{{ config.site_name }}{% endif %}">
|
||||
<meta property="og:description" content="{{ config.site_description }}">
|
||||
<meta property="og:description" content="{% if page.meta and page.meta.description %}{{ page.meta.description }}{% else %}{{ config.site_description }}{% endif %}">
|
||||
<meta property="og:url" content="{{ page.canonical_url }}">
|
||||
<meta property="og:image" content="{{ config.site_url }}img/chat-qa.png">
|
||||
<meta name="twitter:card" content="summary_large_image">
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
<div class="haiku-rag-hero__inner">
|
||||
<div class="haiku-rag-hero__text">
|
||||
<h1 class="haiku-rag-hero__title">haiku.rag</h1>
|
||||
<p class="haiku-rag-hero__tagline">Opinionated agentic RAG powered by <a href="https://lancedb.com/">LanceDB</a>, <a href="https://ai.pydantic.dev/">Pydantic AI</a>, and <a href="https://docling-project.github.io/docling/">Docling</a>. Runs locally, scales to production.</p>
|
||||
<p class="haiku-rag-hero__tagline">Ask questions about your own documents and get answers that cite their sources. Agentic RAG on <a href="https://lancedb.com/">LanceDB</a>, <a href="https://ai.pydantic.dev/">Pydantic AI</a>, and <a href="https://docling-project.github.io/docling/">Docling</a>. Runs locally, scales to production.</p>
|
||||
<div class="haiku-rag-hero__actions">
|
||||
<a class="md-button md-button--primary" href="tutorial/">Get started</a>
|
||||
<a class="md-button" href="overview/">Learn more</a>
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
[project]
|
||||
|
||||
name = "haiku.rag"
|
||||
description = "Opinionated agentic RAG powered by LanceDB, Pydantic AI, and Docling"
|
||||
description = "Local-first agentic RAG with citations - hybrid search, reranking and multimodal retrieval over your own documents, no database server required"
|
||||
version = "0.75.0"
|
||||
authors = [{ name = "Yiorgis Gozadinos", email = "ggozadinos@gmail.com" }]
|
||||
license = { text = "MIT" }
|
||||
|
|
@ -9,10 +9,17 @@ readme = { file = "README.md", content-type = "text/markdown" }
|
|||
requires-python = ">=3.12"
|
||||
keywords = [
|
||||
"RAG",
|
||||
"agentic-rag",
|
||||
"lancedb",
|
||||
"vector-database",
|
||||
"ml",
|
||||
"hybrid-search",
|
||||
"reranking",
|
||||
"multimodal-rag",
|
||||
"embeddings",
|
||||
"citations",
|
||||
"document-ingestion",
|
||||
"mcp",
|
||||
"mcp-server",
|
||||
"pydantic-ai",
|
||||
"docling",
|
||||
]
|
||||
|
|
@ -33,6 +40,13 @@ dependencies = [
|
|||
"haiku.rag-slim[docling,voyageai,cohere,zeroentropy,tui,cross-encoder]==0.75.0",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://ggozad.github.io/haiku.rag/"
|
||||
Documentation = "https://ggozad.github.io/haiku.rag/"
|
||||
Repository = "https://github.com/ggozad/haiku.rag"
|
||||
Issues = "https://github.com/ggozad/haiku.rag/issues"
|
||||
Changelog = "https://ggozad.github.io/haiku.rag/changelog/"
|
||||
|
||||
[project.scripts]
|
||||
haiku-rag = "haiku.rag.cli:cli"
|
||||
|
||||
|
|
|
|||
20
server.json
20
server.json
|
|
@ -1,24 +1,14 @@
|
|||
{
|
||||
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
|
||||
"name": "io.github.ggozad/haiku-rag",
|
||||
"title": "haiku.rag",
|
||||
"description": "Local-first agentic RAG with citations - hybrid search, reranking, multimodal document retrieval",
|
||||
"version": "{{VERSION}}",
|
||||
"description": "Opinionated agentic RAG powered by LanceDB, Pydantic AI, and Docling",
|
||||
"websiteUrl": "https://ggozad.github.io/haiku.rag/",
|
||||
"repository": {
|
||||
"url": "https://github.com/ggozad/haiku.rag",
|
||||
"source": "github"
|
||||
},
|
||||
"license": "MIT",
|
||||
"keywords": [
|
||||
"rag",
|
||||
"lancedb",
|
||||
"vector-database",
|
||||
"embeddings",
|
||||
"search",
|
||||
"qa",
|
||||
"research",
|
||||
"docling",
|
||||
"pydantic-ai"
|
||||
],
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "pypi",
|
||||
|
|
@ -29,11 +19,11 @@
|
|||
"runtimeArguments": [
|
||||
{
|
||||
"type": "positional",
|
||||
"value": "serve"
|
||||
"value": "mcp"
|
||||
},
|
||||
{
|
||||
"type": "named",
|
||||
"name": "--mcp"
|
||||
"name": "--stdio"
|
||||
}
|
||||
],
|
||||
"transport": {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
[project]
|
||||
site_name = "haiku.rag"
|
||||
site_description = "Opinionated agentic RAG powered by LanceDB, Pydantic AI, and Docling."
|
||||
site_description = "Local-first agentic RAG. Index your documents, then ask questions and get answers cited to page numbers and section headings. Hybrid search, reranking, and multimodal retrieval on embedded LanceDB."
|
||||
site_url = "https://ggozad.github.io/haiku.rag/"
|
||||
repo_url = "https://github.com/ggozad/haiku.rag"
|
||||
repo_name = "ggozad/haiku.rag"
|
||||
|
|
@ -46,8 +46,8 @@ nav = [
|
|||
{ Toolsets = "tools.md" },
|
||||
{ "Web app" = "apps.md" },
|
||||
] },
|
||||
{ Benchmarks = "benchmarks.md" },
|
||||
{ Reference = [
|
||||
{ Benchmarks = "benchmarks.md" },
|
||||
{ Development = "development.md" },
|
||||
{ Changelog = "changelog.md" },
|
||||
] },
|
||||
|
|
|
|||
Loading…
Reference in a new issue