danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #85 from ggozad/chore/qwen3-embeddings

Browse source 
Change default embedding to qwen3-embedding. Update benchmarks.
This commit is contained in:
Yiorgis Gozadinos 2025-09-30 11:30:02 +03:00 committed by GitHub
commit 53557b0629
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 6 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/benchmarks.md
View file

@ -17,6 +17,7 @@ The recall obtained is ~0.79 for matching in the top result, raising to ~0.91 fo
| Ollama / `mxbai-embed-large` | 0.79 | 0.91 | None |
| Ollama / `mxbai-embed-large` | 0.90 | 0.95 | `mxbai-rerank-base-v2` |
| Ollama / `nomic-embed-text-v1.5` | 0.74 | 0.90 | None |
| Ollama / `qwen3-embedding` | 0.81 | 0.95 | None |
<!-- | OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.83 | 0.90 | Cohere / `rerank-v3.5` | -->
@ -27,6 +28,7 @@ Again using the same dataset, we use a QA agent to answer the question. In addit
| Embedding Model | QA Model | Accuracy | Reranker |
|------------------------------------|-----------------------------------|-----------|------------------------|
| Ollama / `qwen3-embedding. ` | Ollama / `gpt-oss` | 0.93 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.85 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.87 | `mxbai-rerank-base-v2` |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None |

4
src/haiku/rag/config.py
View file

@ -20,8 +20,8 @@ class AppConfig(BaseModel):
MONITOR_DIRECTORIES: list[Path] = []
EMBEDDINGS_PROVIDER: str = "ollama"
EMBEDDINGS_MODEL: str = "mxbai-embed-large"
EMBEDDINGS_VECTOR_DIM: int = 1024
EMBEDDINGS_MODEL: str = "qwen3-embedding"
EMBEDDINGS_VECTOR_DIM: int = 4096
RERANK_PROVIDER: str = ""
RERANK_MODEL: str = ""

3
tests/test_client.py
View file

@ -7,6 +7,7 @@ import pytest
from datasets import Dataset
from haiku.rag.client import HaikuRAG
from haiku.rag.config import Config
from haiku.rag.store.models.chunk import Chunk
@ -504,7 +505,7 @@ async def test_client_create_document_with_custom_chunks(temp_db_path):
Chunk(
content="This is the second chunk",
metadata={"custom": "metadata2"},
embedding=[0.1] * 1024,
embedding=[0.1] * Config.EMBEDDINGS_VECTOR_DIM,
order=1,
), # With embedding
Chunk(