danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Change default embedding to qwen3-embedding. Update benchmarks using qwen3-embedding and gpt-oss

Browse source
This commit is contained in:
Yiorgis Gozadinos 2025-09-25 13:29:25 +03:00
parent cb93f0913a
commit f0333e33ac
No known key found for this signature in database
2 changed files with 4 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/benchmarks.md
View file

@ -17,6 +17,7 @@ The recall obtained is ~0.79 for matching in the top result, raising to ~0.91 fo
| Ollama / `mxbai-embed-large` | 0.79 | 0.91 | None |
| Ollama / `mxbai-embed-large` | 0.90 | 0.95 | `mxbai-rerank-base-v2` |
| Ollama / `nomic-embed-text-v1.5` | 0.74 | 0.90 | None |
| Ollama / `qwen3-embedding` | 0.81 | 0.95 | None |
<!-- | OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 | None |
| OpenAI / `text-embeddings-3-small` | 0.83 | 0.90 | Cohere / `rerank-v3.5` | -->
@ -27,6 +28,7 @@ Again using the same dataset, we use a QA agent to answer the question. In addit
| Embedding Model | QA Model | Accuracy | Reranker |
|------------------------------------|-----------------------------------|-----------|------------------------|
| Ollama / `qwen3-embedding. ` | Ollama / `gpt-oss` | 0.93 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.85 | None |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.87 | `mxbai-rerank-base-v2` |
| Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None |

4
src/haiku/rag/config.py
View file

@ -20,8 +20,8 @@ class AppConfig(BaseModel):
MONITOR_DIRECTORIES: list[Path] = []
EMBEDDINGS_PROVIDER: str = "ollama"
EMBEDDINGS_MODEL: str = "mxbai-embed-large"
EMBEDDINGS_VECTOR_DIM: int = 1024
EMBEDDINGS_MODEL: str = "qwen3-embedding"
EMBEDDINGS_VECTOR_DIM: int = 4096
RERANK_PROVIDER: str = ""
RERANK_MODEL: str = ""