Update docs for wix benchmarks
This commit is contained in:
parent
abfc796c42
commit
06dbdbd474
1 changed files with 26 additions and 0 deletions
|
|
@ -37,3 +37,29 @@ determine whether the answer is correct. The obtained accuracy is as follows:
|
||||||
| Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None |
|
| Ollama / `mxbai-embed-large` | Ollama / `qwen3:0.6b` | 0.28 | None |
|
||||||
|
|
||||||
Note the significant degradation when very small models are used such as `qwen3:0.6b`.
|
Note the significant degradation when very small models are used such as `qwen3:0.6b`.
|
||||||
|
|
||||||
|
## Wix dataset
|
||||||
|
|
||||||
|
We also track retrieval performance on [WixQA](https://huggingface.co/datasets/Wix/WixQA),
|
||||||
|
a dataset of real customer support questions paired with curated answers from
|
||||||
|
Wix. The benchmark follows the evaluation protocol described in the
|
||||||
|
[WixQA paper](https://arxiv.org/abs/2505.08643) and gives us a view into how the
|
||||||
|
system handles conversational, product-specific support queries.
|
||||||
|
|
||||||
|
For recall, we index the reference answer passages shipped with the dataset and
|
||||||
|
run retrieval against each user question. Each sample supplies one or more
|
||||||
|
relevant passage URIs; we count how many of those URIs land inside the top *k*
|
||||||
|
retrieved documents, divide by the number of relevant passages for that query,
|
||||||
|
and average across all queries.
|
||||||
|
|
||||||
|
The results for recall using the `WixQA` dataset are as follows:
|
||||||
|
|
||||||
|
| Embedding Model | Document in top 1 | Document in top 3 | Reranker |
|
||||||
|
|----------------------------|-------------------|-------------------|------------------------|
|
||||||
|
| `qwen3-embedding` | 0.36 | 0.57 | `mxbai-rerank-base-v2` |
|
||||||
|
|
||||||
|
And for QA accuracy,
|
||||||
|
|
||||||
|
| Embedding Model | QA Model | Accuracy | Reranker |
|
||||||
|
|----------------------------|-----------|----------|------------------------|
|
||||||
|
| `qwen3-embedding` | `gpt-oss` | 0.75 | `mxbai-rerank-base-v2` |
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue