danvics/haiku.rag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Use upload_large_folder for hf upload

Browse source
This commit is contained in:
Yiorgis Gozadinos 2026-05-06 13:06:28 +03:00
parent e55e3d13c8
commit cc59dc7203
No known key found for this signature in database
2 changed files with 26 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
docs/benchmarks.md
View file

@ -35,8 +35,8 @@ Available datasets:
| `repliqa` | ~30MB | | `repliqa` | ~30MB |
| `hotpotqa` | ~331MB | | `hotpotqa` | ~331MB |
| `wix` | ~511MB | | `wix` | ~511MB |
| `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~15GB | | `orb_text` — OpenRAG Bench, text embedder (`qwen3-embedding:4b`) with VLM picture descriptions baked into chunk content | ~18 GB |
| `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16GB | | `orb_multimodal` — OpenRAG Bench, multimodal embedder (`qwen3-vl-embedding-8b`); picture vectors live in the same space as text for cross-modal retrieval | ~16 GB |
After downloading, run benchmarks with `--skip-db` to use the pre-built database: After downloading, run benchmarks with `--skip-db` to use the pre-built database:

32
evaluations/evaluations/benchmark.py
View file

@ -771,9 +771,14 @@ def upload(
for the multi-GB ORB databases which would otherwise abort on any transient for the multi-GB ORB databases which would otherwise abort on any transient
network failure under plain ``upload_folder``. network failure under plain ``upload_folder``.
The local folder basename equals ``spec.db_filename`` (see ``upload_large_folder`` has no ``path_in_repo`` — it ships the contents of
``DatasetSpec.db_path``), so the folder lands at that path on the Hub. ``folder_path`` to the repo root. Stage the db under a temp parent with
hardlinks so the basename becomes the remote path, leaving everything
else at the root undisturbed.
""" """
import os
import tempfile
specs = _resolve_datasets(dataset) specs = _resolve_datasets(dataset)
api = HfApi() api = HfApi()
@ -796,12 +801,23 @@ def upload(
except Exception: except Exception:
pass pass
console.print(f"[blue]Uploading {spec.key} ({db})...[/blue]") with tempfile.TemporaryDirectory() as staging:
api.upload_large_folder( target = Path(staging) / spec.db_filename
folder_path=str(db), target.mkdir()
repo_id=HF_REPO_ID, for src in db.rglob("*"):
repo_type="dataset", if not src.is_file():
) continue
rel = src.relative_to(db)
dest = target / rel
dest.parent.mkdir(parents=True, exist_ok=True)
os.link(src, dest)
console.print(f"[blue]Uploading {spec.key} ({db})...[/blue]")
api.upload_large_folder(
folder_path=staging,
repo_id=HF_REPO_ID,
repo_type="dataset",
)
console.print(f"[green]Uploaded {spec.key} to {HF_REPO_ID}[/green]") console.print(f"[green]Uploaded {spec.key} to {HF_REPO_ID}[/green]")