Pin the chunk size the table actually splits at

The build-time assertion fired on the first real build: the chunker keeps the
whole twelve-row table in one chunk at chunk_size 256 and at 128, which would
have let a single search hit answer the whole-document question. 64 splits it.

The cost is a corpus chunked more finely than a real one. The alternative is a
longer table at a realistic chunk size, which would change what the question
asks.
This commit is contained in:
Yiorgis Gozadinos 2026-08-26 15:21:40 +03:00
parent 18e1127375
commit d2b6b78d16
No known key found for this signature in database
3 changed files with 20 additions and 6 deletions

View file

@ -19,9 +19,13 @@ lancedb:
equipment: ${HOME}/.local/share/haiku.rag/evaluations/dbs/multidb_equipment.lancedb
processing:
# Pinned: the builder asserts no single chunk holds all twelve monthly
# readings, so S3 cannot be answered from one search hit.
chunk_size: 256
# Pinned, and small deliberately. The builder asserts that no single chunk
# holds all twelve monthly readings, so S3 cannot be answered from one search
# hit; measured, the chunker keeps the whole table in one chunk at 256 and at
# 128, and splits it at 64. The cost is a corpus chunked more finely than a
# real one. The alternative is a longer table at a realistic chunk size, which
# would change what S3 asks.
chunk_size: 64
search:
# Three databases at limit 5 means RRF fuses 15 candidates to 5, so the

View file

@ -19,9 +19,13 @@ lancedb:
equipment: ${HOME}/.local/share/haiku.rag/evaluations/dbs/multidb_equipment.lancedb
processing:
# Pinned: the builder asserts no single chunk holds all twelve monthly
# readings, so S3 cannot be answered from one search hit.
chunk_size: 256
# Pinned, and small deliberately. The builder asserts that no single chunk
# holds all twelve monthly readings, so S3 cannot be answered from one search
# hit; measured, the chunker keeps the whole table in one chunk at 256 and at
# 128, and splits it at 64. The cost is a corpus chunked more finely than a
# real one. The alternative is a longer table at a realistic chunk size, which
# would change what S3 asks.
chunk_size: 64
search:
# Three databases at limit 5 means RRF fuses 15 candidates to 5, so the

View file

@ -67,6 +67,12 @@ readings**, and fails the build if one does. Without that guarantee S3 collapses
search question, and the guarantee has to survive a chunker change — so it is asserted,
not assumed.
It earned its keep on the first real build: at `chunk_size` 256 the chunker keeps the
entire table in one chunk, and so does 128. Only 64 splits it, which is what the config
pins. The cost is a corpus chunked more finely than a real one; the alternative is a
longer table at a realistic chunk size, which would change what S3 asks. Worth
revisiting if the dataset grows.
## Question families
B-family runs against both capabilities, S-family against analysis only.