581ddf4f41
Vacuum compacts documents and document_items through lance directly with a fragment target derived from storage.compaction_target_bytes (default 2 GiB), instead of AsyncTable.optimize, whose default 1,048,576-row fragment target re-merges the whole table on every pass and peaks at 5-6x the table's bytes. The target is sized from the widest fragment's bytes per row, so peak memory is bounded by the target instead of the corpus. lancedb's async API exposes no compaction options (lancedb/lancedb#2325), hence the pylance dependency. |
||
|---|---|---|
| .. | ||
| index.md | ||
| processing.md | ||
| prompts.md | ||
| providers.md | ||
| qa.md | ||
| storage.md | ||