From 6c04dd5881d868b60458c020c676b94311fa26c6 Mon Sep 17 00:00:00 2001 From: Yiorgis Gozadinos Date: Thu, 11 Jun 2026 12:37:32 +0300 Subject: [PATCH] docs --- docs/cli.md | 6 +++--- docs/configuration/storage.md | 2 +- docs/python.md | 8 ++++---- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 7fe283ad..037d544b 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -275,9 +275,9 @@ Shows: - path to the database - stored haiku.rag version (from settings) - embeddings provider/model and vector dimension -- number of documents and chunks (with storage sizes) +- per-table row counts and storage sizes (documents, document_meta, chunks, document_items) - vector index status (exists/not created, indexed/unindexed chunks) -- table versions per table (documents, chunks) +- table versions per table (documents, document_meta, chunks) At the end, a separate "Versions" section lists runtime package versions: - haiku.rag @@ -401,7 +401,7 @@ Reduce disk usage by optimizing and pruning old table versions across all tables haiku-rag vacuum ``` -**Automatic Cleanup:** Vacuum runs automatically in the background after document operations. By default, it removes versions older than 1 day (configurable via `storage.vacuum_retention_seconds`), preserving recent versions for concurrent connections. Manual vacuum can be useful for cleanup after bulk operations or to free disk space immediately. +**Automatic Cleanup:** Vacuum runs automatically in the background after document operations, throttled to at most once every 5 minutes so sustained ingestion does not trigger continuous compaction (a final vacuum runs when the client closes). By default, it removes versions older than 1 day (configurable via `storage.vacuum_retention_seconds`), preserving recent versions for concurrent connections. Manual vacuum can be useful for cleanup after bulk operations or to free disk space immediately. ## MCP Server diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md index 380fdc07..40e4d66a 100644 --- a/docs/configuration/storage.md +++ b/docs/configuration/storage.md @@ -12,7 +12,7 @@ storage: ``` - **data_dir**: Directory for local database storage. When empty, uses platform-specific default locations -- **auto_vacuum**: When enabled (default), automatically runs vacuum after document create/update operations and database rebuilds. Set to `false` to disable automatic vacuuming and rely on manual `haiku-rag vacuum` commands only. Disabling can help avoid potential crashes in high-concurrency scenarios +- **auto_vacuum**: When enabled (default), automatically runs vacuum after document create/update/delete operations and database rebuilds. Background vacuums are throttled to at most one every 5 minutes, so sustained ingestion does not trigger continuous compaction, and a final vacuum runs when the client closes. Set to `false` to disable automatic vacuuming and rely on manual `haiku-rag vacuum` commands only. Disabling can help avoid potential crashes in high-concurrency scenarios - **vacuum_retention_seconds**: When vacuum runs, old table versions older than this threshold are removed. Default: 86400 seconds (1 day). Set to 0 for aggressive cleanup (removes all old versions immediately) !!! warning "Vacuum Retention Threshold" diff --git a/docs/python.md b/docs/python.md index c4cc9f0a..cc028fb1 100644 --- a/docs/python.md +++ b/docs/python.md @@ -396,7 +396,7 @@ The `docling_document` provides rich metadata for visual grounding, page numbers ### Batch Import -Each `create_document*` / `import_document` call writes new versions of the `documents`, `chunks`, and `document_items` tables. Ingesting many documents in a loop therefore creates a table version per document. Use `import_documents()` to write the whole batch in a single version per table: +Each `create_document*` / `import_document` call writes new versions of the `documents`, `document_meta`, `chunks`, and `document_items` tables. Ingesting many documents in a loop therefore creates a table version per document. Use `import_documents()` to write the whole batch in a single version per table: ```python from haiku.rag.client import DocumentImport @@ -487,8 +487,8 @@ See [Automatic Title Generation](configuration/processing.md#automatic-title-gen ### Atomic Writes and Rollback -Document create and update operations take a snapshot of table versions before any write and automatically roll back to that snapshot if something fails (for example, during chunking or embedding). This restores both the `documents` and `chunks` tables to their pre‑operation state using LanceDB’s table versioning. +Document create, update, and delete operations take a snapshot of table versions before any write and automatically roll back to that snapshot if something fails (for example, during chunking or embedding). This restores the `documents`, `document_meta`, `chunks`, and `document_items` tables to their pre‑operation state using LanceDB’s table versioning. These writes are serialized under a single lock, so the rollback is safe under concurrent ingester workers. -- Applies to: `create_document(...)`, `create_document_from_source(...)`, `update_document(...)`, and internal rebuild/update flows. -- Scope: Both document rows and all associated chunks are rolled back together. +- Applies to: `create_document(...)`, `create_document_from_source(...)`, `update_document(...)`, `delete_document(...)` (including the `parent_uri` cascade), and internal rebuild/update flows. +- Scope: Document rows, their mutable attributes, and all associated chunks and items are rolled back together. - Vacuum: Running `vacuum()` later prunes old versions for disk efficiency. Rollbacks occur immediately during the failing operation and are not impacted.