Merge pull request #55 from ggozad/feat/vacuum

Vacuum operation, optimise and remove old lanceDB versions

docs/cli.md

@@ -2,22 +2,17 @@
 
 The `haiku-rag` CLI provides complete document management functionality.
 
-## Shell Autocompletion
+!!! note
+    All commands support:
 
-Enable shell autocompletion for faster, error‑free usage.
+    - `--db` - Specify custom database path
+    - `-h` - Show help for specific command
 
-- Temporary (current shell only):
+    Example:
-  ```bash
+    ```bash
-  eval "$(haiku-rag --show-completion)"
+    haiku-rag list --db /path/to/custom.db
-  ```
+    haiku-rag add -h
-- Permanent installation:
+    ```
-  ```bash
-  haiku-rag --install-completion
-  ```
-
-What’s completed:
-- `get` and `delete`/`rm`: Document IDs from the selected database (respects `--db`).
-- `add-src`: Local filesystem paths (URLs can still be typed manually).
 
 ## Document Management
 
@@ -40,6 +35,10 @@ haiku-rag add-src /path/to/document.pdf
 haiku-rag add-src https://example.com/article.html
 ```
 
+!!! note
+    As you add documents to `haiku.rag` the database keeps growing. By default, `lanceDB` supports versioning
+    of your data. You can optimize and compact the database by running the [vaccum](#vacuum-optimize-and-cleanup) command.
+
 ### Get Document
 
 ```bash
@@ -55,33 +54,8 @@ haiku-rag delete <TAB>
 haiku-rag rm <TAB>       # alias
 ```
 
-### Rebuild Database
-
-Rebuild the database by deleting all chunks & embeddings and re-indexing all documents:
-
-```bash
-haiku-rag rebuild
-```
-
 Use this when you want to change things like the embedding model or chunk size for example.
 
-## Migration
-
-### Migrate from SQLite to LanceDB
-
-Migrate an existing SQLite database to LanceDB:
-
-```bash
-haiku-rag migrate /path/to/old_database.sqlite
-```
-
-This will:
-- Read all documents, chunks, embeddings, and settings from the SQLite database
-- Create a new LanceDB database with the same data in the same directory
-- Optimize the new database for best performance
-
-The original SQLite database remains unchanged, so you can safely migrate without risk of data loss.
-
 ## Search
 
 Basic search:
@@ -108,13 +82,6 @@ haiku-rag ask "Who is the author of haiku.rag?" --cite
 
 The QA agent will search your documents for relevant information and provide a comprehensive answer. With `--cite`, responses include citations showing which documents were used.
 
-## Configuration
-
-View current configuration settings:
-```bash
-haiku-rag settings
-```
-
 ## Server
 
 Start the MCP server:
@@ -129,14 +96,62 @@ haiku-rag serve --stdio
 haiku-rag serve --sse
 ```
 
-## Options
+## Settings
 
-All commands support:
+View current configuration settings:
-- `--db` - Specify custom database path
-- `-h` - Show help for specific command
-
-Example:
 ```bash
-haiku-rag list --db /path/to/custom.db
+haiku-rag settings
-haiku-rag add -h
 ```
+
+## Maintenance
+
+### Vacuum (Optimize and Cleanup)
+
+Reduce disk usage by optimizing and pruning old table versions across all tables:
+
+```bash
+haiku-rag vacuum
+```
+
+### Rebuild Database
+
+Rebuild the database by deleting all chunks & embeddings and re-indexing all documents. This is useful
+when want to switch embeddings provider or model:
+
+```bash
+haiku-rag rebuild
+```
+
+## Migration
+
+### Migrate from SQLite to LanceDB
+
+Migrate an existing SQLite database to LanceDB:
+
+```bash
+haiku-rag migrate /path/to/old_database.sqlite
+```
+
+This will:
+- Read all documents, chunks, embeddings, and settings from the SQLite database
+- Create a new LanceDB database with the same data in the same directory
+- Optimize the new database for best performance
+
+The original SQLite database remains unchanged, so you can safely migrate without risk of data loss.
+
+## Shell Autocompletion
+
+Enable shell autocompletion for faster, error‑free usage.
+
+- Temporary (current shell only):
+  ```bash
+  eval "$(haiku-rag --show-completion)"
+  ```
+- Permanent installation:
+  ```bash
+  haiku-rag --install-completion
+  ```
+
+What’s completed:
+- `get` and `delete`/`rm`: Document IDs from the selected database (respects `--db`).
+- `add-src`: Local filesystem paths (URLs can still be typed manually).

docs/index.md

@@ -52,7 +52,6 @@ haiku-rag migrate old_database.sqlite  # Migrate from SQLite
 - [Installation](installation.md) - Install haiku.rag with different providers
 - [Configuration](configuration.md) - Environment variables and settings
 - [CLI](cli.md) - Command line interface usage
-- [Question Answering](qa.md) - QA agents and natural language queries
 - [Server](server.md) - File monitoring and server mode
 - [MCP](mcp.md) - Model Context Protocol integration
 - [Python](python.md) - Python API reference

docs/python.md

@@ -99,6 +99,16 @@ async for doc_id in client.rebuild_database():
     print(f"Processed document {doc_id}")
 ```
 
+## Maintenance
+
+Run maintenance to optimize storage and prune old table versions:
+
+```python
+await client.vacuum()
+```
+
+This compacts tables and removes historical versions to keep disk usage in check. It’s safe to run anytime, for example after bulk imports or periodically in long‑running apps.
+
 ## Searching Documents
 
 The search method performs native hybrid search (vector + full-text) using LanceDB with optional reranking for improved relevance:

src/haiku/rag/app.py

@@ -102,6 +102,15 @@ class HaikuRAGApp:
             except Exception as e:
                 self.console.print(f"[red]Error rebuilding database: {e}[/red]")
 
+    async def vacuum(self):
+        """Run database maintenance: optimize and cleanup table history."""
+        try:
+            async with HaikuRAG(db_path=self.db_path, skip_validation=True) as client:
+                await client.vacuum()
+            self.console.print("[b]Vacuum completed successfully.[/b]")
+        except Exception as e:
+            self.console.print(f"[red]Error during vacuum: {e}[/red]")
+
     def show_settings(self):
         """Display current configuration settings."""
         self.console.print("[bold]haiku.rag configuration[/bold]")

src/haiku/rag/cli.py

@@ -256,6 +256,18 @@ def rebuild(
     asyncio.run(app.rebuild())
 
 
+@cli.command("vacuum", help="Optimize and clean up all tables to reduce disk usage")
+def vacuum(
+    db: Path = typer.Option(
+        Config.DEFAULT_DATA_DIR / "haiku.rag.lancedb",
+        "--db",
+        help="Path to the LanceDB database file",
+    ),
+):
+    app = HaikuRAGApp(db_path=db)
+    asyncio.run(app.vacuum())
+
+
 @cli.command(
     "serve", help="Start the haiku.rag MCP server (by default in streamable HTTP mode)"
 )

src/haiku/rag/client.py

@@ -550,6 +550,16 @@ class HaikuRAG:
                 )
                 yield doc.id
 
+        # Final maintenance: centralized vacuum to curb disk usage
+        try:
+            self.store.vacuum()
+        except Exception:
+            pass
+
+    async def vacuum(self) -> None:
+        """Optimize and clean up old versions across all tables."""
+        self.store.vacuum()
+
     def close(self):
         """Close the underlying store connection."""
         self.store.close()

src/haiku/rag/logging.py

@@ -1,4 +1,5 @@
 import logging
+import warnings
 
 from rich.console import Console
 from rich.logging import RichHandler
@@ -50,4 +51,6 @@ def configure_cli_logging(level: int = logging.INFO) -> logging.Logger:
     logger = get_logger()
     logger.setLevel(level)
     logger.propagate = False
+
+    warnings.filterwarnings("ignore")
     return logger

src/haiku/rag/migration.py

@@ -47,7 +47,7 @@ class SQLiteToLanceDBMigrator:
 
             # Load the sqlite-vec extension
             try:
-                import sqlite_vec
+                import sqlite_vec  # type: ignore
 
                 sqlite_conn.enable_load_extension(True)
                 sqlite_vec.load(sqlite_conn)
@@ -91,10 +91,10 @@ class SQLiteToLanceDBMigrator:
 
             sqlite_conn.close()
 
-            # Optimize the chunks table after migration
+            # Optimize and cleanup using centralized vacuum
             self.console.print("[blue]Optimizing LanceDB...[/blue]")
             try:
-                lance_store.chunks_table.optimize()
+                lance_store.vacuum()
                 self.console.print("[green]✅ Optimization completed[/green]")
             except Exception as e:
                 self.console.print(

src/haiku/rag/store/engine.py

@@ -1,5 +1,6 @@
 import json
 import logging
+from datetime import timedelta
 from importlib import metadata
 from pathlib import Path
 from uuid import uuid4
@@ -62,6 +63,15 @@ class Store:
         if not skip_validation:
             self._validate_configuration()
 
+    def vacuum(self) -> None:
+        """Optimize and clean up old versions across all tables to reduce disk usage."""
+        if self._has_cloud_config() and str(Config.LANCEDB_URI).startswith("db://"):
+            return
+
+        # Perform maintenance per table using optimize() with cleanup_older_than 0
+        for table in [self.documents_table, self.chunks_table, self.settings_table]:
+            table.optimize(cleanup_older_than=timedelta(0))
+
     def _connect_to_lancedb(self, db_path: Path):
         """Establish connection to LanceDB (local, cloud, or object storage)."""
         # Check if we have cloud configuration
@@ -159,16 +169,18 @@ class Store:
             self.settings_table.search().limit(1).to_pydantic(SettingsRecord)
         )
         if settings_records:
-            settings = (
+            # Only write if version actually changes to avoid creating new table versions
+            current = (
                 json.loads(settings_records[0].settings)
                 if settings_records[0].settings
                 else {}
             )
-            settings["version"] = version
+            if current.get("version") != version:
-            # Update the record
+                current["version"] = version
-            self.settings_table.update(
+                self.settings_table.update(
-                where="id = 'settings'", values={"settings": json.dumps(settings)}
+                    where="id = 'settings'",
-            )
+                    values={"settings": json.dumps(current)},
+                )
         else:
             # Create new settings record
             settings_data = Config.model_dump(mode="json")

src/haiku/rag/store/repositories/settings.py

@@ -84,10 +84,15 @@ class SettingsRepository:
         )
 
         if existing:
-            # Update existing settings
+            # Only update when configuration actually changed to avoid needless new versions
-            self.store.settings_table.update(
+            existing_payload = (
-                where="id = 'settings'", values={"settings": json.dumps(current_config)}
+                json.loads(existing[0].settings) if existing[0].settings else {}
             )
+            if existing_payload != current_config:
+                self.store.settings_table.update(
+                    where="id = 'settings'",
+                    values={"settings": json.dumps(current_config)},
+                )
         else:
             # Create new settings
             settings_record = SettingsRecord(

tests/generate_benchmark_db.py

@@ -8,8 +8,10 @@ from rich.progress import Progress
 
 from haiku.rag import logging  # noqa
 from haiku.rag.client import HaikuRAG
+from haiku.rag.logging import configure_cli_logging
 from haiku.rag.qa import get_qa_agent
 
+configure_cli_logging()
 console = Console()
 
 db_path = Path(__file__).parent / "data" / "benchmark.lancedb"
@@ -35,6 +37,7 @@ async def populate_db():
                     uri=uri,
                 )
                 progress.advance(task)
+            rag.store.vacuum()
 
 
 async def run_match_benchmark():