Update docs

CHANGELOG.md

@@ -1,6 +1,12 @@
 # Changelog
 ## [Unreleased]
 
+### Changed
+
+- `doctor` near-duplicate detection compares whole-document embedding centroids instead of per-chunk overlap, and no longer flags a small document contained in a larger one. The per-chunk check could take hours and tens of GB of memory on corpora with large documents. The centroid check is independent of document size and reduces each document to a centroid during the vector scan without a second copy of the matrix.
+- **Breaking:** the `doctor.duplicates` keys `containment_threshold`, `candidate_threshold`, and `twin_similarity` are replaced by a single `similarity_threshold` (default `0.97`). `min_chunks` is unchanged.
+- `doctor --duplicates-out` YAML reports `similarity` per document instead of `contained_fraction`.
+
 ## [0.62.1] - 2026-06-27
 
 ### Fixed

docs/cli.md

@@ -292,7 +292,7 @@ Check the database for consistency problems and print a pass/warn/fail report:
 haiku-rag doctor [--db /path/to/your.lancedb] [--duplicates-out groups.yaml]
 ```
 
-`--duplicates-out PATH` additionally writes the near-duplicate document groups to a YAML file (one block per group with `keep` and a list of `documents`, each carrying `document_id`, `document`, `chunks`, `contained_fraction`, and `keep_suggested`) for offline review.
+`--duplicates-out PATH` additionally writes the near-duplicate document groups to a YAML file (one block per group with `keep` and a list of `documents`, each carrying `document_id`, `document`, `chunks`, `similarity`, and `keep_suggested`) for offline review.
 
 Checks include:
 
@@ -309,7 +309,7 @@ Checks include:
 - the configured embedding identity matches the stored settings
 - no database migrations are pending
 - the vector index covers all chunks
-- near-duplicate documents (revisions sharing most of their chunks) are grouped and reported, with the largest member flagged as the likely one to keep (advisory only, never deleted; tuned via `doctor.duplicates` in config)
+- near-identical documents (by embedding-centroid similarity) are grouped and reported, with the largest member flagged as the likely one to keep (advisory only, never deleted, tuned via `doctor.duplicates` in config)
 - API keys are set for configured providers
 
 It also probes the external endpoints the config uses and reports them under a Providers section:

docs/configuration/index.md

@@ -116,9 +116,7 @@ search:
 
 doctor:
   duplicates:                    # Near-duplicate document detection (doctor command)
-    containment_threshold: 0.75  # flag a group when one doc shares >= this fraction of the smaller's chunks
-    candidate_threshold: 0.85    # centroid similarity gate for proposing candidate pairs (recall)
-    twin_similarity: 0.95        # cosine at which two chunks count as the same chunk
+    similarity_threshold: 0.97   # cosine cutoff on document embedding centroids
     min_chunks: 3                # documents with fewer chunks are excluded
 
 prompts: