diff --git a/CHANGELOG.md b/CHANGELOG.md index 65a52dec..4b7498fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,12 @@ # Changelog ## [Unreleased] +### Changed + +- `doctor` near-duplicate detection compares whole-document embedding centroids instead of per-chunk overlap, and no longer flags a small document contained in a larger one. The per-chunk check could take hours and tens of GB of memory on corpora with large documents. The centroid check is independent of document size and reduces each document to a centroid during the vector scan without a second copy of the matrix. +- **Breaking:** the `doctor.duplicates` keys `containment_threshold`, `candidate_threshold`, and `twin_similarity` are replaced by a single `similarity_threshold` (default `0.97`). `min_chunks` is unchanged. +- `doctor --duplicates-out` YAML reports `similarity` per document instead of `contained_fraction`. + ## [0.62.1] - 2026-06-27 ### Fixed diff --git a/docs/cli.md b/docs/cli.md index d432c94f..58ce3360 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -292,7 +292,7 @@ Check the database for consistency problems and print a pass/warn/fail report: haiku-rag doctor [--db /path/to/your.lancedb] [--duplicates-out groups.yaml] ``` -`--duplicates-out PATH` additionally writes the near-duplicate document groups to a YAML file (one block per group with `keep` and a list of `documents`, each carrying `document_id`, `document`, `chunks`, `contained_fraction`, and `keep_suggested`) for offline review. +`--duplicates-out PATH` additionally writes the near-duplicate document groups to a YAML file (one block per group with `keep` and a list of `documents`, each carrying `document_id`, `document`, `chunks`, `similarity`, and `keep_suggested`) for offline review. Checks include: @@ -309,7 +309,7 @@ Checks include: - the configured embedding identity matches the stored settings - no database migrations are pending - the vector index covers all chunks -- near-duplicate documents (revisions sharing most of their chunks) are grouped and reported, with the largest member flagged as the likely one to keep (advisory only, never deleted; tuned via `doctor.duplicates` in config) +- near-identical documents (by embedding-centroid similarity) are grouped and reported, with the largest member flagged as the likely one to keep (advisory only, never deleted, tuned via `doctor.duplicates` in config) - API keys are set for configured providers It also probes the external endpoints the config uses and reports them under a Providers section: diff --git a/docs/configuration/index.md b/docs/configuration/index.md index 4c3c558d..2c76fd66 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -116,9 +116,7 @@ search: doctor: duplicates: # Near-duplicate document detection (doctor command) - containment_threshold: 0.75 # flag a group when one doc shares >= this fraction of the smaller's chunks - candidate_threshold: 0.85 # centroid similarity gate for proposing candidate pairs (recall) - twin_similarity: 0.95 # cosine at which two chunks count as the same chunk + similarity_threshold: 0.97 # cosine cutoff on document embedding centroids min_chunks: 3 # documents with fewer chunks are excluded prompts: