diff --git a/CHANGELOG.md b/CHANGELOG.md index 282d25c3..ca912af8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,6 +70,12 @@ - Cleaner separation: LLM outputs chunk IDs, citations resolved programmatically - `Citation` fields are now required (no defaults) for type safety +### Removed + +- **BREAKING: `markdown_preprocessor` Config Option**: Removed the `processing.markdown_preprocessor` configuration option + - Use `convert()`, `chunk()`, and `embed_chunks()` primitives for custom processing pipelines + - Transform content at any stage before calling `import_document()` + ### Migration This release requires a database rebuild to populate the new DoclingDocument fields: diff --git a/docs/configuration/index.md b/docs/configuration/index.md index af7ab533..2d18adfd 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -119,7 +119,6 @@ processing: chunking_tokenizer: "Qwen/Qwen3-Embedding-0.6B" chunking_merge_peers: true chunking_use_markdown_tables: false - markdown_preprocessor: "" conversion_options: do_ocr: true force_ocr: false diff --git a/docs/configuration/processing.md b/docs/configuration/processing.md index 89847b37..b3a8fbab 100644 --- a/docs/configuration/processing.md +++ b/docs/configuration/processing.md @@ -11,7 +11,6 @@ processing: # Chunking configuration chunk_size: 256 # Maximum tokens per chunk context_chunk_radius: 0 # Context radius for chunk expansion - markdown_preprocessor: "" # Optional preprocessor script # Converter selection converter: docling-local # docling-local or docling-serve @@ -148,39 +147,6 @@ processing: context_chunk_radius: 0 ``` -### Markdown Preprocessor - -Optionally preprocess Markdown before chunking by pointing to a callable that receives and returns Markdown text. This is useful for normalizing content, stripping boilerplate, or applying custom transformations before chunk boundaries are computed. - -```yaml -processing: - # A callable path in one of these formats: - # - package.module:func - # - package.module.func - # - /abs/or/relative/path/to/file.py:func - markdown_preprocessor: my_pkg.preprocess:clean_md -``` - -!!! note - - The function signature should be `def clean_md(text: str) -> str` or `async def clean_md(text: str) -> str`. - - If the function raises or returns a non-string, haiku.rag logs a warning and proceeds without preprocessing. - - The preprocessor affects only the chunking pipeline. The stored document content remains unchanged. - -Example implementation: - -```python -# my_pkg/preprocess.py -def clean_md(text: str) -> str: - # strip HTML comments and collapse multiple blank lines - lines = [line for line in text.splitlines() if not line.strip().startswith("