Update docs
This commit is contained in:
Yiorgis Gozadinos
parent
2572125804
commit
0ea3717497
5 changed files with 298 additions and 116 deletions
73
CHANGELOG.md
73
CHANGELOG.md
|
|
@ -21,60 +21,47 @@
|
|||
- **Enhanced Search Results**: `search()` and `expand_context()` now return full provenance information
|
||||
- `SearchResult` includes `page_numbers`, `headings`, `labels`, and `doc_item_refs`
|
||||
- QA and research agents use provenance for better citations (page numbers, section headings)
|
||||
- **Inspector Visual Grounding**: New visual grounding modal in the database inspector
|
||||
- View page images with highlighted bounding boxes for chunks
|
||||
- Keyboard navigation between pages (←/→ arrows)
|
||||
- Access from both main detail view and search results
|
||||
- Requires `textual-image` dependency
|
||||
- **Visual Grounding CLI**: New `haiku-rag visualize <chunk_id>` command
|
||||
- Displays page images with highlighted bounding boxes for a chunk
|
||||
- Requires terminal with image support (iTerm2, Kitty, etc.)
|
||||
- **Processing Primitives**: New methods for custom document processing pipelines
|
||||
- `convert()` - Convert files, URLs, or text to DoclingDocument
|
||||
- `chunk()` - Chunk a DoclingDocument into Chunk objects
|
||||
- `contextualize()` - Prepend section headings to chunk content for embedding
|
||||
- `embed_chunks()` - Generate embeddings for chunks
|
||||
- **New `import_document()` Method**: Import pre-processed documents with custom chunks
|
||||
- Use when document conversion, chunking, and embedding were done externally
|
||||
- `chunks` required; `content` optional if `docling_document_json` is provided
|
||||
- When `docling_document_json` is provided without `content`, content is extracted from the DoclingDocument
|
||||
- Validates docling JSON parses correctly if provided
|
||||
- **`update_document_fields()` DoclingDocument Support**: Added `docling_document_json` and `docling_version` parameters
|
||||
- When `docling_document_json` is provided without `chunks`, content is extracted and document is rechunked
|
||||
- When `docling_document_json` is provided with `chunks`, both are stored (chunks used as-is)
|
||||
- `content` and `docling_document_json` are mutually exclusive to avoid ambiguity
|
||||
- **New `convert()` Method**: Convert files, URLs, or text to DoclingDocument
|
||||
- `client.convert(Path(...))` - convert local file
|
||||
- `client.convert("https://...")` - download and convert URL
|
||||
- `client.convert("text content")` - convert plain text
|
||||
- Supports `file://` URIs
|
||||
- **New `chunk()` Method**: Chunk a DoclingDocument into Chunk objects
|
||||
- `client.chunk(docling_doc)` - returns `list[Chunk]` without embeddings
|
||||
- **New `contextualize()` and `embed_chunks()` Utilities**: Standalone embedding utilities in `haiku.rag.embeddings`
|
||||
- `contextualize(chunks)` - prepend section headings to chunk content for better semantic search
|
||||
- `embed_chunks(chunks)` - generate embeddings for chunks, returns new Chunk objects with embeddings set
|
||||
- Use when document conversion, chunking, or embedding were done externally
|
||||
- Chunks without embeddings are automatically embedded
|
||||
- Supports `docling_document_json` for rich metadata (visual grounding, page numbers)
|
||||
- **Automatic Chunk Embedding**: `import_document()` and `update_document()` automatically embed chunks that don't have embeddings
|
||||
- Pass chunks with or without embeddings - missing embeddings are generated
|
||||
- Chunks with pre-computed embeddings are stored as-is
|
||||
- **Visual Grounding**: View page images with highlighted bounding boxes for chunks
|
||||
- Inspector modal with keyboard navigation between pages
|
||||
- CLI command: `haiku-rag visualize <chunk_id>`
|
||||
- Requires `textual-image` dependency and terminal with image support
|
||||
|
||||
### Changed
|
||||
|
||||
- **BREAKING: `create_document()` API**: Removed `chunks` parameter
|
||||
- `create_document()` now always processes content (converts, chunks, embeds)
|
||||
- Use new `import_document()` for pre-processed documents with custom chunks
|
||||
- Use `import_document()` for pre-processed documents with custom chunks
|
||||
- **BREAKING: `update_document()` API**: Unified with `update_document_fields()`
|
||||
- Old: `update_document(document)` - pass modified Document object
|
||||
- New: `update_document(document_id, content=, metadata=, chunks=, title=, docling_document_json=, docling_version=)`
|
||||
- `content` and `docling_document_json` are mutually exclusive
|
||||
- **BREAKING: Chunker Interface**: `DocumentChunker.chunk()` now returns `list[Chunk]` instead of `list[str]`
|
||||
- Chunks include structured metadata (doc_item_refs, labels, headings, page_numbers) in the `metadata` dict
|
||||
- All chunker implementations updated: `DoclingLocalChunker`, `DoclingServeChunker`
|
||||
- **Page Image Generation**: `generate_page_images=True` is now always enabled for local docling converter
|
||||
- Required for visual grounding features
|
||||
- Removed `generate_page_images` config option (docling-serve already generates page images by default)
|
||||
- **Chunk Text Storage**: Chunks now store raw text without heading contextualization
|
||||
- Section headings are prepended only at embedding time for better semantic search
|
||||
- Chunks include structured metadata (doc_item_refs, labels, headings, page_numbers)
|
||||
- **Chunk Text Storage**: Chunks store raw text; headings prepended only at embedding time
|
||||
- Stored chunk content stays clean without duplicate heading prefixes
|
||||
- Headings remain available in `ChunkMetadata` for display and citations
|
||||
- Local and serve chunkers now produce identical output
|
||||
- **QA Prompts**: Updated to use page numbers and section headings in citations when available
|
||||
- **Citation Models**: Introduced `RawSearchAnswer` for LLM output, `SearchAnswer` extends it with resolved citations
|
||||
- Cleaner separation: LLM outputs chunk IDs, citations resolved programmatically
|
||||
- `Citation` fields are now required (no defaults) for type safety
|
||||
- **Embedding Architecture**: Moved embedding generation from `ChunkRepository` to client layer
|
||||
- Repository is now a pure persistence layer
|
||||
- Client handles embedding via `_ensure_chunks_embedded()`
|
||||
- **Citation Models**: Introduced `RawSearchAnswer` for LLM output, `SearchAnswer` with resolved citations
|
||||
- **Page Image Generation**: Always enabled for local docling converter (required for visual grounding)
|
||||
|
||||
### Removed
|
||||
|
||||
- **BREAKING: `markdown_preprocessor` Config Option**: Removed the `processing.markdown_preprocessor` configuration option
|
||||
- Use `convert()`, `chunk()`, and `embed_chunks()` primitives for custom processing pipelines
|
||||
- Transform content at any stage before calling `import_document()`
|
||||
- **BREAKING: `markdown_preprocessor` Config Option**: Use processing primitives (`convert()`, `chunk()`, `embed_chunks()`) for custom pipelines
|
||||
- **`update_document_fields()`**: Merged into `update_document()`
|
||||
|
||||
### Migration
|
||||
|
||||
|
|
@ -84,7 +71,7 @@ This release requires a database rebuild to populate the new DoclingDocument fie
|
|||
haiku-rag rebuild
|
||||
```
|
||||
|
||||
Existing documents without DoclingDocument data will work but won't have provenance information. The `rebuild` command re-processes all documents to populate the new fields.
|
||||
Existing documents without DoclingDocument data will work but won't have provenance information.
|
||||
|
||||
## [0.19.6] - 2025-12-03
|
||||
|
||||
|
|
|
|||
232
docs/custom-pipelines.md
Normal file
232
docs/custom-pipelines.md
Normal file
|
|
@ -0,0 +1,232 @@
|
|||
# Custom Processing Pipelines
|
||||
|
||||
haiku.rag provides processing primitives that let you build custom document pipelines. Use these when you need control over conversion, chunking, or embedding—for example, to preprocess content, use external services, or implement custom chunking logic.
|
||||
|
||||
## Processing Primitives
|
||||
|
||||
The client exposes four primitives that can be composed into custom workflows:
|
||||
|
||||
| Primitive | Input | Output | Purpose |
|
||||
|-----------|-------|--------|---------|
|
||||
| `convert()` | file, URL, or text | `DoclingDocument` | Convert source to structured document |
|
||||
| `chunk()` | `DoclingDocument` | `list[Chunk]` | Split document into chunks |
|
||||
| `contextualize()` | `list[Chunk]` | `list[str]` | Prepare chunk text for embedding |
|
||||
| `embed_chunks()` | `list[Chunk]` | `list[Chunk]` | Generate embeddings for chunks |
|
||||
|
||||
## Basic Pipeline
|
||||
|
||||
The standard pipeline mirrors what `create_document()` does internally:
|
||||
|
||||
```python
|
||||
from haiku.rag.client import HaikuRAG
|
||||
from haiku.rag.embeddings import contextualize, embed_chunks
|
||||
|
||||
async with HaikuRAG("database.lancedb", create=True) as client:
|
||||
# 1. Convert source to DoclingDocument
|
||||
docling_doc = await client.convert("path/to/document.pdf")
|
||||
|
||||
# 2. Chunk the document
|
||||
chunks = await client.chunk(docling_doc)
|
||||
|
||||
# 3. Generate embeddings
|
||||
embedded_chunks = await embed_chunks(chunks)
|
||||
|
||||
# 4. Store the document with chunks
|
||||
doc = await client.import_document(
|
||||
chunks=embedded_chunks,
|
||||
docling_document_json=docling_doc.model_dump_json(),
|
||||
docling_version=docling_doc.version,
|
||||
uri="file:///path/to/document.pdf",
|
||||
title="My Document",
|
||||
)
|
||||
```
|
||||
|
||||
## Convert
|
||||
|
||||
`convert()` accepts files, URLs, or plain text and returns a `DoclingDocument`:
|
||||
|
||||
```python
|
||||
# From local file
|
||||
docling_doc = await client.convert("report.pdf")
|
||||
docling_doc = await client.convert(Path("/absolute/path/to/file.docx"))
|
||||
|
||||
# From URL (downloads and converts)
|
||||
docling_doc = await client.convert("https://example.com/paper.pdf")
|
||||
|
||||
# From plain text
|
||||
docling_doc = await client.convert("Your text content here")
|
||||
|
||||
# From file:// URI
|
||||
docling_doc = await client.convert("file:///path/to/document.md")
|
||||
```
|
||||
|
||||
Supported formats depend on your converter configuration (docling-local or docling-serve). Common formats include PDF, DOCX, HTML, Markdown, and images.
|
||||
|
||||
## Chunk
|
||||
|
||||
`chunk()` splits a `DoclingDocument` into `Chunk` objects with metadata:
|
||||
|
||||
```python
|
||||
chunks = await client.chunk(docling_doc)
|
||||
|
||||
for chunk in chunks:
|
||||
print(f"Order: {chunk.order}")
|
||||
print(f"Content: {chunk.content[:100]}...")
|
||||
|
||||
# Access structured metadata
|
||||
meta = chunk.get_chunk_metadata()
|
||||
print(f"Headings: {meta.headings}")
|
||||
print(f"Page numbers: {meta.page_numbers}")
|
||||
print(f"Labels: {meta.labels}")
|
||||
```
|
||||
|
||||
Chunks are returned with:
|
||||
|
||||
- `content` - The chunk text
|
||||
- `order` - Position in document (0-indexed)
|
||||
- `metadata` - Dict with `doc_item_refs`, `headings`, `labels`, `page_numbers`
|
||||
- `embedding` - `None` (not yet embedded)
|
||||
- `document_id` - `None` (not yet stored)
|
||||
|
||||
## Contextualize and Embed
|
||||
|
||||
`contextualize()` prepares chunk content for embedding by prepending section headings. This improves semantic search quality without modifying stored content:
|
||||
|
||||
```python
|
||||
from haiku.rag.embeddings import contextualize, embed_chunks
|
||||
|
||||
# Get embedding-ready text
|
||||
texts = contextualize(chunks)
|
||||
# texts[0] might be: "Chapter 1\nIntroduction\nThe actual chunk content..."
|
||||
|
||||
# Generate embeddings (returns new Chunk objects)
|
||||
embedded_chunks = await embed_chunks(chunks)
|
||||
|
||||
# Original chunks unchanged
|
||||
assert chunks[0].embedding is None
|
||||
|
||||
# New chunks have embeddings
|
||||
assert embedded_chunks[0].embedding is not None
|
||||
```
|
||||
|
||||
`embed_chunks()` returns **new** `Chunk` objects with embeddings set. The original chunks are not modified.
|
||||
|
||||
## Custom Processing Examples
|
||||
|
||||
### Preprocessing Content
|
||||
|
||||
Transform content before chunking:
|
||||
|
||||
```python
|
||||
def clean_markdown(text: str) -> str:
|
||||
"""Remove HTML comments and normalize whitespace."""
|
||||
import re
|
||||
text = re.sub(r'<!--.*?-->', '', text, flags=re.DOTALL)
|
||||
text = re.sub(r'\n{3,}', '\n\n', text)
|
||||
return text.strip()
|
||||
|
||||
async with HaikuRAG("database.lancedb", create=True) as client:
|
||||
# Convert to get raw content
|
||||
docling_doc = await client.convert("document.md")
|
||||
|
||||
# Extract and preprocess markdown
|
||||
markdown = docling_doc.export_to_markdown()
|
||||
cleaned = clean_markdown(markdown)
|
||||
|
||||
# Re-convert the cleaned content
|
||||
processed_doc = await client.convert(cleaned)
|
||||
|
||||
# Continue with standard pipeline
|
||||
chunks = await client.chunk(processed_doc)
|
||||
embedded_chunks = await embed_chunks(chunks)
|
||||
|
||||
await client.import_document(
|
||||
chunks=embedded_chunks,
|
||||
content=cleaned,
|
||||
)
|
||||
```
|
||||
|
||||
### Filtering Chunks
|
||||
|
||||
Remove unwanted chunks before embedding:
|
||||
|
||||
```python
|
||||
async with HaikuRAG("database.lancedb", create=True) as client:
|
||||
docling_doc = await client.convert("document.pdf")
|
||||
chunks = await client.chunk(docling_doc)
|
||||
|
||||
# Filter out short chunks or boilerplate
|
||||
filtered = [
|
||||
c for c in chunks
|
||||
if len(c.content) > 50
|
||||
and "copyright" not in c.content.lower()
|
||||
]
|
||||
|
||||
# Re-number the order field after filtering
|
||||
for i, chunk in enumerate(filtered):
|
||||
chunk.order = i
|
||||
|
||||
embedded_chunks = await embed_chunks(filtered)
|
||||
|
||||
await client.import_document(
|
||||
chunks=embedded_chunks,
|
||||
docling_document_json=docling_doc.model_dump_json(),
|
||||
docling_version=docling_doc.version,
|
||||
)
|
||||
```
|
||||
|
||||
### Custom Embeddings
|
||||
|
||||
Use your own embedding service:
|
||||
|
||||
```python
|
||||
async def my_embedder(texts: list[str]) -> list[list[float]]:
|
||||
"""Your custom embedding function."""
|
||||
# Call your embedding API here
|
||||
...
|
||||
|
||||
async with HaikuRAG("database.lancedb", create=True) as client:
|
||||
docling_doc = await client.convert("document.pdf")
|
||||
chunks = await client.chunk(docling_doc)
|
||||
|
||||
# Use contextualize for consistent embedding input
|
||||
texts = contextualize(chunks)
|
||||
|
||||
# Generate embeddings with your service
|
||||
embeddings = await my_embedder(texts)
|
||||
|
||||
# Create chunks with embeddings
|
||||
from haiku.rag.store.models.chunk import Chunk
|
||||
|
||||
embedded_chunks = [
|
||||
Chunk(
|
||||
content=chunk.content,
|
||||
metadata=chunk.metadata,
|
||||
order=chunk.order,
|
||||
embedding=embedding,
|
||||
)
|
||||
for chunk, embedding in zip(chunks, embeddings)
|
||||
]
|
||||
|
||||
await client.import_document(
|
||||
chunks=embedded_chunks,
|
||||
docling_document_json=docling_doc.model_dump_json(),
|
||||
docling_version=docling_doc.version,
|
||||
)
|
||||
```
|
||||
|
||||
## When to Use Custom Pipelines
|
||||
|
||||
Use the primitives when you need to:
|
||||
|
||||
- Preprocess or clean content before chunking
|
||||
- Filter or modify chunks before embedding
|
||||
- Use external embedding services
|
||||
- Implement custom chunking strategies
|
||||
- Debug or inspect intermediate processing steps
|
||||
|
||||
For standard use cases, prefer the convenience methods:
|
||||
|
||||
- `create_document()` - Create from text content
|
||||
- `create_document_from_source()` - Create from file or URL
|
||||
- `import_document()` - Store pre-processed documents with custom chunks
|
||||
|
|
@ -57,10 +57,11 @@ haiku-rag ask "Who is the author of haiku.rag?"
|
|||
- [Installation](installation.md) - Install haiku.rag with different providers
|
||||
- [Configuration](configuration/index.md) - Environment variables and settings
|
||||
- [CLI](cli.md) - Command line interface usage
|
||||
- [Python](python.md) - Python API reference
|
||||
- [Custom Pipelines](custom-pipelines.md) - Build custom processing workflows
|
||||
- [Agents](agents.md) - QA agent and multi-agent research
|
||||
- [Server](server.md) - File monitoring and server mode
|
||||
- [MCP](mcp.md) - Model Context Protocol integration
|
||||
- [Python](python.md) - Python API reference
|
||||
- [Agents](agents.md) - QA agent and multi-agent research
|
||||
- [Remote processing](remote-processing.md) - Remote document processing with docling-serve
|
||||
|
||||
## License
|
||||
|
|
|
|||
103
docs/python.md
103
docs/python.md
|
|
@ -52,12 +52,12 @@ doc = await client.create_document_from_source(
|
|||
|
||||
### Importing Pre-Processed Documents
|
||||
|
||||
If you process documents externally (conversion, chunking, embedding), use `import_document()` to store them:
|
||||
If you process documents externally or need custom processing, use `import_document()`:
|
||||
|
||||
```python
|
||||
from haiku.rag.store.models.chunk import Chunk
|
||||
|
||||
# Create chunks with optional embeddings
|
||||
# Create chunks (embeddings optional - will be generated if missing)
|
||||
chunks = [
|
||||
Chunk(
|
||||
content="This is the first chunk",
|
||||
|
|
@ -67,7 +67,7 @@ chunks = [
|
|||
Chunk(
|
||||
content="This is the second chunk",
|
||||
metadata={"section": "body"},
|
||||
embedding=[0.1] * 1024, # Pre-computed embedding
|
||||
embedding=[0.1] * 1024, # Optional: pre-computed embedding
|
||||
order=1,
|
||||
),
|
||||
]
|
||||
|
|
@ -82,18 +82,9 @@ doc = await client.import_document(
|
|||
)
|
||||
```
|
||||
|
||||
If you also have a DoclingDocument from your processing pipeline, include it for rich metadata support (visual grounding, page numbers, section headings). When providing a DoclingDocument, you can omit `content` - it will be extracted automatically:
|
||||
With a DoclingDocument for rich metadata (visual grounding, page numbers):
|
||||
|
||||
```python
|
||||
# With explicit content
|
||||
doc = await client.import_document(
|
||||
chunks=chunks,
|
||||
content="Full document content",
|
||||
docling_document_json=docling_doc.model_dump_json(),
|
||||
docling_version=docling_doc.version,
|
||||
)
|
||||
|
||||
# Or let content be extracted from DoclingDocument
|
||||
doc = await client.import_document(
|
||||
chunks=chunks,
|
||||
docling_document_json=docling_doc.model_dump_json(),
|
||||
|
|
@ -102,7 +93,9 @@ doc = await client.import_document(
|
|||
```
|
||||
|
||||
!!! note
|
||||
Either `content` or `docling_document_json` must be provided. When `docling_document_json` is provided without `content`, the content is extracted from the DoclingDocument.
|
||||
Either `content` or `docling_document_json` must be provided. When `docling_document_json` is provided without `content`, the content is extracted automatically.
|
||||
|
||||
See [Custom Processing Pipelines](custom-pipelines.md) for building pipelines with `convert()`, `chunk()`, and `embed_chunks()`.
|
||||
|
||||
### Retrieving Documents
|
||||
|
||||
|
|
@ -138,71 +131,40 @@ docs = await client.list_documents(
|
|||
|
||||
### Updating Documents
|
||||
|
||||
Update entire document:
|
||||
```python
|
||||
doc.content = "Updated content"
|
||||
await client.update_document(doc)
|
||||
```
|
||||
# Update content (triggers re-chunking)
|
||||
await client.update_document(document_id=doc.id, content="New content")
|
||||
|
||||
Update specific fields:
|
||||
```python
|
||||
# Update only content (triggers re-chunking)
|
||||
await client.update_document_fields(
|
||||
document_id=doc.id,
|
||||
content="New content"
|
||||
)
|
||||
|
||||
# Update only metadata (no re-chunking)
|
||||
await client.update_document_fields(
|
||||
# Update metadata only (no re-chunking)
|
||||
await client.update_document(
|
||||
document_id=doc.id,
|
||||
metadata={"version": "2.0", "updated_by": "admin"}
|
||||
)
|
||||
|
||||
# Update only title (no re-chunking)
|
||||
await client.update_document_fields(
|
||||
document_id=doc.id,
|
||||
title="New Title"
|
||||
)
|
||||
# Update title only (no re-chunking)
|
||||
await client.update_document(document_id=doc.id, title="New Title")
|
||||
|
||||
# Update multiple fields at once
|
||||
await client.update_document_fields(
|
||||
await client.update_document(
|
||||
document_id=doc.id,
|
||||
content="New content",
|
||||
title="Updated Title",
|
||||
metadata={"status": "final"}
|
||||
)
|
||||
|
||||
# Use custom chunks instead of auto-generation
|
||||
# Use custom chunks (embeddings optional - will be generated if missing)
|
||||
custom_chunks = [
|
||||
Chunk(content="Custom chunk 1"),
|
||||
Chunk(content="Custom chunk 2"),
|
||||
Chunk(content="Custom chunk 2", embedding=[...]), # Pre-computed embedding
|
||||
]
|
||||
await client.update_document_fields(
|
||||
document_id=doc.id,
|
||||
chunks=custom_chunks
|
||||
)
|
||||
|
||||
# Update with DoclingDocument for rich metadata (extracts content and rechunks)
|
||||
await client.update_document_fields(
|
||||
document_id=doc.id,
|
||||
docling_document_json=docling_doc.model_dump_json(),
|
||||
docling_version=docling_doc.version,
|
||||
)
|
||||
|
||||
# Update with DoclingDocument and custom chunks (stores both, uses chunks as-is)
|
||||
await client.update_document_fields(
|
||||
document_id=doc.id,
|
||||
chunks=custom_chunks,
|
||||
docling_document_json=docling_doc.model_dump_json(),
|
||||
docling_version=docling_doc.version,
|
||||
)
|
||||
await client.update_document(document_id=doc.id, chunks=custom_chunks)
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
|
||||
- Updates to only `metadata` or `title` skip re-chunking for efficiency
|
||||
- Updates to `content`, `chunks`, or `docling_document_json` will regenerate or replace chunks
|
||||
- `content` and `docling_document_json` are mutually exclusive - provide one or the other
|
||||
- Updates to only `metadata` or `title` skip re-chunking
|
||||
- Updates to `content` trigger re-chunking and re-embedding
|
||||
- Custom `chunks` with embeddings are stored as-is; missing embeddings are generated automatically
|
||||
|
||||
### Deleting Documents
|
||||
|
||||
|
|
@ -259,10 +221,10 @@ The search method performs native hybrid search (vector + full-text) using Lance
|
|||
Basic hybrid search (default):
|
||||
```python
|
||||
results = await client.search("machine learning algorithms", limit=5)
|
||||
for chunk, score in results:
|
||||
print(f"Score: {score:.3f}")
|
||||
print(f"Content: {chunk.content}")
|
||||
print(f"Document ID: {chunk.document_id}")
|
||||
for result in results:
|
||||
print(f"Score: {result.score:.3f}")
|
||||
print(f"Content: {result.content}")
|
||||
print(f"Document ID: {result.document_id}")
|
||||
```
|
||||
|
||||
Search with different search types:
|
||||
|
|
@ -289,13 +251,12 @@ results = await client.search(
|
|||
)
|
||||
|
||||
# Process results
|
||||
for chunk, relevance_score in results:
|
||||
print(f"Relevance: {relevance_score:.3f}")
|
||||
print(f"Content: {chunk.content}")
|
||||
print(f"From document: {chunk.document_id}")
|
||||
print(f"Document URI: {chunk.document_uri}")
|
||||
print(f"Document Title: {chunk.document_title}") # when available
|
||||
print(f"Document metadata: {chunk.document_meta}")
|
||||
for result in results:
|
||||
print(f"Relevance: {result.score:.3f}")
|
||||
print(f"Content: {result.content}")
|
||||
print(f"From document: {result.document_id}")
|
||||
print(f"Document URI: {result.document_uri}")
|
||||
print(f"Document Title: {result.document_title}") # when available
|
||||
```
|
||||
|
||||
### Filtering Search Results
|
||||
|
|
@ -355,8 +316,8 @@ expanded_results = await client.expand_context(search_results)
|
|||
expanded_results = await client.expand_context(search_results, radius=2)
|
||||
|
||||
# The expanded results contain chunks with combined content from adjacent chunks
|
||||
for chunk, score in expanded_results:
|
||||
print(f"Expanded content: {chunk.content}") # Now includes before/after chunks
|
||||
for result in expanded_results:
|
||||
print(f"Expanded content: {result.content}") # Now includes before/after chunks
|
||||
```
|
||||
|
||||
**Smart Merging**: When expanded chunks overlap or are adjacent within the same document, they are automatically merged into single chunks with continuous content. This eliminates duplication and provides coherent text blocks. The merged chunk uses the highest relevance score from the original chunks.
|
||||
|
|
|
|||
|
|
@ -67,6 +67,7 @@ nav:
|
|||
- Storage: configuration/storage.md
|
||||
- CLI: cli.md
|
||||
- Python: python.md
|
||||
- Custom Pipelines: custom-pipelines.md
|
||||
- Agents: agents.md
|
||||
- Server: server.md
|
||||
- Remote processing: remote-processing.md
|
||||
|
|
|
|||
Loading…
Reference in a new issue