Merge commit '20beb23a389778461e8280f1442ea196e52182e3' into dockerise
This commit is contained in:
commit
7609ac911b
47 changed files with 3065 additions and 614 deletions
28
.github/workflows/build-docs.yml
vendored
Normal file
28
.github/workflows/build-docs.yml
vendored
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
name: build-docs
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- main
|
||||||
|
permissions:
|
||||||
|
contents: write
|
||||||
|
jobs:
|
||||||
|
deploy:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Configure Git Credentials
|
||||||
|
run: |
|
||||||
|
git config user.name github-actions[bot]
|
||||||
|
git config user.email 41898282+github-actions[bot]@users.noreply.github.com
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: 3.x
|
||||||
|
- run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
|
||||||
|
- uses: actions/cache@v4
|
||||||
|
with:
|
||||||
|
key: mkdocs-material-${{ env.cache_id }}
|
||||||
|
path: .cache
|
||||||
|
restore-keys: |
|
||||||
|
mkdocs-material-
|
||||||
|
- run: pip install mkdocs-material
|
||||||
|
- run: mkdocs gh-deploy --force
|
||||||
|
|
@ -20,3 +20,13 @@ repos:
|
||||||
rev: v1.1.399
|
rev: v1.1.399
|
||||||
hooks:
|
hooks:
|
||||||
- id: pyright
|
- id: pyright
|
||||||
|
|
||||||
|
- repo: https://github.com/RodrigoGonzalez/check-mkdocs
|
||||||
|
rev: v1.2.0
|
||||||
|
hooks:
|
||||||
|
- id: check-mkdocs
|
||||||
|
name: check-mkdocs
|
||||||
|
args: ["--config", "mkdocs.yml"] # Optional, mkdocs.yml is the default
|
||||||
|
# If you have additional plugins or libraries that are not included in
|
||||||
|
# check-mkdocs, add them here
|
||||||
|
additional_dependencies: ["mkdocs-material"]
|
||||||
|
|
|
||||||
238
README.md
238
README.md
|
|
@ -1,198 +1,84 @@
|
||||||
# Haiku SQLite RAG
|
# Haiku SQLite RAG
|
||||||
|
|
||||||
A Retrieval-Augmented Generation (RAG) library on SQLite.
|
Retrieval-Augmented Generation (RAG) library on SQLite.
|
||||||
|
|
||||||
|
`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work on SQLite alone without the need for external vector databases. It uses [sqlite-vec](https://github.com/asg017/sqlite-vec) for storing the embeddings and performs semantic (vector) search as well as full-text search combined through Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
- **Local SQLite**: No need to run additional servers
|
|
||||||
- **Support for various embedding providers**: You can use Ollama, VoyageAI, OpenAI or add your own
|
|
||||||
- **Hybrid Search**: Vector search using `sqlite-vec` combined with full-text search `FTS5`, using Reciprocal Rank Fusion
|
|
||||||
- **File monitoring** when run as a server automatically indexing your files
|
|
||||||
- **Extended file format Support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, audio and more. Or add a url!
|
|
||||||
- **MCP server** Exposes functionality as MCP tools.
|
|
||||||
- **CLI commands** Access all functionality from your terminal
|
|
||||||
- **Python client** Call `haiku.rag` from your own python applications.
|
|
||||||
|
|
||||||
## Installation
|
- **Local SQLite**: No external servers required
|
||||||
|
- **Multiple embedding providers**: Ollama, VoyageAI, OpenAI
|
||||||
|
- **Multiple QA providers**: Ollama, OpenAI, Anthropic
|
||||||
|
- **Hybrid search**: Vector + full-text search with Reciprocal Rank Fusion
|
||||||
|
- **Question answering**: Built-in QA agents on your documents
|
||||||
|
- **File monitoring**: Auto-index files when run as server
|
||||||
|
- **40+ file formats**: PDF, DOCX, HTML, Markdown, audio, URLs
|
||||||
|
- **MCP server**: Expose as tools for AI assistants
|
||||||
|
- **CLI & Python API**: Use from command line or Python
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# Install
|
||||||
uv pip install haiku.rag
|
uv pip install haiku.rag
|
||||||
|
|
||||||
|
# Add documents
|
||||||
|
haiku-rag add "Your content here"
|
||||||
|
haiku-rag add-src document.pdf
|
||||||
|
|
||||||
|
# Search
|
||||||
|
haiku-rag search "query"
|
||||||
|
|
||||||
|
# Ask questions
|
||||||
|
haiku-rag ask "Who is the author of haiku.rag?"
|
||||||
|
|
||||||
|
# Rebuild database (re-chunk and re-embed all documents)
|
||||||
|
haiku-rag rebuild
|
||||||
|
|
||||||
|
# Start server with file monitoring
|
||||||
|
export MONITOR_DIRECTORIES="/path/to/docs"
|
||||||
|
haiku-rag serve
|
||||||
```
|
```
|
||||||
|
|
||||||
By default Ollama (with the `mxbai-embed-large` model) is used for the embeddings.
|
## Python Usage
|
||||||
For other providers use:
|
|
||||||
|
|
||||||
- **VoyageAI**: `uv pip install haiku.rag --extra voyageai`
|
|
||||||
- **OpenAI**: `uv pip install haiku.rag --extra openai`
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
You can set the directories to monitor using the `MONITOR_DIRECTORIES` environment variable (as comma separated values) :
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Monitor single directory
|
|
||||||
export MONITOR_DIRECTORIES="/path/to/documents,/another_path/to/documents"
|
|
||||||
```
|
|
||||||
|
|
||||||
If you want to use an alternative embeddings provider (Ollama being the default) you will need to set the provider details through environment variables:
|
|
||||||
|
|
||||||
By default:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
EMBEDDINGS_PROVIDER="ollama"
|
|
||||||
EMBEDDINGS_MODEL="mxbai-embed-large" # or any other model
|
|
||||||
EMBEDDINGS_VECTOR_DIM=1024
|
|
||||||
```
|
|
||||||
|
|
||||||
For VoyageAI:
|
|
||||||
```bash
|
|
||||||
EMBEDDINGS_PROVIDER="voyageai"
|
|
||||||
EMBEDDINGS_MODEL="voyage-3.5" # or any other model
|
|
||||||
EMBEDDINGS_VECTOR_DIM=1024
|
|
||||||
VOYAGE_API_KEY="your-api-key"
|
|
||||||
```
|
|
||||||
|
|
||||||
For OpenAI:
|
|
||||||
```bash
|
|
||||||
EMBEDDINGS_PROVIDER="openai"
|
|
||||||
EMBEDDINGS_MODEL="text-embedding-3-small" # or text-embedding-3-large
|
|
||||||
EMBEDDINGS_VECTOR_DIM=1536
|
|
||||||
OPENAI_API_KEY="your-api-key"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Command Line Interface
|
|
||||||
|
|
||||||
`haiku.rag` includes a CLI application for managing documents and performing searches from the command line:
|
|
||||||
|
|
||||||
### Available Commands
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# List all documents
|
|
||||||
haiku-rag list
|
|
||||||
|
|
||||||
# Add document from text
|
|
||||||
haiku-rag add "Your document content here"
|
|
||||||
|
|
||||||
# Add document from file or URL
|
|
||||||
haiku-rag add-src /path/to/document.pdf
|
|
||||||
haiku-rag add-src https://example.com/article.html
|
|
||||||
|
|
||||||
# Get and display a specific document
|
|
||||||
haiku-rag get 1
|
|
||||||
|
|
||||||
# Delete a document by ID
|
|
||||||
haiku-rag delete 1
|
|
||||||
|
|
||||||
# Search documents
|
|
||||||
haiku-rag search "machine learning"
|
|
||||||
|
|
||||||
# Search with custom options
|
|
||||||
haiku-rag search "python programming" --limit 10 --k 100
|
|
||||||
|
|
||||||
# Start file monitoring & MCP server (default HTTP transport)
|
|
||||||
haiku-rag serve # --stdio for stdio transport or --sse for SSE transport
|
|
||||||
```
|
|
||||||
|
|
||||||
All commands support the `--db` option to specify a custom database path. Run
|
|
||||||
```bash
|
|
||||||
haiku-rag command -h
|
|
||||||
```
|
|
||||||
to see additional parameters for a command.
|
|
||||||
|
|
||||||
## File Monitoring & MCP server
|
|
||||||
|
|
||||||
You can start the server (using Streamble HTTP, stdio or SSE transports) with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Start with default HTTP transport
|
|
||||||
haiku-rag serve # --stdio for stdio transport or --sse for SSE transport
|
|
||||||
```
|
|
||||||
|
|
||||||
You need to have set the `MONITOR_DIRECTORIES` environment variable for monitoring to take place.
|
|
||||||
|
|
||||||
### File monitoring
|
|
||||||
|
|
||||||
`haiku.rag` can watch directories for changes and automatically update the document store:
|
|
||||||
|
|
||||||
- **Startup**: Scan all monitored directories and add any new files
|
|
||||||
- **File Added/Modified**: Automatically parse and add/update the document in the database
|
|
||||||
- **File Deleted**: Remove the corresponding document from the database
|
|
||||||
|
|
||||||
### MCP Server
|
|
||||||
|
|
||||||
`haiku.rag` includes a Model Context Protocol (MCP) server that exposes RAG functionality as tools for AI assistants like Claude Desktop. The MCP server provides the following tools:
|
|
||||||
|
|
||||||
- `add_document_from_file` - Add documents from local file paths
|
|
||||||
- `add_document_from_url` - Add documents from URLs
|
|
||||||
- `add_document_from_text` - Add documents from raw text content
|
|
||||||
- `search_documents` - Search documents using hybrid search
|
|
||||||
- `get_document` - Retrieve specific documents by ID
|
|
||||||
- `list_documents` - List all documents with pagination
|
|
||||||
- `delete_document` - Delete documents by ID
|
|
||||||
|
|
||||||
## Using `haiku.rag` from python
|
|
||||||
|
|
||||||
### Managing documents
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from pathlib import Path
|
|
||||||
from haiku.rag.client import HaikuRAG
|
from haiku.rag.client import HaikuRAG
|
||||||
|
|
||||||
# Use as async context manager (recommended)
|
|
||||||
async with HaikuRAG("path/to/database.db") as client:
|
|
||||||
# Create document from text
|
|
||||||
doc = await client.create_document(
|
|
||||||
content="Your document content here",
|
|
||||||
uri="doc://example",
|
|
||||||
metadata={"source": "manual", "topic": "example"}
|
|
||||||
)
|
|
||||||
|
|
||||||
# Create document from file (auto-parses content)
|
|
||||||
doc = await client.create_document_from_source("path/to/document.pdf")
|
|
||||||
|
|
||||||
# Create document from URL
|
|
||||||
doc = await client.create_document_from_source("https://example.com/article.html")
|
|
||||||
|
|
||||||
# Retrieve documents
|
|
||||||
doc = await client.get_document_by_id(1)
|
|
||||||
doc = await client.get_document_by_uri("file:///path/to/document.pdf")
|
|
||||||
|
|
||||||
# List all documents with pagination
|
|
||||||
docs = await client.list_documents(limit=10, offset=0)
|
|
||||||
|
|
||||||
# Update document content
|
|
||||||
doc.content = "Updated content"
|
|
||||||
await client.update_document(doc)
|
|
||||||
|
|
||||||
# Delete document
|
|
||||||
await client.delete_document(doc.id)
|
|
||||||
|
|
||||||
# Search documents using hybrid search (vector + full-text)
|
|
||||||
results = await client.search("machine learning algorithms", limit=5)
|
|
||||||
for chunk, score in results:
|
|
||||||
print(f"Score: {score:.3f}")
|
|
||||||
print(f"Content: {chunk.content}")
|
|
||||||
print(f"Document ID: {chunk.document_id}")
|
|
||||||
print("---")
|
|
||||||
```
|
|
||||||
|
|
||||||
## Searching documents
|
|
||||||
|
|
||||||
```python
|
|
||||||
async with HaikuRAG("database.db") as client:
|
async with HaikuRAG("database.db") as client:
|
||||||
|
# Add document
|
||||||
|
doc = await client.create_document("Your content")
|
||||||
|
|
||||||
results = await client.search(
|
# Search
|
||||||
query="machine learning",
|
results = await client.search("query")
|
||||||
limit=5, # Maximum results to return, defaults to 5
|
for chunk, score in results:
|
||||||
k=60 # RRF parameter for reciprocal rank fusion, defaults to 60
|
print(f"{score:.3f}: {chunk.content}")
|
||||||
)
|
|
||||||
|
|
||||||
# Process results
|
# Ask questions
|
||||||
for chunk, relevance_score in results:
|
answer = await client.ask("Who is the author of haiku.rag?")
|
||||||
print(f"Relevance: {relevance_score:.3f}")
|
print(answer)
|
||||||
print(f"Content: {chunk.content}")
|
|
||||||
print(f"From document: {chunk.document_id}")
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## MCP Server
|
||||||
|
|
||||||
|
Use with AI assistants like Claude Desktop:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag serve --stdio
|
||||||
|
```
|
||||||
|
|
||||||
|
Provides tools for document management and search directly in your AI assistant.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
Full documentation at: https://ggozad.github.io/haiku.rag/
|
||||||
|
|
||||||
|
- [Installation](https://ggozad.github.io/haiku.rag/installation/) - Provider setup
|
||||||
|
- [Configuration](https://ggozad.github.io/haiku.rag/configuration/) - Environment variables
|
||||||
|
- [CLI](https://ggozad.github.io/haiku.rag/cli/) - Command reference
|
||||||
|
- [Python API](https://ggozad.github.io/haiku.rag/python/) - Complete API docs
|
||||||
|
- [Benchmarks](https://ggozad.github.io/haiku.rag/benchmarks/) - Performance Benchmarks
|
||||||
|
|
||||||
## Docker
|
## Docker
|
||||||
|
|
||||||
``` bash
|
``` bash
|
||||||
|
|
|
||||||
BIN
docs/.DS_Store
vendored
Normal file
BIN
docs/.DS_Store
vendored
Normal file
Binary file not shown.
28
docs/benchmarks.md
Normal file
28
docs/benchmarks.md
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
# Benchmarks
|
||||||
|
|
||||||
|
We use the [repliqa](https://huggingface.co/datasets/ServiceNow/repliqa) dataset for the evaluation of `haiku.rag`.
|
||||||
|
|
||||||
|
You can perform your own evaluations using as example the script found at
|
||||||
|
`tests/generate_benchmark_db.py`.
|
||||||
|
|
||||||
|
## Recall
|
||||||
|
|
||||||
|
In order to calculate recall, we load the `News Stories` from `repliqa_3` which is 1035 documents and index them in a sqlite db. Subsequently, we run a search over the `question` field for each row of the dataset and check whether we match the document that answers the question.
|
||||||
|
|
||||||
|
|
||||||
|
The recall obtained is ~0.73 for matching in the top result, raising to ~0.75 for the top 3 results.
|
||||||
|
|
||||||
|
| Model | Document in top 1 | Document in top 3 |
|
||||||
|
|---------------------------------------|-------------------|-------------------|
|
||||||
|
| Ollama / `mxbai-embed-large` | 0.77 | 0.89 |
|
||||||
|
| Ollama / `nomic-embed-text` | 0.74 | 0.88 |
|
||||||
|
| OpenAI / `text-embeddings-3-small` | 0.75 | 0.88 |
|
||||||
|
|
||||||
|
## Question/Answer evaluation
|
||||||
|
|
||||||
|
Again using the same dataset, we use a QA agent to answer the question. In addition we use an LLM judge (using the Ollama `qwen3`) to evaluate whether the answer is correct or not. The obtained accuracy is as follows:
|
||||||
|
|
||||||
|
| Embedding Model | QA Model | Accuracy |
|
||||||
|
|------------------------------|-----------------------------------|-----------|
|
||||||
|
| Ollama / `mxbai-embed-large` | Ollama / `qwen3` | 0.64 |
|
||||||
|
| Ollama / `mxbai-embed-large` | Anthropic / `Claude Sonnet 3.7` | 0.79 |
|
||||||
100
docs/cli.md
Normal file
100
docs/cli.md
Normal file
|
|
@ -0,0 +1,100 @@
|
||||||
|
# Command Line Interface
|
||||||
|
|
||||||
|
The `haiku-rag` CLI provides complete document management functionality.
|
||||||
|
|
||||||
|
## Document Management
|
||||||
|
|
||||||
|
### List Documents
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag list
|
||||||
|
```
|
||||||
|
|
||||||
|
### Add Documents
|
||||||
|
|
||||||
|
From text:
|
||||||
|
```bash
|
||||||
|
haiku-rag add "Your document content here"
|
||||||
|
```
|
||||||
|
|
||||||
|
From file or URL:
|
||||||
|
```bash
|
||||||
|
haiku-rag add-src /path/to/document.pdf
|
||||||
|
haiku-rag add-src https://example.com/article.html
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get Document
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag get 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete Document
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag delete 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rebuild Database
|
||||||
|
|
||||||
|
Rebuild the database by deleting all chunks & embeddings and re-indexing all documents:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag rebuild
|
||||||
|
```
|
||||||
|
|
||||||
|
Use this when you want to change things like the embedding model or chunk size for example.
|
||||||
|
|
||||||
|
## Search
|
||||||
|
|
||||||
|
Basic search:
|
||||||
|
```bash
|
||||||
|
haiku-rag search "machine learning"
|
||||||
|
```
|
||||||
|
|
||||||
|
With options:
|
||||||
|
```bash
|
||||||
|
haiku-rag search "python programming" --limit 10 --k 100
|
||||||
|
```
|
||||||
|
|
||||||
|
## Question Answering
|
||||||
|
|
||||||
|
Ask questions about your documents:
|
||||||
|
```bash
|
||||||
|
haiku-rag ask "Who is the author of haiku.rag?"
|
||||||
|
```
|
||||||
|
|
||||||
|
The QA agent will search your documents for relevant information and provide a comprehensive answer.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
View current configuration settings:
|
||||||
|
```bash
|
||||||
|
haiku-rag settings
|
||||||
|
```
|
||||||
|
|
||||||
|
## Server
|
||||||
|
|
||||||
|
Start the MCP server:
|
||||||
|
```bash
|
||||||
|
# HTTP transport (default)
|
||||||
|
haiku-rag serve
|
||||||
|
|
||||||
|
# stdio transport
|
||||||
|
haiku-rag serve --stdio
|
||||||
|
|
||||||
|
# SSE transport
|
||||||
|
haiku-rag serve --sse
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
All commands support:
|
||||||
|
- `--db` - Specify custom database path
|
||||||
|
- `-h` - Show help for specific command
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```bash
|
||||||
|
haiku-rag list --db /path/to/custom.db
|
||||||
|
haiku-rag add -h
|
||||||
|
```
|
||||||
123
docs/configuration.md
Normal file
123
docs/configuration.md
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
# Configuration
|
||||||
|
|
||||||
|
Configuration is done through the use of environment variables.
|
||||||
|
|
||||||
|
!!! note
|
||||||
|
If you create a db with certain settings and later change them, `haiku.rag` will detect incompatibilities (for example, if you change embedding provider) and will exit. You can **rebuild** the database to apply the new settings, see [Rebuild Database](./cli.md#rebuild-database).
|
||||||
|
|
||||||
|
## File Monitoring
|
||||||
|
|
||||||
|
Set directories to monitor for automatic indexing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Monitor single directory
|
||||||
|
MONITOR_DIRECTORIES="/path/to/documents"
|
||||||
|
|
||||||
|
# Monitor multiple directories
|
||||||
|
MONITOR_DIRECTORIES="/path/to/documents,/another_path/to/documents"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Embedding Providers
|
||||||
|
|
||||||
|
If you use Ollama, you can use any pulled model that supports embeddings.
|
||||||
|
|
||||||
|
### Ollama (Default)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
EMBEDDINGS_PROVIDER="ollama"
|
||||||
|
EMBEDDINGS_MODEL="mxbai-embed-large"
|
||||||
|
EMBEDDINGS_VECTOR_DIM=1024
|
||||||
|
```
|
||||||
|
|
||||||
|
### VoyageAI
|
||||||
|
If you want to use VoyageAI embeddings you will need to install `haiku.rag` with the VoyageAI extras,
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag --extra voyageai
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
EMBEDDINGS_PROVIDER="voyageai"
|
||||||
|
EMBEDDINGS_MODEL="voyage-3.5"
|
||||||
|
EMBEDDINGS_VECTOR_DIM=1024
|
||||||
|
VOYAGE_API_KEY="your-api-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
### OpenAI
|
||||||
|
If you want to use OpenAI embeddings you will need to install `haiku.rag` with the VoyageAI extras,
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag --extra openai
|
||||||
|
```
|
||||||
|
|
||||||
|
and set environment variables.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
EMBEDDINGS_PROVIDER="openai"
|
||||||
|
EMBEDDINGS_MODEL="text-embedding-3-small" # or text-embedding-3-large
|
||||||
|
EMBEDDINGS_VECTOR_DIM=1536
|
||||||
|
OPENAI_API_KEY="your-api-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Question Answering Providers
|
||||||
|
|
||||||
|
Configure which LLM provider to use for question answering.
|
||||||
|
|
||||||
|
### Ollama (Default)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
QA_PROVIDER="ollama"
|
||||||
|
QA_MODEL="qwen3"
|
||||||
|
OLLAMA_BASE_URL="http://localhost:11434"
|
||||||
|
```
|
||||||
|
|
||||||
|
### OpenAI
|
||||||
|
|
||||||
|
For OpenAI QA, you need to install haiku.rag with OpenAI extras:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag --extra openai
|
||||||
|
```
|
||||||
|
|
||||||
|
Then configure:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
QA_PROVIDER="openai"
|
||||||
|
QA_MODEL="gpt-4o-mini" # or gpt-4, gpt-3.5-turbo, etc.
|
||||||
|
OPENAI_API_KEY="your-api-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Anthropic
|
||||||
|
|
||||||
|
For Anthropic QA, you need to install haiku.rag with Anthropic extras:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag --extra anthropic
|
||||||
|
```
|
||||||
|
|
||||||
|
Then configure:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
QA_PROVIDER="anthropic"
|
||||||
|
QA_MODEL="claude-3-5-haiku-20241022" # or claude-3-5-sonnet-20241022, etc.
|
||||||
|
ANTHROPIC_API_KEY="your-api-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Other Settings
|
||||||
|
|
||||||
|
### Database and Storage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Default data directory (where SQLite database is stored)
|
||||||
|
DEFAULT_DATA_DIR="/path/to/data"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Document Processing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Chunk size for document processing
|
||||||
|
CHUNK_SIZE=256
|
||||||
|
|
||||||
|
# Chunk overlap for better context
|
||||||
|
CHUNK_OVERLAP=32
|
||||||
|
```
|
||||||
BIN
docs/img/.DS_Store
vendored
Normal file
BIN
docs/img/.DS_Store
vendored
Normal file
Binary file not shown.
59
docs/index.md
Normal file
59
docs/index.md
Normal file
|
|
@ -0,0 +1,59 @@
|
||||||
|
# haiku.rag
|
||||||
|
|
||||||
|
`haiku.rag` is a Retrieval-Augmented Generation (RAG) library built to work on SQLite alone without the need for external vector databases. It uses [sqlite-vec](https://github.com/asg017/sqlite-vec) for storing the embeddings and performs semantic (vector) search as well as full-text search combined through Reciprocal Rank Fusion. Both open-source (Ollama) as well as commercial (OpenAI, VoyageAI) embedding providers are supported.
|
||||||
|
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Local SQLite**: No need to run additional servers
|
||||||
|
- **Support for various embedding providers**: Ollama, VoyageAI, OpenAI or add your own
|
||||||
|
- **Hybrid Search**: Vector search using `sqlite-vec` combined with full-text search `FTS5`, using Reciprocal Rank Fusion
|
||||||
|
- **Question Answering**: Built-in QA agents using Ollama, OpenAI, or Anthropic.
|
||||||
|
- **File monitoring**: Automatically index files when run as a server
|
||||||
|
- **Extended file format support**: Parse 40+ file formats including PDF, DOCX, HTML, Markdown, audio and more. Or add a URL!
|
||||||
|
- **MCP server**: Exposes functionality as MCP tools
|
||||||
|
- **CLI commands**: Access all functionality from your terminal
|
||||||
|
- **Python client**: Call `haiku.rag` from your own python applications
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
Install haiku.rag:
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag
|
||||||
|
```
|
||||||
|
|
||||||
|
Use from Python:
|
||||||
|
```python
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
|
||||||
|
async with HaikuRAG("database.db") as client:
|
||||||
|
# Add a document
|
||||||
|
doc = await client.create_document("Your content here")
|
||||||
|
|
||||||
|
# Search documents
|
||||||
|
results = await client.search("query")
|
||||||
|
|
||||||
|
# Ask questions
|
||||||
|
answer = await client.ask("Who is the author of haiku.rag?")
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use the CLI:
|
||||||
|
```bash
|
||||||
|
haiku-rag add "Your document content"
|
||||||
|
haiku-rag search "query"
|
||||||
|
haiku-rag ask "Who is the author of haiku.rag?"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [Installation](installation.md) - Install haiku.rag with different providers
|
||||||
|
- [Configuration](configuration.md) - Environment variables and settings
|
||||||
|
- [CLI](cli.md) - Command line interface usage
|
||||||
|
- [Question Answering](qa.md) - QA agents and natural language queries
|
||||||
|
- [Server](server.md) - File monitoring and server mode
|
||||||
|
- [MCP](mcp.md) - Model Context Protocol integration
|
||||||
|
- [Python](python.md) - Python API reference
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the [MIT License](https://raw.githubusercontent.com/ggozad/haiku.rag/main/LICENSE).
|
||||||
37
docs/installation.md
Normal file
37
docs/installation.md
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
# Installation
|
||||||
|
|
||||||
|
## Basic Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag
|
||||||
|
```
|
||||||
|
|
||||||
|
By default, Ollama (with the `mxbai-embed-large` model) is used for embeddings.
|
||||||
|
|
||||||
|
## Provider-Specific Installation
|
||||||
|
|
||||||
|
For other embedding providers, install with extras:
|
||||||
|
|
||||||
|
### VoyageAI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag --extra voyageai
|
||||||
|
```
|
||||||
|
|
||||||
|
### OpenAI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag --extra openai
|
||||||
|
```
|
||||||
|
|
||||||
|
### Anthropic
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv pip install haiku.rag --extra anthropic
|
||||||
|
```
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.10+
|
||||||
|
- SQLite 3.38+
|
||||||
|
- Ollama (for default embeddings)
|
||||||
33
docs/mcp.md
Normal file
33
docs/mcp.md
Normal file
|
|
@ -0,0 +1,33 @@
|
||||||
|
# Model Context Protocol (MCP)
|
||||||
|
|
||||||
|
The MCP server exposes `haiku.rag` as MCP tools for compatible MCP clients.
|
||||||
|
|
||||||
|
## Available Tools
|
||||||
|
|
||||||
|
### Document Management
|
||||||
|
|
||||||
|
- `add_document_from_file` - Add documents from local file paths
|
||||||
|
- `add_document_from_url` - Add documents from URLs
|
||||||
|
- `add_document_from_text` - Add documents from raw text content
|
||||||
|
- `get_document` - Retrieve specific documents by ID
|
||||||
|
- `list_documents` - List all documents with pagination
|
||||||
|
- `delete_document` - Delete documents by ID
|
||||||
|
|
||||||
|
### Search
|
||||||
|
|
||||||
|
- `search_documents` - Search documents using hybrid search (vector + full-text)
|
||||||
|
|
||||||
|
## Starting MCP Server
|
||||||
|
|
||||||
|
The MCP server starts automatically with the serve command and supports `Streamable HTTP`, `stdio` and `SSE` transports:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Default HTTP transport
|
||||||
|
haiku-rag serve
|
||||||
|
|
||||||
|
# stdio transport (for Claude Desktop)
|
||||||
|
haiku-rag serve --stdio
|
||||||
|
|
||||||
|
# SSE transport
|
||||||
|
haiku-rag serve --sse
|
||||||
|
```
|
||||||
116
docs/python.md
Normal file
116
docs/python.md
Normal file
|
|
@ -0,0 +1,116 @@
|
||||||
|
# Python API
|
||||||
|
|
||||||
|
Use `haiku.rag` directly in your Python applications.
|
||||||
|
|
||||||
|
## Basic Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pathlib import Path
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
|
||||||
|
# Use as async context manager (recommended)
|
||||||
|
async with HaikuRAG("path/to/database.db") as client:
|
||||||
|
# Your code here
|
||||||
|
pass
|
||||||
|
```
|
||||||
|
|
||||||
|
## Document Management
|
||||||
|
|
||||||
|
### Creating Documents
|
||||||
|
|
||||||
|
From text:
|
||||||
|
```python
|
||||||
|
doc = await client.create_document(
|
||||||
|
content="Your document content here",
|
||||||
|
uri="doc://example",
|
||||||
|
metadata={"source": "manual", "topic": "example"}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
From file:
|
||||||
|
```python
|
||||||
|
doc = await client.create_document_from_source("path/to/document.pdf")
|
||||||
|
```
|
||||||
|
|
||||||
|
From URL:
|
||||||
|
```python
|
||||||
|
doc = await client.create_document_from_source("https://example.com/article.html")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Retrieving Documents
|
||||||
|
|
||||||
|
By ID:
|
||||||
|
```python
|
||||||
|
doc = await client.get_document_by_id(1)
|
||||||
|
```
|
||||||
|
|
||||||
|
By URI:
|
||||||
|
```python
|
||||||
|
doc = await client.get_document_by_uri("file:///path/to/document.pdf")
|
||||||
|
```
|
||||||
|
|
||||||
|
List all documents:
|
||||||
|
```python
|
||||||
|
docs = await client.list_documents(limit=10, offset=0)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Updating Documents
|
||||||
|
|
||||||
|
```python
|
||||||
|
doc.content = "Updated content"
|
||||||
|
await client.update_document(doc)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deleting Documents
|
||||||
|
|
||||||
|
```python
|
||||||
|
await client.delete_document(doc.id)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rebuilding the Database
|
||||||
|
|
||||||
|
```python
|
||||||
|
async for doc_id in client.rebuild_database():
|
||||||
|
print(f"Processed document {doc_id}")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Searching Documents
|
||||||
|
|
||||||
|
Basic search:
|
||||||
|
```python
|
||||||
|
results = await client.search("machine learning algorithms", limit=5)
|
||||||
|
for chunk, score in results:
|
||||||
|
print(f"Score: {score:.3f}")
|
||||||
|
print(f"Content: {chunk.content}")
|
||||||
|
print(f"Document ID: {chunk.document_id}")
|
||||||
|
```
|
||||||
|
|
||||||
|
With options:
|
||||||
|
```python
|
||||||
|
results = await client.search(
|
||||||
|
query="machine learning",
|
||||||
|
limit=5, # Maximum results to return
|
||||||
|
k=60 # RRF parameter for reciprocal rank fusion
|
||||||
|
)
|
||||||
|
|
||||||
|
# Process results
|
||||||
|
for chunk, relevance_score in results:
|
||||||
|
print(f"Relevance: {relevance_score:.3f}")
|
||||||
|
print(f"Content: {chunk.content}")
|
||||||
|
print(f"From document: {chunk.document_id}")
|
||||||
|
print(f"Document URI: {chunk.document_uri}")
|
||||||
|
print(f"Document metadata: {chunk.document_meta}")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Question Answering
|
||||||
|
|
||||||
|
Ask questions about your documents:
|
||||||
|
|
||||||
|
```python
|
||||||
|
answer = await client.ask("Who is the author of haiku.rag?")
|
||||||
|
print(answer)
|
||||||
|
```
|
||||||
|
|
||||||
|
The QA agent will search your documents for relevant information and use the configured LLM to generate a comprehensive answer.
|
||||||
|
|
||||||
|
The QA provider and model can be configured via environment variables (see [Configuration](configuration.md)).
|
||||||
41
docs/server.md
Normal file
41
docs/server.md
Normal file
|
|
@ -0,0 +1,41 @@
|
||||||
|
# Server Mode
|
||||||
|
|
||||||
|
The server provides automatic file monitoring and MCP functionality.
|
||||||
|
|
||||||
|
## Starting the Server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
haiku-rag serve
|
||||||
|
```
|
||||||
|
|
||||||
|
Transport options:
|
||||||
|
- `--http` (default) - Streamable HTTP transport
|
||||||
|
- `--stdio` - Standard input/output transport
|
||||||
|
- `--sse` - Server-sent events transport
|
||||||
|
|
||||||
|
## File Monitoring
|
||||||
|
|
||||||
|
Set `MONITOR_DIRECTORIES` environment variable to enable automatic file monitoring:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export MONITOR_DIRECTORIES="/path/to/documents"
|
||||||
|
haiku-rag serve
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monitoring Features
|
||||||
|
|
||||||
|
- **Startup**: Scans all monitored directories and adds new files
|
||||||
|
- **File Added/Modified**: Automatically parses and updates documents
|
||||||
|
- **File Deleted**: Removes corresponding documents from database
|
||||||
|
|
||||||
|
### Supported Formats
|
||||||
|
|
||||||
|
The server can parse 40+ file formats including:
|
||||||
|
- PDF documents
|
||||||
|
- Microsoft Office (DOCX, XLSX, PPTX)
|
||||||
|
- HTML and Markdown
|
||||||
|
- Plain text files
|
||||||
|
- Audio files
|
||||||
|
- And more...
|
||||||
|
|
||||||
|
URLs are also supported for web content.
|
||||||
78
mkdocs.yml
Normal file
78
mkdocs.yml
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
site_name: haiku.rag
|
||||||
|
site_description: Retrieval-Augmented Generation (RAG) library on SQLite.
|
||||||
|
site_url: https://ggozad.github.io/haiku.rag/
|
||||||
|
theme:
|
||||||
|
name: material
|
||||||
|
palette:
|
||||||
|
- media: "(prefers-color-scheme)"
|
||||||
|
toggle:
|
||||||
|
icon: material/lightbulb-auto
|
||||||
|
name: Switch to light mode
|
||||||
|
- media: "(prefers-color-scheme: light)"
|
||||||
|
scheme: default
|
||||||
|
primary: deep purple
|
||||||
|
accent: amber
|
||||||
|
toggle:
|
||||||
|
icon: material/lightbulb
|
||||||
|
name: Switch to dark mode
|
||||||
|
- media: "(prefers-color-scheme: dark)"
|
||||||
|
scheme: slate
|
||||||
|
primary: deep purple
|
||||||
|
accent: amber
|
||||||
|
toggle:
|
||||||
|
icon: material/lightbulb-outline
|
||||||
|
name: Switch to system preference
|
||||||
|
features:
|
||||||
|
- content.code.annotate
|
||||||
|
- content.code.copy
|
||||||
|
- content.code.select
|
||||||
|
- content.footnote.tooltips
|
||||||
|
- content.tabs.link
|
||||||
|
- content.tooltips
|
||||||
|
- navigation.footer
|
||||||
|
- navigation.indexes
|
||||||
|
- navigation.instant
|
||||||
|
- navigation.instant.prefetch
|
||||||
|
- navigation.instant.progress
|
||||||
|
- navigation.path
|
||||||
|
- navigation.tabs
|
||||||
|
- navigation.tabs.sticky
|
||||||
|
- navigation.top
|
||||||
|
- navigation.tracking
|
||||||
|
- search.highlight
|
||||||
|
- search.share
|
||||||
|
- search.suggest
|
||||||
|
- toc.follow
|
||||||
|
|
||||||
|
icon:
|
||||||
|
repo: fontawesome/brands/github-alt
|
||||||
|
# logo: img/icon-white.svg
|
||||||
|
# favicon: img/favicon.png
|
||||||
|
language: en
|
||||||
|
repo_name: ggozad/haiku.rag
|
||||||
|
repo_url: https://github.com/ggozad/haiku.rag
|
||||||
|
plugins:
|
||||||
|
# Material for MkDocs
|
||||||
|
search:
|
||||||
|
nav:
|
||||||
|
- haiku.rag:
|
||||||
|
- index.md
|
||||||
|
- Installation: installation.md
|
||||||
|
- Configuration: configuration.md
|
||||||
|
- CLI: cli.md
|
||||||
|
- Server: server.md
|
||||||
|
- MCP: mcp.md
|
||||||
|
- Python: python.md
|
||||||
|
- Benchmarks: benchmarks.md
|
||||||
|
markdown_extensions:
|
||||||
|
- admonition
|
||||||
|
- attr_list
|
||||||
|
- pymdownx.details
|
||||||
|
- pymdownx.highlight:
|
||||||
|
anchor_linenums: true
|
||||||
|
line_spans: __span
|
||||||
|
pygments_lang_class: true
|
||||||
|
use_pygments: true
|
||||||
|
- pymdownx.inlinehilite
|
||||||
|
- pymdownx.snippets
|
||||||
|
- pymdownx.superfences
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
[project]
|
[project]
|
||||||
name = "haiku.rag"
|
name = "haiku.rag"
|
||||||
version = "0.2.0"
|
version = "0.3.4"
|
||||||
description = "Retrieval Augmented Generation (RAG) with SQLite"
|
description = "Retrieval Augmented Generation (RAG) with SQLite"
|
||||||
authors = [{ name = "Yiorgis Gozadinos", email = "ggozadinos@gmail.com" }]
|
authors = [{ name = "Yiorgis Gozadinos", email = "ggozadinos@gmail.com" }]
|
||||||
license = { text = "MIT" }
|
license = { text = "MIT" }
|
||||||
|
|
@ -38,6 +38,7 @@ dependencies = [
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
voyageai = ["voyageai>=0.3.2"]
|
voyageai = ["voyageai>=0.3.2"]
|
||||||
openai = ["openai>=1.0.0"]
|
openai = ["openai>=1.0.0"]
|
||||||
|
anthropic = ["anthropic>=0.56.0"]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
haiku-rag = "haiku.rag.cli:cli"
|
haiku-rag = "haiku.rag.cli:cli"
|
||||||
|
|
@ -52,8 +53,10 @@ packages = ["src/haiku"]
|
||||||
[dependency-groups]
|
[dependency-groups]
|
||||||
dev = [
|
dev = [
|
||||||
"datasets>=3.6.0",
|
"datasets>=3.6.0",
|
||||||
|
"mkdocs>=1.6.1",
|
||||||
|
"mkdocs-material>=9.6.14",
|
||||||
"pre-commit>=4.2.0",
|
"pre-commit>=4.2.0",
|
||||||
"pyright>=1.1.402",
|
"pyright>=1.1.403",
|
||||||
"pytest>=8.4.0",
|
"pytest>=8.4.0",
|
||||||
"pytest-asyncio>=1.0.0",
|
"pytest-asyncio>=1.0.0",
|
||||||
"pytest-cov>=6.2.1",
|
"pytest-cov>=6.2.1",
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@ from pathlib import Path
|
||||||
|
|
||||||
from rich.console import Console
|
from rich.console import Console
|
||||||
from rich.markdown import Markdown
|
from rich.markdown import Markdown
|
||||||
|
from rich.progress import Progress
|
||||||
|
|
||||||
from haiku.rag.client import HaikuRAG
|
from haiku.rag.client import HaikuRAG
|
||||||
from haiku.rag.config import Config
|
from haiku.rag.config import Config
|
||||||
|
|
@ -61,6 +62,61 @@ class HaikuRAGApp:
|
||||||
for chunk, score in results:
|
for chunk, score in results:
|
||||||
self._rich_print_search_result(chunk, score)
|
self._rich_print_search_result(chunk, score)
|
||||||
|
|
||||||
|
async def ask(self, question: str):
|
||||||
|
async with HaikuRAG(db_path=self.db_path) as self.client:
|
||||||
|
try:
|
||||||
|
answer = await self.client.ask(question)
|
||||||
|
self.console.print(f"[bold blue]Question:[/bold blue] {question}")
|
||||||
|
self.console.print()
|
||||||
|
self.console.print("[bold green]Answer:[/bold green]")
|
||||||
|
self.console.print(Markdown(answer))
|
||||||
|
except Exception as e:
|
||||||
|
self.console.print(f"[red]Error: {e}[/red]")
|
||||||
|
|
||||||
|
async def rebuild(self):
|
||||||
|
async with HaikuRAG(db_path=self.db_path, skip_validation=True) as client:
|
||||||
|
try:
|
||||||
|
documents = await client.list_documents()
|
||||||
|
total_docs = len(documents)
|
||||||
|
|
||||||
|
if total_docs == 0:
|
||||||
|
self.console.print(
|
||||||
|
"[yellow]No documents found in database.[/yellow]"
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
self.console.print(
|
||||||
|
f"[b]Rebuilding database with {total_docs} documents...[/b]"
|
||||||
|
)
|
||||||
|
with Progress() as progress:
|
||||||
|
task = progress.add_task("Rebuilding...", total=total_docs)
|
||||||
|
async for _ in client.rebuild_database():
|
||||||
|
progress.update(task, advance=1)
|
||||||
|
|
||||||
|
self.console.print("[b]Database rebuild completed successfully.[/b]")
|
||||||
|
except Exception as e:
|
||||||
|
self.console.print(f"[red]Error rebuilding database: {e}[/red]")
|
||||||
|
|
||||||
|
def show_settings(self):
|
||||||
|
"""Display current configuration settings."""
|
||||||
|
self.console.print("[bold]haiku.rag configuration[/bold]")
|
||||||
|
self.console.print()
|
||||||
|
|
||||||
|
# Get all config fields dynamically
|
||||||
|
for field_name, field_value in Config.model_dump().items():
|
||||||
|
# Format the display value
|
||||||
|
if isinstance(field_value, str) and (
|
||||||
|
"key" in field_name.lower()
|
||||||
|
or "password" in field_name.lower()
|
||||||
|
or "token" in field_name.lower()
|
||||||
|
):
|
||||||
|
# Hide sensitive values but show if they're set
|
||||||
|
display_value = "✓ Set" if field_value else "✗ Not set"
|
||||||
|
else:
|
||||||
|
display_value = field_value
|
||||||
|
|
||||||
|
self.console.print(f" [cyan]{field_name}[/cyan]: {display_value}")
|
||||||
|
|
||||||
def _rich_print_document(self, doc: Document, truncate: bool = False):
|
def _rich_print_document(self, doc: Document, truncate: bool = False):
|
||||||
"""Format a document for display."""
|
"""Format a document for display."""
|
||||||
if truncate:
|
if truncate:
|
||||||
|
|
@ -88,6 +144,12 @@ class HaikuRAGApp:
|
||||||
f"[repr.attrib_name]document_id[/repr.attrib_name]: {chunk.document_id} "
|
f"[repr.attrib_name]document_id[/repr.attrib_name]: {chunk.document_id} "
|
||||||
f"[repr.attrib_name]score[/repr.attrib_name]: {score:.4f}"
|
f"[repr.attrib_name]score[/repr.attrib_name]: {score:.4f}"
|
||||||
)
|
)
|
||||||
|
if chunk.document_uri:
|
||||||
|
self.console.print("[repr.attrib_name]document uri[/repr.attrib_name]:")
|
||||||
|
self.console.print(chunk.document_uri)
|
||||||
|
if chunk.document_meta:
|
||||||
|
self.console.print("[repr.attrib_name]document meta[/repr.attrib_name]:")
|
||||||
|
self.console.print(chunk.document_meta)
|
||||||
self.console.print("[repr.attrib_name]content[/repr.attrib_name]:")
|
self.console.print("[repr.attrib_name]content[/repr.attrib_name]:")
|
||||||
self.console.print(content)
|
self.console.print(content)
|
||||||
self.console.rule()
|
self.console.rule()
|
||||||
|
|
|
||||||
|
|
@ -6,15 +6,11 @@ from haiku.rag.config import Config
|
||||||
|
|
||||||
|
|
||||||
class Chunker:
|
class Chunker:
|
||||||
"""
|
"""A class that chunks text into smaller pieces for embedding and retrieval.
|
||||||
A class that chunks text into smaller pieces for embedding and retrieval.
|
|
||||||
|
|
||||||
Parameters
|
Args:
|
||||||
----------
|
chunk_size: The maximum size of a chunk in tokens.
|
||||||
chunk_size : int
|
chunk_overlap: The number of tokens of overlap between chunks.
|
||||||
The maximum size of a chunk in characters.
|
|
||||||
chunk_overlap : int
|
|
||||||
The number of characters of overlap between chunks.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
encoder: ClassVar[tiktoken.Encoding] = tiktoken.encoding_for_model("gpt-4o")
|
encoder: ClassVar[tiktoken.Encoding] = tiktoken.encoding_for_model("gpt-4o")
|
||||||
|
|
@ -28,18 +24,13 @@ class Chunker:
|
||||||
self.chunk_overlap = chunk_overlap
|
self.chunk_overlap = chunk_overlap
|
||||||
|
|
||||||
async def chunk(self, text: str) -> list[str]:
|
async def chunk(self, text: str) -> list[str]:
|
||||||
"""
|
"""Split the text into chunks based on token boundaries.
|
||||||
Split the text into chunks.
|
|
||||||
|
|
||||||
Parameters
|
Args:
|
||||||
----------
|
text: The text to be split into chunks.
|
||||||
text : str
|
|
||||||
The text to be split into chunks.
|
|
||||||
|
|
||||||
Returns
|
Returns:
|
||||||
-------
|
A list of text chunks with token-based boundaries and overlap.
|
||||||
list
|
|
||||||
A list of text chunks.
|
|
||||||
"""
|
"""
|
||||||
if not text:
|
if not text:
|
||||||
return []
|
return []
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@ import typer
|
||||||
from rich.console import Console
|
from rich.console import Console
|
||||||
|
|
||||||
from haiku.rag.app import HaikuRAGApp
|
from haiku.rag.app import HaikuRAGApp
|
||||||
from haiku.rag.utils import get_default_data_dir
|
from haiku.rag.utils import get_default_data_dir, is_up_to_date
|
||||||
|
|
||||||
cli = typer.Typer(
|
cli = typer.Typer(
|
||||||
context_settings={"help_option_names": ["-h", "--help"]}, no_args_is_help=True
|
context_settings={"help_option_names": ["-h", "--help"]}, no_args_is_help=True
|
||||||
|
|
@ -15,6 +15,23 @@ console = Console()
|
||||||
event_loop = asyncio.get_event_loop()
|
event_loop = asyncio.get_event_loop()
|
||||||
|
|
||||||
|
|
||||||
|
async def check_version():
|
||||||
|
"""Check if haiku.rag is up to date and show warning if not."""
|
||||||
|
up_to_date, current_version, latest_version = await is_up_to_date()
|
||||||
|
if not up_to_date:
|
||||||
|
console.print(
|
||||||
|
f"[yellow]Warning: haiku.rag is outdated. Current: {current_version}, Latest: {latest_version}[/yellow]"
|
||||||
|
)
|
||||||
|
console.print("[yellow]Please update.[/yellow]")
|
||||||
|
|
||||||
|
|
||||||
|
@cli.callback()
|
||||||
|
def main():
|
||||||
|
"""haiku.rag CLI - SQLite-based RAG system"""
|
||||||
|
# Run version check before any command
|
||||||
|
event_loop.run_until_complete(check_version())
|
||||||
|
|
||||||
|
|
||||||
@cli.command("list", help="List all stored documents")
|
@cli.command("list", help="List all stored documents")
|
||||||
def list_documents(
|
def list_documents(
|
||||||
db: Path = typer.Option(
|
db: Path = typer.Option(
|
||||||
|
|
@ -113,6 +130,42 @@ def search(
|
||||||
event_loop.run_until_complete(app.search(query=query, limit=limit, k=k))
|
event_loop.run_until_complete(app.search(query=query, limit=limit, k=k))
|
||||||
|
|
||||||
|
|
||||||
|
@cli.command("ask", help="Ask a question using the QA agent")
|
||||||
|
def ask(
|
||||||
|
question: str = typer.Argument(
|
||||||
|
help="The question to ask",
|
||||||
|
),
|
||||||
|
db: Path = typer.Option(
|
||||||
|
get_default_data_dir() / "haiku.rag.sqlite",
|
||||||
|
"--db",
|
||||||
|
help="Path to the SQLite database file",
|
||||||
|
),
|
||||||
|
):
|
||||||
|
app = HaikuRAGApp(db_path=db)
|
||||||
|
event_loop.run_until_complete(app.ask(question=question))
|
||||||
|
|
||||||
|
|
||||||
|
@cli.command("settings", help="Display current configuration settings")
|
||||||
|
def settings():
|
||||||
|
app = HaikuRAGApp(db_path=Path()) # Don't need actual DB for settings
|
||||||
|
app.show_settings()
|
||||||
|
|
||||||
|
|
||||||
|
@cli.command(
|
||||||
|
"rebuild",
|
||||||
|
help="Rebuild the database by deleting all chunks and re-indexing all documents",
|
||||||
|
)
|
||||||
|
def rebuild(
|
||||||
|
db: Path = typer.Option(
|
||||||
|
get_default_data_dir() / "haiku.rag.sqlite",
|
||||||
|
"--db",
|
||||||
|
help="Path to the SQLite database file",
|
||||||
|
),
|
||||||
|
):
|
||||||
|
app = HaikuRAGApp(db_path=db)
|
||||||
|
event_loop.run_until_complete(app.rebuild())
|
||||||
|
|
||||||
|
|
||||||
@cli.command(
|
@cli.command(
|
||||||
"serve", help="Start the haiku.rag MCP server (by default in streamable HTTP mode)"
|
"serve", help="Start the haiku.rag MCP server (by default in streamable HTTP mode)"
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
import hashlib
|
import hashlib
|
||||||
import mimetypes
|
import mimetypes
|
||||||
import tempfile
|
import tempfile
|
||||||
|
from collections.abc import AsyncGenerator
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
|
@ -23,12 +24,18 @@ class HaikuRAG:
|
||||||
self,
|
self,
|
||||||
db_path: Path | Literal[":memory:"] = Config.DEFAULT_DATA_DIR
|
db_path: Path | Literal[":memory:"] = Config.DEFAULT_DATA_DIR
|
||||||
/ "haiku.rag.sqlite",
|
/ "haiku.rag.sqlite",
|
||||||
|
skip_validation: bool = False,
|
||||||
):
|
):
|
||||||
"""Initialize the RAG client with a database path."""
|
"""Initialize the RAG client with a database path.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
db_path: Path to the SQLite database file or ":memory:" for in-memory database.
|
||||||
|
skip_validation: Whether to skip configuration validation on database load.
|
||||||
|
"""
|
||||||
if isinstance(db_path, Path):
|
if isinstance(db_path, Path):
|
||||||
if not db_path.parent.exists():
|
if not db_path.parent.exists():
|
||||||
Path.mkdir(db_path.parent, parents=True)
|
Path.mkdir(db_path.parent, parents=True)
|
||||||
self.store = Store(db_path)
|
self.store = Store(db_path, skip_validation=skip_validation)
|
||||||
self.document_repository = DocumentRepository(self.store)
|
self.document_repository = DocumentRepository(self.store)
|
||||||
self.chunk_repository = ChunkRepository(self.store)
|
self.chunk_repository = ChunkRepository(self.store)
|
||||||
|
|
||||||
|
|
@ -36,7 +43,7 @@ class HaikuRAG:
|
||||||
"""Async context manager entry."""
|
"""Async context manager entry."""
|
||||||
return self
|
return self
|
||||||
|
|
||||||
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
async def __aexit__(self, exc_type, exc_val, exc_tb): # noqa: ARG002
|
||||||
"""Async context manager exit."""
|
"""Async context manager exit."""
|
||||||
self.close()
|
self.close()
|
||||||
return False
|
return False
|
||||||
|
|
@ -44,7 +51,16 @@ class HaikuRAG:
|
||||||
async def create_document(
|
async def create_document(
|
||||||
self, content: str, uri: str | None = None, metadata: dict | None = None
|
self, content: str, uri: str | None = None, metadata: dict | None = None
|
||||||
) -> Document:
|
) -> Document:
|
||||||
"""Create a new document with optional URI and metadata."""
|
"""Create a new document with optional URI and metadata.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
content: The text content of the document.
|
||||||
|
uri: Optional URI identifier for the document.
|
||||||
|
metadata: Optional metadata dictionary.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The created Document instance.
|
||||||
|
"""
|
||||||
document = Document(
|
document = Document(
|
||||||
content=content,
|
content=content,
|
||||||
uri=uri,
|
uri=uri,
|
||||||
|
|
@ -164,12 +180,12 @@ class HaikuRAG:
|
||||||
|
|
||||||
# Create a temporary file with the appropriate extension
|
# Create a temporary file with the appropriate extension
|
||||||
with tempfile.NamedTemporaryFile(
|
with tempfile.NamedTemporaryFile(
|
||||||
mode="wb", suffix=file_extension, delete=False
|
mode="wb", suffix=file_extension
|
||||||
) as temp_file:
|
) as temp_file:
|
||||||
temp_file.write(response.content)
|
temp_file.write(response.content)
|
||||||
|
temp_file.flush() # Ensure content is written to disk
|
||||||
temp_path = Path(temp_file.name)
|
temp_path = Path(temp_file.name)
|
||||||
|
|
||||||
try:
|
|
||||||
# Parse the content using FileReader
|
# Parse the content using FileReader
|
||||||
content = FileReader.parse_file(temp_path)
|
content = FileReader.parse_file(temp_path)
|
||||||
|
|
||||||
|
|
@ -184,9 +200,6 @@ class HaikuRAG:
|
||||||
return await self.create_document(
|
return await self.create_document(
|
||||||
content=content, uri=url, metadata=metadata
|
content=content, uri=url, metadata=metadata
|
||||||
)
|
)
|
||||||
finally:
|
|
||||||
# Clean up temporary file
|
|
||||||
temp_path.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
def _get_extension_from_content_type_or_url(
|
def _get_extension_from_content_type_or_url(
|
||||||
self, url: str, content_type: str
|
self, url: str, content_type: str
|
||||||
|
|
@ -220,11 +233,25 @@ class HaikuRAG:
|
||||||
return ".html"
|
return ".html"
|
||||||
|
|
||||||
async def get_document_by_id(self, document_id: int) -> Document | None:
|
async def get_document_by_id(self, document_id: int) -> Document | None:
|
||||||
"""Get a document by its ID."""
|
"""Get a document by its ID.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
document_id: The unique identifier of the document.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The Document instance if found, None otherwise.
|
||||||
|
"""
|
||||||
return await self.document_repository.get_by_id(document_id)
|
return await self.document_repository.get_by_id(document_id)
|
||||||
|
|
||||||
async def get_document_by_uri(self, uri: str) -> Document | None:
|
async def get_document_by_uri(self, uri: str) -> Document | None:
|
||||||
"""Get a document by its URI."""
|
"""Get a document by its URI.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
uri: The URI identifier of the document.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The Document instance if found, None otherwise.
|
||||||
|
"""
|
||||||
return await self.document_repository.get_by_uri(uri)
|
return await self.document_repository.get_by_uri(uri)
|
||||||
|
|
||||||
async def update_document(self, document: Document) -> Document:
|
async def update_document(self, document: Document) -> Document:
|
||||||
|
|
@ -238,7 +265,15 @@ class HaikuRAG:
|
||||||
async def list_documents(
|
async def list_documents(
|
||||||
self, limit: int | None = None, offset: int | None = None
|
self, limit: int | None = None, offset: int | None = None
|
||||||
) -> list[Document]:
|
) -> list[Document]:
|
||||||
"""List all documents with optional pagination."""
|
"""List all documents with optional pagination.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
limit: Maximum number of documents to return.
|
||||||
|
offset: Number of documents to skip.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of Document instances.
|
||||||
|
"""
|
||||||
return await self.document_repository.list_all(limit=limit, offset=offset)
|
return await self.document_repository.list_all(limit=limit, offset=offset)
|
||||||
|
|
||||||
async def search(
|
async def search(
|
||||||
|
|
@ -247,15 +282,56 @@ class HaikuRAG:
|
||||||
"""Search for relevant chunks using hybrid search (vector similarity + full-text search).
|
"""Search for relevant chunks using hybrid search (vector similarity + full-text search).
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
query: The search query string
|
query: The search query string.
|
||||||
limit: Maximum number of results to return
|
limit: Maximum number of results to return.
|
||||||
k: Parameter for Reciprocal Rank Fusion (default: 60)
|
k: Parameter for Reciprocal Rank Fusion (default: 60).
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of (chunk, score) tuples ordered by relevance
|
List of (chunk, score) tuples ordered by relevance.
|
||||||
"""
|
"""
|
||||||
return await self.chunk_repository.search_chunks_hybrid(query, limit, k)
|
return await self.chunk_repository.search_chunks_hybrid(query, limit, k)
|
||||||
|
|
||||||
|
async def ask(self, question: str) -> str:
|
||||||
|
"""Ask a question using the configured QA agent.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
question: The question to ask.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The generated answer as a string.
|
||||||
|
"""
|
||||||
|
from haiku.rag.qa import get_qa_agent
|
||||||
|
|
||||||
|
qa_agent = get_qa_agent(self)
|
||||||
|
return await qa_agent.answer(question)
|
||||||
|
|
||||||
|
async def rebuild_database(self) -> AsyncGenerator[int, None]:
|
||||||
|
"""Rebuild the database by deleting all chunks and re-indexing all documents.
|
||||||
|
|
||||||
|
Yields:
|
||||||
|
int: The ID of the document currently being processed
|
||||||
|
"""
|
||||||
|
await self.chunk_repository.delete_all()
|
||||||
|
self.store.recreate_embeddings_table()
|
||||||
|
|
||||||
|
# Update settings to current config
|
||||||
|
from haiku.rag.store.repositories.settings import SettingsRepository
|
||||||
|
|
||||||
|
settings_repo = SettingsRepository(self.store)
|
||||||
|
settings_repo.save()
|
||||||
|
|
||||||
|
documents = await self.list_documents()
|
||||||
|
|
||||||
|
for doc in documents:
|
||||||
|
if doc.id is not None:
|
||||||
|
await self.chunk_repository.create_chunks_for_document(
|
||||||
|
doc.id, doc.content, commit=False
|
||||||
|
)
|
||||||
|
yield doc.id
|
||||||
|
|
||||||
|
if self.store._connection:
|
||||||
|
self.store._connection.commit()
|
||||||
|
|
||||||
def close(self):
|
def close(self):
|
||||||
"""Close the underlying store connection."""
|
"""Close the underlying store connection."""
|
||||||
self.store.close()
|
self.store.close()
|
||||||
|
|
|
||||||
|
|
@ -19,11 +19,19 @@ class AppConfig(BaseModel):
|
||||||
EMBEDDINGS_MODEL: str = "mxbai-embed-large"
|
EMBEDDINGS_MODEL: str = "mxbai-embed-large"
|
||||||
EMBEDDINGS_VECTOR_DIM: int = 1024
|
EMBEDDINGS_VECTOR_DIM: int = 1024
|
||||||
|
|
||||||
|
QA_PROVIDER: str = "ollama"
|
||||||
|
QA_MODEL: str = "qwen3"
|
||||||
|
|
||||||
CHUNK_SIZE: int = 256
|
CHUNK_SIZE: int = 256
|
||||||
CHUNK_OVERLAP: int = 32
|
CHUNK_OVERLAP: int = 32
|
||||||
|
|
||||||
OLLAMA_BASE_URL: str = "http://localhost:11434"
|
OLLAMA_BASE_URL: str = "http://localhost:11434"
|
||||||
|
|
||||||
|
# Provider keys
|
||||||
|
VOYAGE_API_KEY: str = ""
|
||||||
|
OPENAI_API_KEY: str = ""
|
||||||
|
ANTHROPIC_API_KEY: str = ""
|
||||||
|
|
||||||
@field_validator("MONITOR_DIRECTORIES", mode="before")
|
@field_validator("MONITOR_DIRECTORIES", mode="before")
|
||||||
@classmethod
|
@classmethod
|
||||||
def parse_monitor_directories(cls, v):
|
def parse_monitor_directories(cls, v):
|
||||||
|
|
@ -38,3 +46,9 @@ class AppConfig(BaseModel):
|
||||||
|
|
||||||
# Expose Config object for app to import
|
# Expose Config object for app to import
|
||||||
Config = AppConfig.model_validate(os.environ)
|
Config = AppConfig.model_validate(os.environ)
|
||||||
|
if Config.OPENAI_API_KEY:
|
||||||
|
os.environ["OPENAI_API_KEY"] = Config.OPENAI_API_KEY
|
||||||
|
if Config.VOYAGE_API_KEY:
|
||||||
|
os.environ["VOYAGE_API_KEY"] = Config.VOYAGE_API_KEY
|
||||||
|
if Config.ANTHROPIC_API_KEY:
|
||||||
|
os.environ["ANTHROPIC_API_KEY"] = Config.ANTHROPIC_API_KEY
|
||||||
|
|
|
||||||
|
|
@ -49,7 +49,6 @@ class FileWatcher:
|
||||||
try:
|
try:
|
||||||
uri = file.as_uri()
|
uri = file.as_uri()
|
||||||
existing_doc = await self.client.get_document_by_uri(uri)
|
existing_doc = await self.client.get_document_by_uri(uri)
|
||||||
print(uri)
|
|
||||||
if existing_doc:
|
if existing_doc:
|
||||||
doc = await self.client.create_document_from_source(str(file))
|
doc = await self.client.create_document_from_source(str(file))
|
||||||
logger.info(f"Updated document {existing_doc.id} from {file}")
|
logger.info(f"Updated document {existing_doc.id} from {file}")
|
||||||
|
|
|
||||||
36
src/haiku/rag/qa/__init__.py
Normal file
36
src/haiku/rag/qa/__init__.py
Normal file
|
|
@ -0,0 +1,36 @@
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.config import Config
|
||||||
|
from haiku.rag.qa.base import QuestionAnswerAgentBase
|
||||||
|
from haiku.rag.qa.ollama import QuestionAnswerOllamaAgent
|
||||||
|
|
||||||
|
|
||||||
|
def get_qa_agent(client: HaikuRAG, model: str = "") -> QuestionAnswerAgentBase:
|
||||||
|
"""
|
||||||
|
Factory function to get the appropriate QA agent based on the configuration.
|
||||||
|
"""
|
||||||
|
if Config.QA_PROVIDER == "ollama":
|
||||||
|
return QuestionAnswerOllamaAgent(client, model or Config.QA_MODEL)
|
||||||
|
|
||||||
|
if Config.QA_PROVIDER == "openai":
|
||||||
|
try:
|
||||||
|
from haiku.rag.qa.openai import QuestionAnswerOpenAIAgent
|
||||||
|
except ImportError:
|
||||||
|
raise ImportError(
|
||||||
|
"OpenAI QA agent requires the 'openai' package. "
|
||||||
|
"Please install haiku.rag with the 'openai' extra:"
|
||||||
|
"uv pip install haiku.rag --extra openai"
|
||||||
|
)
|
||||||
|
return QuestionAnswerOpenAIAgent(client, model or Config.QA_MODEL)
|
||||||
|
|
||||||
|
if Config.QA_PROVIDER == "anthropic":
|
||||||
|
try:
|
||||||
|
from haiku.rag.qa.anthropic import QuestionAnswerAnthropicAgent
|
||||||
|
except ImportError:
|
||||||
|
raise ImportError(
|
||||||
|
"Anthropic QA agent requires the 'anthropic' package. "
|
||||||
|
"Please install haiku.rag with the 'anthropic' extra:"
|
||||||
|
"uv pip install haiku.rag --extra anthropic"
|
||||||
|
)
|
||||||
|
return QuestionAnswerAnthropicAgent(client, model or Config.QA_MODEL)
|
||||||
|
|
||||||
|
raise ValueError(f"Unsupported QA provider: {Config.QA_PROVIDER}")
|
||||||
106
src/haiku/rag/qa/anthropic.py
Normal file
106
src/haiku/rag/qa/anthropic.py
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
try:
|
||||||
|
from anthropic import AsyncAnthropic
|
||||||
|
from anthropic.types import MessageParam, TextBlock, ToolParam, ToolUseBlock
|
||||||
|
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.qa.base import QuestionAnswerAgentBase
|
||||||
|
|
||||||
|
class QuestionAnswerAnthropicAgent(QuestionAnswerAgentBase):
|
||||||
|
def __init__(self, client: HaikuRAG, model: str = "claude-3-5-haiku-20241022"):
|
||||||
|
super().__init__(client, model or self._model)
|
||||||
|
self.tools: Sequence[ToolParam] = [
|
||||||
|
ToolParam(
|
||||||
|
name="search_documents",
|
||||||
|
description="Search the knowledge base for relevant documents",
|
||||||
|
input_schema={
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"query": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "The search query to find relevant documents",
|
||||||
|
},
|
||||||
|
"limit": {
|
||||||
|
"type": "integer",
|
||||||
|
"description": "Maximum number of results to return",
|
||||||
|
"default": 3,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"required": ["query"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
async def answer(self, question: str) -> str:
|
||||||
|
anthropic_client = AsyncAnthropic()
|
||||||
|
|
||||||
|
messages: list[MessageParam] = [{"role": "user", "content": question}]
|
||||||
|
|
||||||
|
max_rounds = 5 # Prevent infinite loops
|
||||||
|
|
||||||
|
for _ in range(max_rounds):
|
||||||
|
response = await anthropic_client.messages.create(
|
||||||
|
model=self._model,
|
||||||
|
max_tokens=4096,
|
||||||
|
system=self._system_prompt,
|
||||||
|
messages=messages,
|
||||||
|
tools=self.tools,
|
||||||
|
temperature=0.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
if response.stop_reason == "tool_use":
|
||||||
|
messages.append({"role": "assistant", "content": response.content})
|
||||||
|
|
||||||
|
# Process tool calls
|
||||||
|
tool_results = []
|
||||||
|
for content_block in response.content:
|
||||||
|
if isinstance(content_block, ToolUseBlock):
|
||||||
|
if content_block.name == "search_documents":
|
||||||
|
args = content_block.input
|
||||||
|
query = (
|
||||||
|
args.get("query", question)
|
||||||
|
if isinstance(args, dict)
|
||||||
|
else question
|
||||||
|
)
|
||||||
|
limit = (
|
||||||
|
int(args.get("limit", 3))
|
||||||
|
if isinstance(args, dict)
|
||||||
|
else 3
|
||||||
|
)
|
||||||
|
|
||||||
|
search_results = await self._client.search(
|
||||||
|
query, limit=limit
|
||||||
|
)
|
||||||
|
|
||||||
|
context_chunks = []
|
||||||
|
for chunk, score in search_results:
|
||||||
|
context_chunks.append(
|
||||||
|
f"Content: {chunk.content}\nScore: {score:.4f}"
|
||||||
|
)
|
||||||
|
|
||||||
|
context = "\n\n".join(context_chunks)
|
||||||
|
|
||||||
|
tool_results.append(
|
||||||
|
{
|
||||||
|
"type": "tool_result",
|
||||||
|
"tool_use_id": content_block.id,
|
||||||
|
"content": context,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
if tool_results:
|
||||||
|
messages.append({"role": "user", "content": tool_results})
|
||||||
|
else:
|
||||||
|
# No tool use, return the response
|
||||||
|
if response.content:
|
||||||
|
first_content = response.content[0]
|
||||||
|
if isinstance(first_content, TextBlock):
|
||||||
|
return first_content.text
|
||||||
|
return ""
|
||||||
|
|
||||||
|
# If we've exhausted max rounds, return empty string
|
||||||
|
return ""
|
||||||
|
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
41
src/haiku/rag/qa/base.py
Normal file
41
src/haiku/rag/qa/base.py
Normal file
|
|
@ -0,0 +1,41 @@
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.qa.prompts import SYSTEM_PROMPT
|
||||||
|
|
||||||
|
|
||||||
|
class QuestionAnswerAgentBase:
|
||||||
|
_model: str = ""
|
||||||
|
_system_prompt: str = SYSTEM_PROMPT
|
||||||
|
|
||||||
|
def __init__(self, client: HaikuRAG, model: str = ""):
|
||||||
|
self._model = model
|
||||||
|
self._client = client
|
||||||
|
|
||||||
|
async def answer(self, question: str) -> str:
|
||||||
|
raise NotImplementedError(
|
||||||
|
"QABase is an abstract class. Please implement the answer method in a subclass."
|
||||||
|
)
|
||||||
|
|
||||||
|
tools = [
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "search_documents",
|
||||||
|
"description": "Search the knowledge base for relevant documents",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"query": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "The search query to find relevant documents",
|
||||||
|
},
|
||||||
|
"limit": {
|
||||||
|
"type": "integer",
|
||||||
|
"description": "Maximum number of results to return",
|
||||||
|
"default": 3,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"required": ["query"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
]
|
||||||
64
src/haiku/rag/qa/ollama.py
Normal file
64
src/haiku/rag/qa/ollama.py
Normal file
|
|
@ -0,0 +1,64 @@
|
||||||
|
from ollama import AsyncClient
|
||||||
|
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.config import Config
|
||||||
|
from haiku.rag.qa.base import QuestionAnswerAgentBase
|
||||||
|
|
||||||
|
OLLAMA_OPTIONS = {"temperature": 0.0, "seed": 42, "num_ctx": 64000}
|
||||||
|
|
||||||
|
|
||||||
|
class QuestionAnswerOllamaAgent(QuestionAnswerAgentBase):
|
||||||
|
def __init__(self, client: HaikuRAG, model: str = Config.QA_MODEL):
|
||||||
|
super().__init__(client, model or self._model)
|
||||||
|
|
||||||
|
async def answer(self, question: str) -> str:
|
||||||
|
ollama_client = AsyncClient(host=Config.OLLAMA_BASE_URL)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": self._system_prompt},
|
||||||
|
{"role": "user", "content": question},
|
||||||
|
]
|
||||||
|
|
||||||
|
max_rounds = 5 # Prevent infinite loops
|
||||||
|
|
||||||
|
for _ in range(max_rounds):
|
||||||
|
response = await ollama_client.chat(
|
||||||
|
model=self._model,
|
||||||
|
messages=messages,
|
||||||
|
tools=self.tools,
|
||||||
|
options=OLLAMA_OPTIONS,
|
||||||
|
think=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
if response.get("message", {}).get("tool_calls"):
|
||||||
|
messages.append(response["message"])
|
||||||
|
|
||||||
|
for tool_call in response["message"]["tool_calls"]:
|
||||||
|
if tool_call["function"]["name"] == "search_documents":
|
||||||
|
args = tool_call["function"]["arguments"]
|
||||||
|
query = args.get("query", question)
|
||||||
|
limit = int(args.get("limit", 3))
|
||||||
|
|
||||||
|
search_results = await self._client.search(query, limit=limit)
|
||||||
|
|
||||||
|
context_chunks = []
|
||||||
|
for chunk, score in search_results:
|
||||||
|
context_chunks.append(
|
||||||
|
f"Content: {chunk.content}\nScore: {score:.4f}"
|
||||||
|
)
|
||||||
|
|
||||||
|
context = "\n\n".join(context_chunks)
|
||||||
|
|
||||||
|
messages.append(
|
||||||
|
{
|
||||||
|
"role": "tool",
|
||||||
|
"content": context,
|
||||||
|
"tool_call_id": tool_call.get("id", "search_tool"),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# No tool calls, return the response
|
||||||
|
return response["message"]["content"]
|
||||||
|
|
||||||
|
# If we've exhausted max rounds, return empty string
|
||||||
|
return ""
|
||||||
100
src/haiku/rag/qa/openai.py
Normal file
100
src/haiku/rag/qa/openai.py
Normal file
|
|
@ -0,0 +1,100 @@
|
||||||
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
try:
|
||||||
|
from openai import AsyncOpenAI
|
||||||
|
from openai.types.chat import (
|
||||||
|
ChatCompletionAssistantMessageParam,
|
||||||
|
ChatCompletionMessageParam,
|
||||||
|
ChatCompletionSystemMessageParam,
|
||||||
|
ChatCompletionToolMessageParam,
|
||||||
|
ChatCompletionUserMessageParam,
|
||||||
|
)
|
||||||
|
from openai.types.chat.chat_completion_tool_param import ChatCompletionToolParam
|
||||||
|
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.qa.base import QuestionAnswerAgentBase
|
||||||
|
|
||||||
|
class QuestionAnswerOpenAIAgent(QuestionAnswerAgentBase):
|
||||||
|
def __init__(self, client: HaikuRAG, model: str = "gpt-4o-mini"):
|
||||||
|
super().__init__(client, model or self._model)
|
||||||
|
self.tools: Sequence[ChatCompletionToolParam] = [
|
||||||
|
ChatCompletionToolParam(tool) for tool in self.tools
|
||||||
|
]
|
||||||
|
|
||||||
|
async def answer(self, question: str) -> str:
|
||||||
|
openai_client = AsyncOpenAI()
|
||||||
|
|
||||||
|
messages: list[ChatCompletionMessageParam] = [
|
||||||
|
ChatCompletionSystemMessageParam(
|
||||||
|
role="system", content=self._system_prompt
|
||||||
|
),
|
||||||
|
ChatCompletionUserMessageParam(role="user", content=question),
|
||||||
|
]
|
||||||
|
|
||||||
|
max_rounds = 5 # Prevent infinite loops
|
||||||
|
|
||||||
|
for _ in range(max_rounds):
|
||||||
|
response = await openai_client.chat.completions.create(
|
||||||
|
model=self._model,
|
||||||
|
messages=messages,
|
||||||
|
tools=self.tools,
|
||||||
|
temperature=0.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
response_message = response.choices[0].message
|
||||||
|
|
||||||
|
if response_message.tool_calls:
|
||||||
|
messages.append(
|
||||||
|
ChatCompletionAssistantMessageParam(
|
||||||
|
role="assistant",
|
||||||
|
content=response_message.content,
|
||||||
|
tool_calls=[
|
||||||
|
{
|
||||||
|
"id": tc.id,
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": tc.function.name,
|
||||||
|
"arguments": tc.function.arguments,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for tc in response_message.tool_calls
|
||||||
|
],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for tool_call in response_message.tool_calls:
|
||||||
|
if tool_call.function.name == "search_documents":
|
||||||
|
import json
|
||||||
|
|
||||||
|
args = json.loads(tool_call.function.arguments)
|
||||||
|
query = args.get("query", question)
|
||||||
|
limit = int(args.get("limit", 3))
|
||||||
|
|
||||||
|
search_results = await self._client.search(
|
||||||
|
query, limit=limit
|
||||||
|
)
|
||||||
|
|
||||||
|
context_chunks = []
|
||||||
|
for chunk, score in search_results:
|
||||||
|
context_chunks.append(
|
||||||
|
f"Content: {chunk.content}\nScore: {score:.4f}"
|
||||||
|
)
|
||||||
|
|
||||||
|
context = "\n\n".join(context_chunks)
|
||||||
|
|
||||||
|
messages.append(
|
||||||
|
ChatCompletionToolMessageParam(
|
||||||
|
role="tool",
|
||||||
|
content=context,
|
||||||
|
tool_call_id=tool_call.id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# No tool calls, return the response
|
||||||
|
return response_message.content or ""
|
||||||
|
|
||||||
|
# If we've exhausted max rounds, return empty string
|
||||||
|
return ""
|
||||||
|
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
20
src/haiku/rag/qa/prompts.py
Normal file
20
src/haiku/rag/qa/prompts.py
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
SYSTEM_PROMPT = """
|
||||||
|
You are a knowledgeable assistant that helps users find information from a document knowledge base.
|
||||||
|
|
||||||
|
Your process:
|
||||||
|
1. When a user asks a question, use the search_documents tool to find relevant information
|
||||||
|
2. Search with specific keywords and phrases from the user's question
|
||||||
|
3. Review the search results and their relevance scores
|
||||||
|
4. If you need additional context, perform follow-up searches with different keywords
|
||||||
|
5. Provide a comprehensive answer based only on the retrieved documents
|
||||||
|
|
||||||
|
Guidelines:
|
||||||
|
- Base your answers strictly on the provided document content
|
||||||
|
- Quote or reference specific information when possible
|
||||||
|
- If multiple documents contain relevant information, synthesize them coherently
|
||||||
|
- Indicate when information is incomplete or when you need to search for additional context
|
||||||
|
- If the retrieved documents don't contain sufficient information, clearly state: "I cannot find enough information in the knowledge base to answer this question."
|
||||||
|
- For complex questions, consider breaking them down and performing multiple searches
|
||||||
|
|
||||||
|
Be concise, and always maintain accuracy over completeness. Prefer short, direct answers that are well-supported by the documents.
|
||||||
|
"""
|
||||||
|
|
@ -1,23 +1,65 @@
|
||||||
import sqlite3
|
import sqlite3
|
||||||
import struct
|
import struct
|
||||||
|
from importlib import metadata
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
|
|
||||||
import sqlite_vec
|
import sqlite_vec
|
||||||
|
from packaging.version import parse
|
||||||
|
from rich.console import Console
|
||||||
|
|
||||||
|
from haiku.rag.config import Config
|
||||||
from haiku.rag.embeddings import get_embedder
|
from haiku.rag.embeddings import get_embedder
|
||||||
|
from haiku.rag.store.upgrades import upgrades
|
||||||
|
from haiku.rag.utils import int_to_semantic_version, semantic_version_to_int
|
||||||
|
|
||||||
|
|
||||||
class Store:
|
class Store:
|
||||||
def __init__(self, db_path: Path | Literal[":memory:"]):
|
def __init__(
|
||||||
|
self, db_path: Path | Literal[":memory:"], skip_validation: bool = False
|
||||||
|
):
|
||||||
self.db_path: Path | Literal[":memory:"] = db_path
|
self.db_path: Path | Literal[":memory:"] = db_path
|
||||||
self._connection = self.create_db()
|
self.create_or_update_db()
|
||||||
|
|
||||||
def create_db(self) -> sqlite3.Connection:
|
# Validate config compatibility after connection is established
|
||||||
|
if not skip_validation:
|
||||||
|
from haiku.rag.store.repositories.settings import SettingsRepository
|
||||||
|
|
||||||
|
settings_repo = SettingsRepository(self)
|
||||||
|
settings_repo.validate_config_compatibility()
|
||||||
|
current_version = metadata.version("haiku.rag")
|
||||||
|
self.set_user_version(current_version)
|
||||||
|
|
||||||
|
def create_or_update_db(self):
|
||||||
"""Create the database and tables with sqlite-vec support for embeddings."""
|
"""Create the database and tables with sqlite-vec support for embeddings."""
|
||||||
|
current_version = metadata.version("haiku.rag")
|
||||||
|
|
||||||
db = sqlite3.connect(self.db_path)
|
db = sqlite3.connect(self.db_path)
|
||||||
db.enable_load_extension(True)
|
db.enable_load_extension(True)
|
||||||
sqlite_vec.load(db)
|
sqlite_vec.load(db)
|
||||||
|
self._connection = db
|
||||||
|
existing_tables = [
|
||||||
|
row[0]
|
||||||
|
for row in db.execute(
|
||||||
|
"SELECT name FROM sqlite_master WHERE type='table';"
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
|
||||||
|
# If we have a db already, perform upgrades and return
|
||||||
|
if self.db_path != ":memory:" and "documents" in existing_tables:
|
||||||
|
# Upgrade database
|
||||||
|
console = Console()
|
||||||
|
db_version = self.get_user_version()
|
||||||
|
for version, steps in upgrades:
|
||||||
|
if parse(current_version) >= parse(version) and parse(version) > parse(
|
||||||
|
db_version
|
||||||
|
):
|
||||||
|
for step in steps:
|
||||||
|
step(db)
|
||||||
|
console.print(
|
||||||
|
f"[green][b]DB Upgrade: [/b]{step.__doc__}[/green]"
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
# Create documents table
|
# Create documents table
|
||||||
db.execute("""
|
db.execute("""
|
||||||
|
|
@ -30,7 +72,6 @@ class Store:
|
||||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
)
|
)
|
||||||
""")
|
""")
|
||||||
|
|
||||||
# Create chunks table
|
# Create chunks table
|
||||||
db.execute("""
|
db.execute("""
|
||||||
CREATE TABLE IF NOT EXISTS chunks (
|
CREATE TABLE IF NOT EXISTS chunks (
|
||||||
|
|
@ -41,7 +82,6 @@ class Store:
|
||||||
FOREIGN KEY (document_id) REFERENCES documents (id) ON DELETE CASCADE
|
FOREIGN KEY (document_id) REFERENCES documents (id) ON DELETE CASCADE
|
||||||
)
|
)
|
||||||
""")
|
""")
|
||||||
|
|
||||||
# Create vector table for chunk embeddings
|
# Create vector table for chunk embeddings
|
||||||
embedder = get_embedder()
|
embedder = get_embedder()
|
||||||
db.execute(f"""
|
db.execute(f"""
|
||||||
|
|
@ -50,7 +90,6 @@ class Store:
|
||||||
embedding FLOAT[{embedder._vector_dim}]
|
embedding FLOAT[{embedder._vector_dim}]
|
||||||
)
|
)
|
||||||
""")
|
""")
|
||||||
|
|
||||||
# Create FTS5 table for full-text search
|
# Create FTS5 table for full-text search
|
||||||
db.execute("""
|
db.execute("""
|
||||||
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
|
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
|
||||||
|
|
@ -59,14 +98,61 @@ class Store:
|
||||||
content_rowid='id'
|
content_rowid='id'
|
||||||
)
|
)
|
||||||
""")
|
""")
|
||||||
|
# Create settings table for storing current configuration
|
||||||
|
db.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS settings (
|
||||||
|
id INTEGER PRIMARY KEY DEFAULT 1,
|
||||||
|
settings TEXT NOT NULL DEFAULT '{}'
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
# Save current settings to the new database
|
||||||
|
settings_json = Config.model_dump_json()
|
||||||
|
db.execute(
|
||||||
|
"INSERT OR IGNORE INTO settings (id, settings) VALUES (1, ?)",
|
||||||
|
(settings_json,),
|
||||||
|
)
|
||||||
# Create indexes for better performance
|
# Create indexes for better performance
|
||||||
db.execute(
|
db.execute(
|
||||||
"CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON chunks(document_id)"
|
"CREATE INDEX IF NOT EXISTS idx_chunks_document_id ON chunks(document_id)"
|
||||||
)
|
)
|
||||||
|
|
||||||
db.commit()
|
db.commit()
|
||||||
return db
|
|
||||||
|
def get_user_version(self) -> str:
|
||||||
|
"""Returns the SQLite user version"""
|
||||||
|
if self._connection is None:
|
||||||
|
raise ValueError("Store connection is not available")
|
||||||
|
|
||||||
|
cursor = self._connection.execute("PRAGMA user_version;")
|
||||||
|
version = cursor.fetchone()
|
||||||
|
return int_to_semantic_version(version[0])
|
||||||
|
|
||||||
|
def set_user_version(self, version: str) -> None:
|
||||||
|
"""Updates the SQLite user version"""
|
||||||
|
if self._connection is None:
|
||||||
|
raise ValueError("Store connection is not available")
|
||||||
|
|
||||||
|
self._connection.execute(
|
||||||
|
f"PRAGMA user_version = {semantic_version_to_int(version)};"
|
||||||
|
)
|
||||||
|
|
||||||
|
def recreate_embeddings_table(self) -> None:
|
||||||
|
"""Recreate the embeddings table with current vector dimensions."""
|
||||||
|
if self._connection is None:
|
||||||
|
raise ValueError("Store connection is not available")
|
||||||
|
|
||||||
|
# Drop existing embeddings table
|
||||||
|
self._connection.execute("DROP TABLE IF EXISTS chunk_embeddings")
|
||||||
|
|
||||||
|
# Recreate with current dimensions
|
||||||
|
embedder = get_embedder()
|
||||||
|
self._connection.execute(f"""
|
||||||
|
CREATE VIRTUAL TABLE chunk_embeddings USING vec0(
|
||||||
|
chunk_id INTEGER PRIMARY KEY,
|
||||||
|
embedding FLOAT[{embedder._vector_dim}]
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
|
||||||
|
self._connection.commit()
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def serialize_embedding(embedding: list[float]) -> bytes:
|
def serialize_embedding(embedding: list[float]) -> bytes:
|
||||||
|
|
|
||||||
|
|
@ -3,10 +3,12 @@ from pydantic import BaseModel
|
||||||
|
|
||||||
class Chunk(BaseModel):
|
class Chunk(BaseModel):
|
||||||
"""
|
"""
|
||||||
Represents a document with an ID, content, and metadata.
|
Represents a chunk with content, metadata, and optional document information.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
id: int | None = None
|
id: int | None = None
|
||||||
document_id: int
|
document_id: int
|
||||||
content: str
|
content: str
|
||||||
metadata: dict = {}
|
metadata: dict = {}
|
||||||
|
document_uri: str | None = None
|
||||||
|
document_meta: dict = {}
|
||||||
|
|
|
||||||
|
|
@ -208,6 +208,22 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
|
|
||||||
return created_chunks
|
return created_chunks
|
||||||
|
|
||||||
|
async def delete_all(self, commit: bool = True) -> bool:
|
||||||
|
"""Delete all chunks from the database."""
|
||||||
|
if self.store._connection is None:
|
||||||
|
raise ValueError("Store connection is not available")
|
||||||
|
|
||||||
|
cursor = self.store._connection.cursor()
|
||||||
|
|
||||||
|
cursor.execute("DELETE FROM chunks_fts")
|
||||||
|
cursor.execute("DELETE FROM chunk_embeddings")
|
||||||
|
cursor.execute("DELETE FROM chunks")
|
||||||
|
|
||||||
|
deleted = cursor.rowcount > 0
|
||||||
|
if commit:
|
||||||
|
self.store._connection.commit()
|
||||||
|
return deleted
|
||||||
|
|
||||||
async def delete_by_document_id(
|
async def delete_by_document_id(
|
||||||
self, document_id: int, commit: bool = True
|
self, document_id: int, commit: bool = True
|
||||||
) -> bool:
|
) -> bool:
|
||||||
|
|
@ -240,9 +256,10 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
# Search for similar chunks using sqlite-vec
|
# Search for similar chunks using sqlite-vec
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
"""
|
"""
|
||||||
SELECT c.id, c.document_id, c.content, c.metadata, distance
|
SELECT c.id, c.document_id, c.content, c.metadata, distance, d.uri, d.metadata as document_metadata
|
||||||
FROM chunk_embeddings
|
FROM chunk_embeddings
|
||||||
JOIN chunks c ON c.id = chunk_embeddings.chunk_id
|
JOIN chunks c ON c.id = chunk_embeddings.chunk_id
|
||||||
|
JOIN documents d ON c.document_id = d.id
|
||||||
WHERE embedding MATCH :embedding AND k = :k
|
WHERE embedding MATCH :embedding AND k = :k
|
||||||
ORDER BY distance
|
ORDER BY distance
|
||||||
""",
|
""",
|
||||||
|
|
@ -257,10 +274,14 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
document_id=document_id,
|
document_id=document_id,
|
||||||
content=content,
|
content=content,
|
||||||
metadata=json.loads(metadata_json) if metadata_json else {},
|
metadata=json.loads(metadata_json) if metadata_json else {},
|
||||||
|
document_uri=document_uri,
|
||||||
|
document_meta=json.loads(document_metadata_json)
|
||||||
|
if document_metadata_json
|
||||||
|
else {},
|
||||||
),
|
),
|
||||||
1.0 / (1.0 + distance),
|
1.0 / (1.0 + distance),
|
||||||
)
|
)
|
||||||
for chunk_id, document_id, content, metadata_json, distance in results
|
for chunk_id, document_id, content, metadata_json, distance, document_uri, document_metadata_json in results
|
||||||
]
|
]
|
||||||
|
|
||||||
async def search_chunks_fts(
|
async def search_chunks_fts(
|
||||||
|
|
@ -281,9 +302,10 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
# Search using FTS5
|
# Search using FTS5
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
"""
|
"""
|
||||||
SELECT c.id, c.document_id, c.content, c.metadata, rank
|
SELECT c.id, c.document_id, c.content, c.metadata, rank, d.uri, d.metadata as document_metadata
|
||||||
FROM chunks_fts
|
FROM chunks_fts
|
||||||
JOIN chunks c ON c.id = chunks_fts.rowid
|
JOIN chunks c ON c.id = chunks_fts.rowid
|
||||||
|
JOIN documents d ON c.document_id = d.id
|
||||||
WHERE chunks_fts MATCH :query
|
WHERE chunks_fts MATCH :query
|
||||||
ORDER BY rank
|
ORDER BY rank
|
||||||
LIMIT :limit
|
LIMIT :limit
|
||||||
|
|
@ -300,10 +322,14 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
document_id=document_id,
|
document_id=document_id,
|
||||||
content=content,
|
content=content,
|
||||||
metadata=json.loads(metadata_json) if metadata_json else {},
|
metadata=json.loads(metadata_json) if metadata_json else {},
|
||||||
|
document_uri=document_uri,
|
||||||
|
document_meta=json.loads(document_metadata_json)
|
||||||
|
if document_metadata_json
|
||||||
|
else {},
|
||||||
),
|
),
|
||||||
-rank,
|
-rank,
|
||||||
)
|
)
|
||||||
for chunk_id, document_id, content, metadata_json, rank in results
|
for chunk_id, document_id, content, metadata_json, rank, document_uri, document_metadata_json in results
|
||||||
# FTS5 rank is negative BM25 score
|
# FTS5 rank is negative BM25 score
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
@ -325,7 +351,6 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
words = re.findall(r"\b\w+\b", query.lower())
|
words = re.findall(r"\b\w+\b", query.lower())
|
||||||
# Join with OR to find chunks containing any of the keywords
|
# Join with OR to find chunks containing any of the keywords
|
||||||
fts_query = " OR ".join(words) if words else query
|
fts_query = " OR ".join(words) if words else query
|
||||||
|
|
||||||
# Perform hybrid search using RRF (Reciprocal Rank Fusion)
|
# Perform hybrid search using RRF (Reciprocal Rank Fusion)
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
"""
|
"""
|
||||||
|
|
@ -369,9 +394,10 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
LEFT JOIN vector_search v ON a.id = v.id
|
LEFT JOIN vector_search v ON a.id = v.id
|
||||||
LEFT JOIN fts_search f ON a.id = f.id
|
LEFT JOIN fts_search f ON a.id = f.id
|
||||||
)
|
)
|
||||||
SELECT id, document_id, content, metadata, rrf_score
|
SELECT r.id, r.document_id, r.content, r.metadata, r.rrf_score, d.uri, d.metadata as document_metadata
|
||||||
FROM rrf_scores
|
FROM rrf_scores r
|
||||||
ORDER BY rrf_score DESC
|
JOIN documents d ON r.document_id = d.id
|
||||||
|
ORDER BY r.rrf_score DESC
|
||||||
LIMIT :limit
|
LIMIT :limit
|
||||||
""",
|
""",
|
||||||
{
|
{
|
||||||
|
|
@ -391,10 +417,14 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
document_id=document_id,
|
document_id=document_id,
|
||||||
content=content,
|
content=content,
|
||||||
metadata=json.loads(metadata_json) if metadata_json else {},
|
metadata=json.loads(metadata_json) if metadata_json else {},
|
||||||
|
document_uri=document_uri,
|
||||||
|
document_meta=json.loads(document_metadata_json)
|
||||||
|
if document_metadata_json
|
||||||
|
else {},
|
||||||
),
|
),
|
||||||
rrf_score,
|
rrf_score,
|
||||||
)
|
)
|
||||||
for chunk_id, document_id, content, metadata_json, rrf_score in results
|
for chunk_id, document_id, content, metadata_json, rrf_score, document_uri, document_metadata_json in results
|
||||||
]
|
]
|
||||||
|
|
||||||
async def get_by_document_id(self, document_id: int) -> list[Chunk]:
|
async def get_by_document_id(self, document_id: int) -> list[Chunk]:
|
||||||
|
|
@ -405,9 +435,11 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
cursor = self.store._connection.cursor()
|
cursor = self.store._connection.cursor()
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
"""
|
"""
|
||||||
SELECT id, document_id, content, metadata
|
SELECT c.id, c.document_id, c.content, c.metadata, d.uri, d.metadata as document_metadata
|
||||||
FROM chunks WHERE document_id = :document_id
|
FROM chunks c
|
||||||
ORDER BY JSON_EXTRACT(metadata, '$.order')
|
JOIN documents d ON c.document_id = d.id
|
||||||
|
WHERE c.document_id = :document_id
|
||||||
|
ORDER BY JSON_EXTRACT(c.metadata, '$.order')
|
||||||
""",
|
""",
|
||||||
{"document_id": document_id},
|
{"document_id": document_id},
|
||||||
)
|
)
|
||||||
|
|
@ -419,6 +451,10 @@ class ChunkRepository(BaseRepository[Chunk]):
|
||||||
document_id=document_id,
|
document_id=document_id,
|
||||||
content=content,
|
content=content,
|
||||||
metadata=json.loads(metadata_json) if metadata_json else {},
|
metadata=json.loads(metadata_json) if metadata_json else {},
|
||||||
|
document_uri=document_uri,
|
||||||
|
document_meta=json.loads(document_metadata_json)
|
||||||
|
if document_metadata_json
|
||||||
|
else {},
|
||||||
)
|
)
|
||||||
for chunk_id, document_id, content, metadata_json in rows
|
for chunk_id, document_id, content, metadata_json, document_uri, document_metadata_json in rows
|
||||||
]
|
]
|
||||||
|
|
|
||||||
78
src/haiku/rag/store/repositories/settings.py
Normal file
78
src/haiku/rag/store/repositories/settings.py
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
import json
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from haiku.rag.store.engine import Store
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigMismatchError(Exception):
|
||||||
|
"""Raised when current config doesn't match stored settings."""
|
||||||
|
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class SettingsRepository:
|
||||||
|
def __init__(self, store: Store):
|
||||||
|
self.store = store
|
||||||
|
|
||||||
|
def get(self) -> dict[str, Any]:
|
||||||
|
"""Get all settings from the database."""
|
||||||
|
if self.store._connection is None:
|
||||||
|
raise ValueError("Store connection is not available")
|
||||||
|
|
||||||
|
cursor = self.store._connection.execute("SELECT settings FROM settings LIMIT 1")
|
||||||
|
row = cursor.fetchone()
|
||||||
|
if row:
|
||||||
|
return json.loads(row[0])
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def save(self) -> None:
|
||||||
|
"""Sync settings from the current AppConfig to database."""
|
||||||
|
if self.store._connection is None:
|
||||||
|
raise ValueError("Store connection is not available")
|
||||||
|
|
||||||
|
from haiku.rag.config import Config
|
||||||
|
|
||||||
|
settings_json = Config.model_dump_json()
|
||||||
|
|
||||||
|
self.store._connection.execute(
|
||||||
|
"INSERT INTO settings (id, settings) VALUES (1, ?) ON CONFLICT(id) DO UPDATE SET settings = excluded.settings",
|
||||||
|
(settings_json,),
|
||||||
|
)
|
||||||
|
|
||||||
|
self.store._connection.commit()
|
||||||
|
|
||||||
|
def validate_config_compatibility(self) -> None:
|
||||||
|
"""Check if current config is compatible with stored settings.
|
||||||
|
|
||||||
|
Raises ConfigMismatchError if there are incompatible differences.
|
||||||
|
If no settings exist, saves current config.
|
||||||
|
"""
|
||||||
|
db_settings = self.get()
|
||||||
|
if not db_settings:
|
||||||
|
# No settings in DB, save current config
|
||||||
|
self.save()
|
||||||
|
return
|
||||||
|
|
||||||
|
from haiku.rag.config import Config
|
||||||
|
|
||||||
|
current_config = Config.model_dump(mode="json")
|
||||||
|
|
||||||
|
# Critical settings that must match
|
||||||
|
critical_settings = [
|
||||||
|
"EMBEDDINGS_PROVIDER",
|
||||||
|
"EMBEDDINGS_MODEL",
|
||||||
|
"EMBEDDINGS_VECTOR_DIM",
|
||||||
|
"CHUNK_SIZE",
|
||||||
|
"CHUNK_OVERLAP",
|
||||||
|
]
|
||||||
|
|
||||||
|
errors = []
|
||||||
|
for setting in critical_settings:
|
||||||
|
if db_settings.get(setting) != current_config.get(setting):
|
||||||
|
errors.append(
|
||||||
|
f"{setting}: current={current_config.get(setting)}, stored={db_settings.get(setting)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
error_msg = f"Config mismatch detected: {'; '.join(errors)}. Consider rebuilding the database with the current configuration."
|
||||||
|
raise ConfigMismatchError(error_msg)
|
||||||
3
src/haiku/rag/store/upgrades/__init__.py
Normal file
3
src/haiku/rag/store/upgrades/__init__.py
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
from haiku.rag.store.upgrades.v0_3_4 import upgrades as v0_3_4_upgrades
|
||||||
|
|
||||||
|
upgrades = v0_3_4_upgrades
|
||||||
26
src/haiku/rag/store/upgrades/v0_3_4.py
Normal file
26
src/haiku/rag/store/upgrades/v0_3_4.py
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
from collections.abc import Callable
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from haiku.rag.config import Config
|
||||||
|
|
||||||
|
|
||||||
|
def add_settings_table(db: Connection) -> None:
|
||||||
|
"""Create settings table for storing current configuration"""
|
||||||
|
db.execute("""
|
||||||
|
CREATE TABLE settings (
|
||||||
|
id INTEGER PRIMARY KEY DEFAULT 1,
|
||||||
|
settings TEXT NOT NULL DEFAULT '{}'
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
|
||||||
|
settings_json = Config.model_dump_json()
|
||||||
|
db.execute(
|
||||||
|
"INSERT INTO settings (id, settings) VALUES (1, ?)",
|
||||||
|
(settings_json,),
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
upgrades: list[tuple[str, list[Callable[[Connection], None]]]] = [
|
||||||
|
("0.3.4", [add_settings_table])
|
||||||
|
]
|
||||||
|
|
@ -1,17 +1,20 @@
|
||||||
import sys
|
import sys
|
||||||
|
from importlib import metadata
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from packaging.version import Version, parse
|
||||||
|
|
||||||
|
|
||||||
def get_default_data_dir() -> Path:
|
def get_default_data_dir() -> Path:
|
||||||
"""
|
"""Get the user data directory for the current system platform.
|
||||||
Get the user data directory for the current system platform.
|
|
||||||
|
|
||||||
Linux: ~/.local/share/haiku.rag
|
Linux: ~/.local/share/haiku.rag
|
||||||
macOS: ~/Library/Application Support/haiku.rag
|
macOS: ~/Library/Application Support/haiku.rag
|
||||||
Windows: C:/Users/<USER>/AppData/Roaming/haiku.rag
|
Windows: C:/Users/<USER>/AppData/Roaming/haiku.rag
|
||||||
|
|
||||||
:return: User Data Path
|
Returns:
|
||||||
:rtype: Path
|
User Data Path.
|
||||||
"""
|
"""
|
||||||
home = Path.home()
|
home = Path.home()
|
||||||
|
|
||||||
|
|
@ -23,3 +26,54 @@ def get_default_data_dir() -> Path:
|
||||||
|
|
||||||
data_path = system_paths[sys.platform]
|
data_path = system_paths[sys.platform]
|
||||||
return data_path
|
return data_path
|
||||||
|
|
||||||
|
|
||||||
|
def semantic_version_to_int(version: str) -> int:
|
||||||
|
"""Convert a semantic version string to an integer.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
version: Semantic version string.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Integer representation of semantic version.
|
||||||
|
"""
|
||||||
|
major, minor, patch = version.split(".")
|
||||||
|
major = int(major) << 16
|
||||||
|
minor = int(minor) << 8
|
||||||
|
patch = int(patch)
|
||||||
|
return major + minor + patch
|
||||||
|
|
||||||
|
|
||||||
|
def int_to_semantic_version(version: int) -> str:
|
||||||
|
"""Convert an integer to a semantic version string.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
version: Integer representation of semantic version.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Semantic version string.
|
||||||
|
"""
|
||||||
|
major = version >> 16
|
||||||
|
minor = (version >> 8) & 255
|
||||||
|
patch = version & 255
|
||||||
|
return f"{major}.{minor}.{patch}"
|
||||||
|
|
||||||
|
|
||||||
|
async def is_up_to_date() -> tuple[bool, Version, Version]:
|
||||||
|
"""Check whether haiku.rag is current.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A tuple containing a boolean indicating whether haiku.rag is current,
|
||||||
|
the running version and the latest version.
|
||||||
|
"""
|
||||||
|
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
running_version = parse(metadata.version("haiku.rag"))
|
||||||
|
try:
|
||||||
|
response = await client.get("https://pypi.org/pypi/haiku.rag/json")
|
||||||
|
data = response.json()
|
||||||
|
pypi_version = parse(data["info"]["version"])
|
||||||
|
except Exception:
|
||||||
|
# If no network connection, do not raise alarms.
|
||||||
|
pypi_version = running_version
|
||||||
|
return running_version >= pypi_version, running_version, pypi_version
|
||||||
|
|
|
||||||
0
tests/__init__.py
Normal file
0
tests/__init__.py
Normal file
151
tests/generate_benchmark_db.py
Normal file
151
tests/generate_benchmark_db.py
Normal file
|
|
@ -0,0 +1,151 @@
|
||||||
|
import asyncio
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from datasets import Dataset, load_dataset
|
||||||
|
from llm_judge import LLMJudge
|
||||||
|
from rich.console import Console
|
||||||
|
from rich.progress import Progress
|
||||||
|
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.qa import get_qa_agent
|
||||||
|
|
||||||
|
console = Console()
|
||||||
|
|
||||||
|
db_path = Path(__file__).parent / "data" / "benchmark.sqlite"
|
||||||
|
|
||||||
|
|
||||||
|
async def populate_db():
|
||||||
|
ds: Dataset = load_dataset("ServiceNow/repliqa")["repliqa_3"] # type: ignore
|
||||||
|
corpus = ds.filter(lambda doc: doc["document_topic"] == "News Stories")
|
||||||
|
|
||||||
|
with Progress() as progress:
|
||||||
|
task = progress.add_task("[green]Populating database...", total=len(corpus))
|
||||||
|
|
||||||
|
async with HaikuRAG(db_path) as rag:
|
||||||
|
for doc in corpus:
|
||||||
|
uri = doc["document_id"] # type: ignore
|
||||||
|
existing_doc = await rag.get_document_by_uri(uri)
|
||||||
|
if existing_doc is not None:
|
||||||
|
progress.advance(task)
|
||||||
|
continue
|
||||||
|
|
||||||
|
await rag.create_document(
|
||||||
|
content=doc["document_extracted"], # type: ignore
|
||||||
|
uri=uri,
|
||||||
|
)
|
||||||
|
progress.advance(task)
|
||||||
|
|
||||||
|
|
||||||
|
async def run_match_benchmark():
|
||||||
|
ds: Dataset = load_dataset("ServiceNow/repliqa")["repliqa_3"] # type: ignore
|
||||||
|
corpus = ds.filter(lambda doc: doc["document_topic"] == "News Stories")
|
||||||
|
|
||||||
|
correct_at_1 = 0
|
||||||
|
correct_at_2 = 0
|
||||||
|
correct_at_3 = 0
|
||||||
|
total_queries = 0
|
||||||
|
|
||||||
|
with Progress() as progress:
|
||||||
|
task = progress.add_task(
|
||||||
|
"[blue]Running retrieval benchmark...", total=len(corpus)
|
||||||
|
)
|
||||||
|
|
||||||
|
async with HaikuRAG(db_path) as rag:
|
||||||
|
for doc in corpus:
|
||||||
|
doc_id = doc["document_id"] # type: ignore
|
||||||
|
matches = await rag.search(
|
||||||
|
query=doc["question"], # type: ignore
|
||||||
|
limit=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
total_queries += 1
|
||||||
|
|
||||||
|
# Check position of correct document in results
|
||||||
|
for position, (chunk, _) in enumerate(matches):
|
||||||
|
retrieved = await rag.get_document_by_id(chunk.document_id)
|
||||||
|
if retrieved and retrieved.uri == doc_id:
|
||||||
|
if position == 0: # First position
|
||||||
|
correct_at_1 += 1
|
||||||
|
correct_at_2 += 1
|
||||||
|
correct_at_3 += 1
|
||||||
|
elif position == 1: # Second position
|
||||||
|
correct_at_2 += 1
|
||||||
|
correct_at_3 += 1
|
||||||
|
elif position == 2: # Third position
|
||||||
|
correct_at_3 += 1
|
||||||
|
break
|
||||||
|
|
||||||
|
progress.advance(task)
|
||||||
|
|
||||||
|
# Calculate recall metrics
|
||||||
|
recall_at_1 = correct_at_1 / total_queries
|
||||||
|
recall_at_2 = correct_at_2 / total_queries
|
||||||
|
recall_at_3 = correct_at_3 / total_queries
|
||||||
|
|
||||||
|
console.print("\n=== Retrieval Benchmark Results ===", style="bold cyan")
|
||||||
|
console.print(f"Total queries: {total_queries}")
|
||||||
|
console.print(f"Recall@1: {recall_at_1:.4f}")
|
||||||
|
console.print(f"Recall@2: {recall_at_2:.4f}")
|
||||||
|
console.print(f"Recall@3: {recall_at_3:.4f}")
|
||||||
|
|
||||||
|
return {"recall@1": recall_at_1, "recall@2": recall_at_2, "recall@3": recall_at_3}
|
||||||
|
|
||||||
|
|
||||||
|
async def run_qa_benchmark(k: int | None = None):
|
||||||
|
"""Run QA benchmarking on the corpus."""
|
||||||
|
ds: Dataset = load_dataset("ServiceNow/repliqa")["repliqa_3"] # type: ignore
|
||||||
|
corpus = ds.filter(lambda doc: doc["document_topic"] == "News Stories")
|
||||||
|
|
||||||
|
if k is not None:
|
||||||
|
corpus = corpus.select(range(min(k, len(corpus))))
|
||||||
|
|
||||||
|
judge = LLMJudge()
|
||||||
|
correct_answers = 0
|
||||||
|
total_questions = 0
|
||||||
|
|
||||||
|
with Progress() as progress:
|
||||||
|
task = progress.add_task("[yellow]Running QA benchmark...", total=len(corpus))
|
||||||
|
|
||||||
|
async with HaikuRAG(db_path) as rag:
|
||||||
|
qa = get_qa_agent(rag)
|
||||||
|
|
||||||
|
for doc in corpus:
|
||||||
|
question = doc["question"] # type: ignore
|
||||||
|
expected_answer = doc["answer"] # type: ignore
|
||||||
|
|
||||||
|
generated_answer = await qa.answer(question)
|
||||||
|
is_equivalent = await judge.judge_answers(
|
||||||
|
question, generated_answer, expected_answer
|
||||||
|
)
|
||||||
|
console.print(f"Question: {question}")
|
||||||
|
console.print(f"Expected: {expected_answer}")
|
||||||
|
console.print(f"Generated: {generated_answer}")
|
||||||
|
console.print(f"Equivalent: {is_equivalent}\n")
|
||||||
|
|
||||||
|
if is_equivalent:
|
||||||
|
correct_answers += 1
|
||||||
|
total_questions += 1
|
||||||
|
console.print("Current score:", correct_answers, "/", total_questions)
|
||||||
|
|
||||||
|
progress.advance(task)
|
||||||
|
|
||||||
|
accuracy = correct_answers / total_questions if total_questions > 0 else 0
|
||||||
|
|
||||||
|
console.print("\n=== QA Benchmark Results ===", style="bold cyan")
|
||||||
|
console.print(f"Total questions: {total_questions}")
|
||||||
|
console.print(f"Correct answers: {correct_answers}")
|
||||||
|
console.print(f"QA Accuracy: {accuracy:.4f} ({accuracy * 100:.2f}%)")
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
await populate_db()
|
||||||
|
|
||||||
|
console.print("Running retrieval benchmarks...", style="bold blue")
|
||||||
|
await run_match_benchmark()
|
||||||
|
|
||||||
|
console.print("\nRunning QA benchmarks...", style="bold yellow")
|
||||||
|
await run_qa_benchmark()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
69
tests/llm_judge.py
Normal file
69
tests/llm_judge.py
Normal file
|
|
@ -0,0 +1,69 @@
|
||||||
|
import json
|
||||||
|
|
||||||
|
from ollama import AsyncClient
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
from haiku.rag.config import Config
|
||||||
|
|
||||||
|
|
||||||
|
class LLMJudgeResponseSchema(BaseModel):
|
||||||
|
equivalent: bool
|
||||||
|
|
||||||
|
|
||||||
|
class LLMJudge:
|
||||||
|
"""LLM-as-judge for evaluating answer equivalence using Ollama."""
|
||||||
|
|
||||||
|
def __init__(self, model: str = "qwen3"):
|
||||||
|
self.model = model
|
||||||
|
self.client = AsyncClient(host=Config.OLLAMA_BASE_URL)
|
||||||
|
|
||||||
|
async def judge_answers(
|
||||||
|
self, question: str, answer: str, expected_answer: str
|
||||||
|
) -> bool:
|
||||||
|
"""
|
||||||
|
Judge whether two answers are equivalent for a given question.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
question: The original question
|
||||||
|
answer: The generated answer to evaluate
|
||||||
|
expected_answer: The reference/expected answer
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dictionary with judgment result:
|
||||||
|
- equivalent: bool indicating if answers are equivalent
|
||||||
|
- explanation: str explaining the reasoning
|
||||||
|
- score: str rating from 1-5
|
||||||
|
"""
|
||||||
|
|
||||||
|
prompt = f"""
|
||||||
|
You are an expert judge evaluating the equivalence of two answers to the same question.
|
||||||
|
|
||||||
|
Question: {question}
|
||||||
|
|
||||||
|
Generated Answer: {answer}
|
||||||
|
|
||||||
|
Expected Answer: {expected_answer}
|
||||||
|
|
||||||
|
Your task is to determine if these two answers are equivalent in meaning and both correctly answer the question. Consider:
|
||||||
|
|
||||||
|
1. Do both answers provide the same answer?
|
||||||
|
2. Do both answers directly address the question asked?
|
||||||
|
3. Minor differences in wording or style are acceptable if the meaning of the answer is the same.
|
||||||
|
4. If one answer is more detailed but the other is correct, they can still be considered equivalent.
|
||||||
|
|
||||||
|
Be strict but fair in your evaluation. Focus on factual correctness and whether both answers would satisfy someone asking the question."""
|
||||||
|
|
||||||
|
response = await self.client.chat(
|
||||||
|
model=self.model,
|
||||||
|
messages=[{"role": "user", "content": prompt}],
|
||||||
|
format=LLMJudgeResponseSchema.model_json_schema(),
|
||||||
|
think=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
answer = response["message"]["content"].strip()
|
||||||
|
try:
|
||||||
|
res = json.loads(answer)
|
||||||
|
assert "equivalent" in res, "Response must contain 'equivalent' key"
|
||||||
|
return res["equivalent"]
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
assert False, "Response is not valid JSON"
|
||||||
208
tests/test_app.py
Normal file
208
tests/test_app.py
Normal file
|
|
@ -0,0 +1,208 @@
|
||||||
|
import asyncio
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from haiku.rag.app import HaikuRAGApp
|
||||||
|
from haiku.rag.store.models.document import Document
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def app():
|
||||||
|
return HaikuRAGApp(db_path=Path(":memory:"))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_list_documents(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test listing documents."""
|
||||||
|
mock_docs = [
|
||||||
|
Document(id=1, content="doc 1"),
|
||||||
|
Document(id=2, content="doc 2"),
|
||||||
|
]
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.list_documents.return_value = mock_docs
|
||||||
|
# The async context manager should return the mock client itself
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_rich_print = MagicMock()
|
||||||
|
mock_console_print = MagicMock()
|
||||||
|
monkeypatch.setattr(app, "_rich_print_document", mock_rich_print)
|
||||||
|
monkeypatch.setattr(app.console, "print", mock_console_print)
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.list_documents()
|
||||||
|
|
||||||
|
mock_client.list_documents.assert_called_once()
|
||||||
|
assert mock_rich_print.call_count == len(mock_docs)
|
||||||
|
mock_rich_print.assert_any_call(mock_docs[0], truncate=True)
|
||||||
|
mock_rich_print.assert_any_call(mock_docs[1], truncate=True)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_add_document_from_text(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test adding a document from text."""
|
||||||
|
mock_doc = Document(id=1, content="test document")
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.create_document.return_value = mock_doc
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_rich_print = MagicMock()
|
||||||
|
mock_print = MagicMock()
|
||||||
|
monkeypatch.setattr(app, "_rich_print_document", mock_rich_print)
|
||||||
|
monkeypatch.setattr(app.console, "print", mock_print)
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.add_document_from_text("test document")
|
||||||
|
|
||||||
|
mock_client.create_document.assert_called_once_with("test document")
|
||||||
|
mock_rich_print.assert_called_once_with(mock_doc, truncate=True)
|
||||||
|
mock_print.assert_called_once_with(
|
||||||
|
"[b]Document with id [cyan]1[/cyan] added successfully.[/b]"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_add_document_from_source(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test adding a document from a source path."""
|
||||||
|
mock_doc = Document(id=1, content="test document")
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.create_document_from_source.return_value = mock_doc
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_rich_print = MagicMock()
|
||||||
|
mock_print = MagicMock()
|
||||||
|
monkeypatch.setattr(app, "_rich_print_document", mock_rich_print)
|
||||||
|
monkeypatch.setattr(app.console, "print", mock_print)
|
||||||
|
|
||||||
|
file_path = Path("test.txt")
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.add_document_from_source(file_path)
|
||||||
|
|
||||||
|
mock_client.create_document_from_source.assert_called_once_with(file_path)
|
||||||
|
mock_rich_print.assert_called_once_with(mock_doc, truncate=True)
|
||||||
|
mock_print.assert_called_once_with(
|
||||||
|
"[b]Document with id [cyan]1[/cyan] added successfully.[/b]"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_get_document(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test getting a document."""
|
||||||
|
mock_doc = Document(id=1, content="test document")
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.get_document_by_id.return_value = mock_doc
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_rich_print = MagicMock()
|
||||||
|
monkeypatch.setattr(app, "_rich_print_document", mock_rich_print)
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.get_document(1)
|
||||||
|
|
||||||
|
mock_client.get_document_by_id.assert_called_once_with(1)
|
||||||
|
mock_rich_print.assert_called_once_with(mock_doc, truncate=False)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_get_document_not_found(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test getting a document that does not exist."""
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.get_document_by_id.return_value = None
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_print = MagicMock()
|
||||||
|
monkeypatch.setattr(app.console, "print", mock_print)
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.get_document(1)
|
||||||
|
|
||||||
|
mock_client.get_document_by_id.assert_called_once_with(1)
|
||||||
|
mock_print.assert_called_once_with("[red]Document with id 1 not found.[/red]")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_delete_document(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test deleting a document."""
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_print = MagicMock()
|
||||||
|
monkeypatch.setattr(app.console, "print", mock_print)
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.delete_document(1)
|
||||||
|
|
||||||
|
mock_client.delete_document.assert_called_once_with(1)
|
||||||
|
mock_print.assert_called_once_with("[b]Document 1 deleted successfully.[/b]")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_search(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test searching for documents."""
|
||||||
|
mock_results = [("chunk1", 0.9), ("chunk2", 0.8)]
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.search.return_value = mock_results
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_rich_print_search = MagicMock()
|
||||||
|
monkeypatch.setattr(app, "_rich_print_search_result", mock_rich_print_search)
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.search("query")
|
||||||
|
|
||||||
|
mock_client.search.assert_called_once_with("query", limit=5, k=60)
|
||||||
|
assert mock_rich_print_search.call_count == len(mock_results)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_search_no_results(app: HaikuRAGApp, monkeypatch):
|
||||||
|
"""Test searching with no results."""
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.search.return_value = []
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
mock_print = MagicMock()
|
||||||
|
monkeypatch.setattr(app.console, "print", mock_print)
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
await app.search("query")
|
||||||
|
|
||||||
|
mock_client.search.assert_called_once_with("query", limit=5, k=60)
|
||||||
|
mock_print.assert_called_once_with("[red]No results found.[/red]")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
@pytest.mark.parametrize("transport", ["stdio", "sse", "http", None])
|
||||||
|
async def test_serve(app: HaikuRAGApp, monkeypatch, transport):
|
||||||
|
"""Test the serve method with different transports."""
|
||||||
|
mock_server = AsyncMock()
|
||||||
|
mock_watcher = MagicMock()
|
||||||
|
mock_task = asyncio.create_task(asyncio.sleep(0))
|
||||||
|
mock_task.cancel = MagicMock()
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"haiku.rag.app.create_mcp_server", MagicMock(return_value=mock_server)
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"haiku.rag.app.FileWatcher", MagicMock(return_value=mock_watcher)
|
||||||
|
)
|
||||||
|
monkeypatch.setattr("asyncio.create_task", MagicMock(return_value=mock_task))
|
||||||
|
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_client.__aenter__.return_value = mock_client
|
||||||
|
|
||||||
|
with patch("haiku.rag.app.HaikuRAG", return_value=mock_client):
|
||||||
|
if transport:
|
||||||
|
await app.serve(transport=transport)
|
||||||
|
else:
|
||||||
|
await app.serve()
|
||||||
|
|
||||||
|
if transport == "stdio":
|
||||||
|
mock_server.run_stdio_async.assert_called_once()
|
||||||
|
elif transport == "sse":
|
||||||
|
mock_server.run_sse_async.assert_called_once_with("sse")
|
||||||
|
else:
|
||||||
|
mock_server.run_http_async.assert_called_once_with("streamable-http")
|
||||||
|
|
||||||
|
mock_task.cancel.assert_called_once()
|
||||||
129
tests/test_cli.py
Normal file
129
tests/test_cli.py
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
from typer.testing import CliRunner
|
||||||
|
|
||||||
|
from haiku.rag.cli import cli
|
||||||
|
|
||||||
|
runner = CliRunner()
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_documents():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.list_documents = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["list"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.list_documents.assert_called_once()
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_document_text():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.add_document_from_text = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["add", "test document"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.add_document_from_text.assert_called_once_with(
|
||||||
|
text="test document"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_add_document_src():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.add_document_from_source = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["add-src", "test.txt"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.add_document_from_source.assert_called_once()
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_document():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.get_document = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["get", "1"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.get_document.assert_called_once_with(doc_id=1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_document():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.delete_document = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["delete", "1"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.delete_document.assert_called_once_with(doc_id=1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_search():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.search = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["search", "query"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.search.assert_called_once_with(query="query", limit=5, k=60)
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.serve = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["serve"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.serve.assert_called_once_with(transport=None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve_stdio():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.serve = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["serve", "--stdio"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.serve.assert_called_once_with(transport="stdio")
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve_sse():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.serve = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["serve", "--sse"])
|
||||||
|
|
||||||
|
assert result.exit_code == 0
|
||||||
|
mock_app_instance.serve.assert_called_once_with(transport="sse")
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve_stdio_and_sse():
|
||||||
|
with patch("haiku.rag.cli.HaikuRAGApp") as mock_app:
|
||||||
|
mock_app_instance = MagicMock()
|
||||||
|
mock_app_instance.serve = AsyncMock()
|
||||||
|
mock_app.return_value = mock_app_instance
|
||||||
|
|
||||||
|
result = runner.invoke(cli, ["serve", "--stdio", "--sse"])
|
||||||
|
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "Error: Cannot use both --stdio and --http options" in result.stdout
|
||||||
|
|
@ -12,9 +12,7 @@ from haiku.rag.client import HaikuRAG
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_document_crud(qa_corpus: Dataset):
|
async def test_client_document_crud(qa_corpus: Dataset):
|
||||||
"""Test HaikuRAG CRUD operations for documents."""
|
"""Test HaikuRAG CRUD operations for documents."""
|
||||||
# Create client with in-memory database
|
async with HaikuRAG(":memory:") as client:
|
||||||
client = HaikuRAG(":memory:")
|
|
||||||
|
|
||||||
# Get test data
|
# Get test data
|
||||||
first_doc = qa_corpus[0]
|
first_doc = qa_corpus[0]
|
||||||
document_text = first_doc["document_extracted"]
|
document_text = first_doc["document_extracted"]
|
||||||
|
|
@ -76,30 +74,22 @@ async def test_client_document_crud(qa_corpus: Dataset):
|
||||||
deleted_again = await client.delete_document(created_doc.id)
|
deleted_again = await client.delete_document(created_doc.id)
|
||||||
assert deleted_again is False
|
assert deleted_again is False
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_document_from_source():
|
async def test_client_create_document_from_source():
|
||||||
"""Test creating a document from a file source."""
|
"""Test creating a document from a file source."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
# Create a temporary text file
|
|
||||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
|
|
||||||
test_content = "This is test content from a file."
|
test_content = "This is test content from a file."
|
||||||
f.write(test_content)
|
temp_path = Path(temp_dir) / "test.txt"
|
||||||
temp_path = Path(f.name)
|
temp_path.write_text(test_content)
|
||||||
|
|
||||||
try:
|
|
||||||
# Test create_document_from_source with Path
|
# Test create_document_from_source with Path
|
||||||
doc = await client.create_document_from_source(
|
doc = await client.create_document_from_source(source=temp_path)
|
||||||
source=temp_path, metadata={"source_type": "file"}
|
|
||||||
)
|
|
||||||
|
|
||||||
assert doc.id is not None
|
assert doc.id is not None
|
||||||
assert doc.content == test_content
|
assert doc.content == test_content
|
||||||
assert doc.uri == temp_path.as_uri()
|
assert doc.uri == temp_path.as_uri()
|
||||||
assert doc.metadata["source_type"] == "file"
|
|
||||||
assert "contentType" in doc.metadata
|
assert "contentType" in doc.metadata
|
||||||
assert "md5" in doc.metadata
|
assert "md5" in doc.metadata
|
||||||
assert doc.metadata["contentType"] == "text/plain"
|
assert doc.metadata["contentType"] == "text/plain"
|
||||||
|
|
@ -113,17 +103,11 @@ async def test_client_create_document_from_source():
|
||||||
assert "contentType" in doc2.metadata
|
assert "contentType" in doc2.metadata
|
||||||
assert "md5" in doc2.metadata
|
assert "md5" in doc2.metadata
|
||||||
|
|
||||||
finally:
|
|
||||||
# Clean up
|
|
||||||
temp_path.unlink()
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_document_from_source_unsupported():
|
async def test_client_create_document_from_source_unsupported():
|
||||||
"""Test creating a document from an unsupported file type."""
|
"""Test creating a document from an unsupported file type."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Create a temporary file with unsupported extension
|
# Create a temporary file with unsupported extension
|
||||||
with tempfile.NamedTemporaryFile(
|
with tempfile.NamedTemporaryFile(
|
||||||
mode="w", suffix=".unsupported", delete=False
|
mode="w", suffix=".unsupported", delete=False
|
||||||
|
|
@ -131,35 +115,26 @@ async def test_client_create_document_from_source_unsupported():
|
||||||
f.write("content")
|
f.write("content")
|
||||||
temp_path = Path(f.name)
|
temp_path = Path(f.name)
|
||||||
|
|
||||||
try:
|
|
||||||
# Should raise ValueError for unsupported extension
|
# Should raise ValueError for unsupported extension
|
||||||
with pytest.raises(ValueError, match="Unsupported file extension"):
|
with pytest.raises(ValueError, match="Unsupported file extension"):
|
||||||
await client.create_document_from_source(temp_path)
|
await client.create_document_from_source(temp_path)
|
||||||
|
|
||||||
finally:
|
|
||||||
temp_path.unlink()
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_document_from_source_nonexistent():
|
async def test_client_create_document_from_source_nonexistent():
|
||||||
"""Test creating a document from a non-existent file."""
|
"""Test creating a document from a non-existent file."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
non_existent_path = Path("/non/existent/file.txt")
|
non_existent_path = Path("/non/existent/file.txt")
|
||||||
|
|
||||||
# Should raise ValueError when file doesn't exist
|
# Should raise ValueError when file doesn't exist
|
||||||
with pytest.raises(ValueError, match="File does not exist"):
|
with pytest.raises(ValueError, match="File does not exist"):
|
||||||
await client.create_document_from_source(non_existent_path)
|
await client.create_document_from_source(non_existent_path)
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_document_from_url():
|
async def test_client_create_document_from_url():
|
||||||
"""Test creating a document from a URL."""
|
"""Test creating a document from a URL."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Mock the HTTP response
|
# Mock the HTTP response
|
||||||
mock_response = AsyncMock()
|
mock_response = AsyncMock()
|
||||||
mock_response.content = b"<html><body><h1>Test Page</h1><p>This is test content from a webpage.</p></body></html>"
|
mock_response.content = b"<html><body><h1>Test Page</h1><p>This is test content from a webpage.</p></body></html>"
|
||||||
|
|
@ -180,14 +155,11 @@ async def test_client_create_document_from_url():
|
||||||
assert "md5" in doc.metadata
|
assert "md5" in doc.metadata
|
||||||
assert doc.metadata["contentType"] == "text/html"
|
assert doc.metadata["contentType"] == "text/html"
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_document_from_url_with_different_content_types():
|
async def test_client_create_document_from_url_with_different_content_types():
|
||||||
"""Test creating documents from URLs with different content types."""
|
"""Test creating documents from URLs with different content types."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Test JSON content
|
# Test JSON content
|
||||||
mock_json_response = AsyncMock()
|
mock_json_response = AsyncMock()
|
||||||
mock_json_response.content = (
|
mock_json_response.content = (
|
||||||
|
|
@ -215,7 +187,9 @@ async def test_client_create_document_from_url_with_different_content_types():
|
||||||
mock_text_response.raise_for_status = AsyncMock()
|
mock_text_response.raise_for_status = AsyncMock()
|
||||||
|
|
||||||
with patch("httpx.AsyncClient.get", return_value=mock_text_response):
|
with patch("httpx.AsyncClient.get", return_value=mock_text_response):
|
||||||
doc = await client.create_document_from_source("https://example.com/readme.txt")
|
doc = await client.create_document_from_source(
|
||||||
|
"https://example.com/readme.txt"
|
||||||
|
)
|
||||||
|
|
||||||
assert doc.id is not None
|
assert doc.id is not None
|
||||||
assert doc.content == "This is plain text content from a URL."
|
assert doc.content == "This is plain text content from a URL."
|
||||||
|
|
@ -224,14 +198,11 @@ async def test_client_create_document_from_url_with_different_content_types():
|
||||||
assert "md5" in doc.metadata
|
assert "md5" in doc.metadata
|
||||||
assert doc.metadata["contentType"] == "text/plain"
|
assert doc.metadata["contentType"] == "text/plain"
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_document_from_url_unsupported_content():
|
async def test_client_create_document_from_url_unsupported_content():
|
||||||
"""Test creating a document from URL with unsupported content type."""
|
"""Test creating a document from URL with unsupported content type."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Mock response with unsupported content type
|
# Mock response with unsupported content type
|
||||||
mock_response = AsyncMock()
|
mock_response = AsyncMock()
|
||||||
mock_response.content = b"binary content"
|
mock_response.content = b"binary content"
|
||||||
|
|
@ -240,16 +211,15 @@ async def test_client_create_document_from_url_unsupported_content():
|
||||||
|
|
||||||
with patch("httpx.AsyncClient.get", return_value=mock_response):
|
with patch("httpx.AsyncClient.get", return_value=mock_response):
|
||||||
with pytest.raises(ValueError, match="Unsupported content type"):
|
with pytest.raises(ValueError, match="Unsupported content type"):
|
||||||
await client.create_document_from_source("https://example.com/binary.bin")
|
await client.create_document_from_source(
|
||||||
|
"https://example.com/binary.bin"
|
||||||
client.close()
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_document_from_url_http_error():
|
async def test_client_create_document_from_url_http_error():
|
||||||
"""Test handling HTTP errors when creating document from URL."""
|
"""Test handling HTTP errors when creating document from URL."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
with patch("httpx.AsyncClient.get") as mock_get:
|
with patch("httpx.AsyncClient.get") as mock_get:
|
||||||
mock_get.side_effect = httpx.HTTPStatusError(
|
mock_get.side_effect = httpx.HTTPStatusError(
|
||||||
"404 Not Found",
|
"404 Not Found",
|
||||||
|
|
@ -262,20 +232,22 @@ async def test_client_create_document_from_url_http_error():
|
||||||
"https://example.com/notfound.html"
|
"https://example.com/notfound.html"
|
||||||
)
|
)
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_get_extension_from_content_type_or_url():
|
async def test_get_extension_from_content_type_or_url():
|
||||||
"""Test the helper method for determining file extensions."""
|
"""Test the helper method for determining file extensions."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Test content type mappings
|
# Test content type mappings
|
||||||
assert client._get_extension_from_content_type_or_url("", "text/html") == ".html"
|
|
||||||
assert (
|
assert (
|
||||||
client._get_extension_from_content_type_or_url("", "application/pdf") == ".pdf"
|
client._get_extension_from_content_type_or_url("", "text/html") == ".html"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
client._get_extension_from_content_type_or_url("", "application/pdf")
|
||||||
|
== ".pdf"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
client._get_extension_from_content_type_or_url("", "text/plain") == ".txt"
|
||||||
)
|
)
|
||||||
assert client._get_extension_from_content_type_or_url("", "text/plain") == ".txt"
|
|
||||||
|
|
||||||
# Test URL extension detection
|
# Test URL extension detection
|
||||||
assert (
|
assert (
|
||||||
|
|
@ -305,25 +277,21 @@ async def test_get_extension_from_content_type_or_url():
|
||||||
== ".pdf"
|
== ".pdf"
|
||||||
)
|
)
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_metadata_content_type_and_md5():
|
async def test_client_metadata_content_type_and_md5():
|
||||||
"""Test that contentType and md5 metadata are correctly set."""
|
"""Test that contentType and md5 metadata are correctly set."""
|
||||||
import hashlib
|
import hashlib
|
||||||
|
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Create a temporary file with known content
|
# Create a temporary file with known content
|
||||||
test_content = "Test content for MD5 calculation."
|
test_content = "Test content for MD5 calculation."
|
||||||
expected_md5 = hashlib.md5(test_content.encode()).hexdigest()
|
expected_md5 = hashlib.md5(test_content.encode()).hexdigest()
|
||||||
|
|
||||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
f.write(test_content)
|
temp_path = Path(temp_dir) / "test.txt"
|
||||||
temp_path = Path(f.name)
|
temp_path.write_text(test_content)
|
||||||
|
|
||||||
try:
|
|
||||||
doc = await client.create_document_from_source(temp_path)
|
doc = await client.create_document_from_source(temp_path)
|
||||||
|
|
||||||
assert doc.metadata["contentType"] == "text/plain"
|
assert doc.metadata["contentType"] == "text/plain"
|
||||||
|
|
@ -342,23 +310,17 @@ async def test_client_metadata_content_type_and_md5():
|
||||||
assert url_doc.metadata["contentType"] == "text/plain"
|
assert url_doc.metadata["contentType"] == "text/plain"
|
||||||
assert url_doc.metadata["md5"] == expected_md5
|
assert url_doc.metadata["md5"] == expected_md5
|
||||||
|
|
||||||
finally:
|
|
||||||
temp_path.unlink()
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_create_update_no_op_behavior():
|
async def test_client_create_update_no_op_behavior():
|
||||||
"""Test create/update/no-op behavior based on MD5 changes."""
|
"""Test create/update/no-op behavior based on MD5 changes."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Create a temporary file
|
# Create a temporary file
|
||||||
test_content = "Original content for testing."
|
test_content = "Original content for testing."
|
||||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
f.write(test_content)
|
temp_path = Path(temp_dir) / "test.txt"
|
||||||
temp_path = Path(f.name)
|
temp_path.write_text(test_content)
|
||||||
|
|
||||||
try:
|
|
||||||
# First call - should create new document
|
# First call - should create new document
|
||||||
doc1 = await client.create_document_from_source(temp_path)
|
doc1 = await client.create_document_from_source(temp_path)
|
||||||
assert doc1.id is not None
|
assert doc1.id is not None
|
||||||
|
|
@ -384,16 +346,11 @@ async def test_client_create_update_no_op_behavior():
|
||||||
assert retrieved_doc is not None
|
assert retrieved_doc is not None
|
||||||
assert retrieved_doc.content == updated_content
|
assert retrieved_doc.content == updated_content
|
||||||
|
|
||||||
finally:
|
|
||||||
temp_path.unlink()
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_url_create_update_no_op_behavior():
|
async def test_client_url_create_update_no_op_behavior():
|
||||||
"""Test create/update/no-op behavior for URLs based on MD5 changes."""
|
"""Test create/update/no-op behavior for URLs based on MD5 changes."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
url = "https://example.com/test.txt"
|
url = "https://example.com/test.txt"
|
||||||
original_content = b"Original URL content"
|
original_content = b"Original URL content"
|
||||||
updated_content = b"Updated URL content"
|
updated_content = b"Updated URL content"
|
||||||
|
|
@ -425,14 +382,11 @@ async def test_client_url_create_update_no_op_behavior():
|
||||||
assert doc3.id == original_id # Same document ID
|
assert doc3.id == original_id # Same document ID
|
||||||
assert doc3.content == updated_content.decode() # Updated content
|
assert doc3.content == updated_content.decode() # Updated content
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_search():
|
async def test_client_search():
|
||||||
"""Test HaikuRAG search functionality."""
|
"""Test HaikuRAG search functionality."""
|
||||||
client = HaikuRAG(":memory:")
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
|
||||||
# Add multiple documents to search from
|
# Add multiple documents to search from
|
||||||
doc1_text = "Python is a high-level programming language known for its simplicity and readability."
|
doc1_text = "Python is a high-level programming language known for its simplicity and readability."
|
||||||
doc2_text = "Machine learning algorithms help computers learn patterns from data without explicit programming."
|
doc2_text = "Machine learning algorithms help computers learn patterns from data without explicit programming."
|
||||||
|
|
@ -471,8 +425,6 @@ async def test_client_search():
|
||||||
limited_results = await client.search("programming", limit=1)
|
limited_results = await client.search("programming", limit=1)
|
||||||
assert len(limited_results) <= 1
|
assert len(limited_results) <= 1
|
||||||
|
|
||||||
client.close()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_client_async_context_manager():
|
async def test_client_async_context_manager():
|
||||||
|
|
|
||||||
|
|
@ -13,11 +13,10 @@ from haiku.rag.store.models.document import Document
|
||||||
async def test_file_watcher_upsert_document():
|
async def test_file_watcher_upsert_document():
|
||||||
"""Test FileWatcher._upsert_document method."""
|
"""Test FileWatcher._upsert_document method."""
|
||||||
|
|
||||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
f.write("Test content for file watcher")
|
temp_path = Path(temp_dir) / "test.txt"
|
||||||
temp_path = Path(f.name)
|
temp_path.write_text("Test content for file watcher")
|
||||||
|
|
||||||
try:
|
|
||||||
mock_client = AsyncMock(spec=HaikuRAG)
|
mock_client = AsyncMock(spec=HaikuRAG)
|
||||||
mock_doc = Document(id=1, content="Test content", uri=temp_path.as_uri())
|
mock_doc = Document(id=1, content="Test content", uri=temp_path.as_uri())
|
||||||
mock_client.create_document_from_source.return_value = mock_doc
|
mock_client.create_document_from_source.return_value = mock_doc
|
||||||
|
|
@ -32,19 +31,15 @@ async def test_file_watcher_upsert_document():
|
||||||
mock_client.get_document_by_uri.assert_called_once_with(temp_path.as_uri())
|
mock_client.get_document_by_uri.assert_called_once_with(temp_path.as_uri())
|
||||||
mock_client.create_document_from_source.assert_called_once_with(str(temp_path))
|
mock_client.create_document_from_source.assert_called_once_with(str(temp_path))
|
||||||
|
|
||||||
finally:
|
|
||||||
temp_path.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_file_watcher_upsert_existing_document():
|
async def test_file_watcher_upsert_existing_document():
|
||||||
"""Test FileWatcher._upsert_document with existing document."""
|
"""Test FileWatcher._upsert_document with existing document."""
|
||||||
|
|
||||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
f.write("Test content for file watcher")
|
temp_path = Path(temp_dir) / "test.txt"
|
||||||
temp_path = Path(f.name)
|
temp_path.write_text("Test content for file watcher")
|
||||||
|
|
||||||
try:
|
|
||||||
mock_client = AsyncMock(spec=HaikuRAG)
|
mock_client = AsyncMock(spec=HaikuRAG)
|
||||||
existing_doc = Document(id=1, content="Old content", uri=temp_path.as_uri())
|
existing_doc = Document(id=1, content="Old content", uri=temp_path.as_uri())
|
||||||
updated_doc = Document(id=1, content="Updated content", uri=temp_path.as_uri())
|
updated_doc = Document(id=1, content="Updated content", uri=temp_path.as_uri())
|
||||||
|
|
@ -61,9 +56,6 @@ async def test_file_watcher_upsert_existing_document():
|
||||||
mock_client.get_document_by_uri.assert_called_once_with(temp_path.as_uri())
|
mock_client.get_document_by_uri.assert_called_once_with(temp_path.as_uri())
|
||||||
mock_client.create_document_from_source.assert_called_once_with(str(temp_path))
|
mock_client.create_document_from_source.assert_called_once_with(str(temp_path))
|
||||||
|
|
||||||
finally:
|
|
||||||
temp_path.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_file_watcher_delete_document():
|
async def test_file_watcher_delete_document():
|
||||||
|
|
|
||||||
94
tests/test_qa.py
Normal file
94
tests/test_qa.py
Normal file
|
|
@ -0,0 +1,94 @@
|
||||||
|
import pytest
|
||||||
|
from datasets import Dataset
|
||||||
|
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.qa.ollama import QuestionAnswerOllamaAgent
|
||||||
|
|
||||||
|
try:
|
||||||
|
from haiku.rag.qa.openai import QuestionAnswerOpenAIAgent
|
||||||
|
|
||||||
|
OPENAI_AVAILABLE = True
|
||||||
|
except ImportError:
|
||||||
|
QuestionAnswerOpenAIAgent = None
|
||||||
|
OPENAI_AVAILABLE = False
|
||||||
|
|
||||||
|
try:
|
||||||
|
from haiku.rag.qa.anthropic import QuestionAnswerAnthropicAgent
|
||||||
|
|
||||||
|
ANTHROPIC_AVAILABLE = True
|
||||||
|
except ImportError:
|
||||||
|
QuestionAnswerAnthropicAgent = None
|
||||||
|
ANTHROPIC_AVAILABLE = False
|
||||||
|
|
||||||
|
from .llm_judge import LLMJudge
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_qa_ollama(qa_corpus: Dataset):
|
||||||
|
"""Test QA with actual question from the dataset using LLM judge."""
|
||||||
|
client = HaikuRAG(":memory:")
|
||||||
|
qa = QuestionAnswerOllamaAgent(client)
|
||||||
|
llm_judge = LLMJudge()
|
||||||
|
|
||||||
|
doc = qa_corpus[1]
|
||||||
|
await client.create_document(
|
||||||
|
content=doc["document_extracted"], uri=doc["document_id"]
|
||||||
|
)
|
||||||
|
|
||||||
|
question = doc["question"]
|
||||||
|
expected_answer = doc["answer"]
|
||||||
|
|
||||||
|
answer = await qa.answer(question)
|
||||||
|
is_equivalent = await llm_judge.judge_answers(question, answer, expected_answer)
|
||||||
|
|
||||||
|
assert is_equivalent, (
|
||||||
|
f"Generated answer not equivalent to expected answer.\nQuestion: {question}\nGenerated: {answer}\nExpected: {expected_answer}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
@pytest.mark.skipif(not OPENAI_AVAILABLE, reason="OpenAI not available")
|
||||||
|
async def test_qa_openai(qa_corpus: Dataset):
|
||||||
|
"""Test OpenAI QA basic functionality."""
|
||||||
|
client = HaikuRAG(":memory:")
|
||||||
|
qa = QuestionAnswerOpenAIAgent(client) # type: ignore
|
||||||
|
llm_judge = LLMJudge()
|
||||||
|
|
||||||
|
doc = qa_corpus[1]
|
||||||
|
await client.create_document(
|
||||||
|
content=doc["document_extracted"], uri=doc["document_id"]
|
||||||
|
)
|
||||||
|
|
||||||
|
question = doc["question"]
|
||||||
|
expected_answer = doc["answer"]
|
||||||
|
|
||||||
|
answer = await qa.answer(question)
|
||||||
|
is_equivalent = await llm_judge.judge_answers(question, answer, expected_answer)
|
||||||
|
|
||||||
|
assert is_equivalent, (
|
||||||
|
f"Generated answer not equivalent to expected answer.\nQuestion: {question}\nGenerated: {answer}\nExpected: {expected_answer}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
@pytest.mark.skipif(not ANTHROPIC_AVAILABLE, reason="Anthropic not available")
|
||||||
|
async def test_qa_anthropic(qa_corpus: Dataset):
|
||||||
|
"""Test Anthropic QA basic functionality."""
|
||||||
|
client = HaikuRAG(":memory:")
|
||||||
|
qa = QuestionAnswerAnthropicAgent(client) # type: ignore
|
||||||
|
llm_judge = LLMJudge()
|
||||||
|
|
||||||
|
doc = qa_corpus[1]
|
||||||
|
await client.create_document(
|
||||||
|
content=doc["document_extracted"], uri=doc["document_id"]
|
||||||
|
)
|
||||||
|
|
||||||
|
question = doc["question"]
|
||||||
|
expected_answer = doc["answer"]
|
||||||
|
|
||||||
|
answer = await qa.answer(question)
|
||||||
|
is_equivalent = await llm_judge.judge_answers(question, answer, expected_answer)
|
||||||
|
|
||||||
|
assert is_equivalent, (
|
||||||
|
f"Generated answer not equivalent to expected answer.\nQuestion: {question}\nGenerated: {answer}\nExpected: {expected_answer}"
|
||||||
|
)
|
||||||
49
tests/test_rebuild.py
Normal file
49
tests/test_rebuild.py
Normal file
|
|
@ -0,0 +1,49 @@
|
||||||
|
import pytest
|
||||||
|
from datasets import Dataset
|
||||||
|
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.store.models.document import Document
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_rebuild_database(qa_corpus: Dataset):
|
||||||
|
"""Test rebuild functionality with existing documents."""
|
||||||
|
async with HaikuRAG(":memory:") as client:
|
||||||
|
created_docs: list[Document] = []
|
||||||
|
for content in qa_corpus["document_extracted"][:3]:
|
||||||
|
doc = await client.create_document(
|
||||||
|
content=content,
|
||||||
|
)
|
||||||
|
created_docs.append(doc)
|
||||||
|
|
||||||
|
documents_before = await client.list_documents()
|
||||||
|
assert len(documents_before) == 3
|
||||||
|
|
||||||
|
chunks_before = []
|
||||||
|
for doc in created_docs:
|
||||||
|
assert doc.id is not None
|
||||||
|
doc_chunks = await client.chunk_repository.get_by_document_id(doc.id)
|
||||||
|
chunks_before.extend(doc_chunks)
|
||||||
|
|
||||||
|
assert len(chunks_before) > 0
|
||||||
|
|
||||||
|
# Perform rebuild
|
||||||
|
processed_doc_ids = []
|
||||||
|
async for doc_id in client.rebuild_database():
|
||||||
|
processed_doc_ids.append(doc_id)
|
||||||
|
|
||||||
|
# Verify all documents were processed
|
||||||
|
expected_doc_ids = [doc.id for doc in created_docs]
|
||||||
|
assert set(processed_doc_ids) == set(expected_doc_ids)
|
||||||
|
|
||||||
|
documents_after = await client.list_documents()
|
||||||
|
assert len(documents_after) == 3
|
||||||
|
|
||||||
|
# Verify chunks were recreated
|
||||||
|
chunks_after = []
|
||||||
|
for doc in documents_after:
|
||||||
|
if doc.id is not None:
|
||||||
|
doc_chunks = await client.chunk_repository.get_by_document_id(doc.id)
|
||||||
|
chunks_after.extend(doc_chunks)
|
||||||
|
|
||||||
|
assert len(chunks_after) > 0
|
||||||
|
|
@ -56,3 +56,33 @@ async def test_search_qa_corpus(qa_corpus: Dataset):
|
||||||
assert target_document.id in target_document_ids
|
assert target_document.id in target_document_ids
|
||||||
|
|
||||||
store.close()
|
store.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_chunks_include_document_info():
|
||||||
|
"""Test that search results include document URI and metadata."""
|
||||||
|
store = Store(":memory:")
|
||||||
|
doc_repo = DocumentRepository(store)
|
||||||
|
chunk_repo = ChunkRepository(store)
|
||||||
|
|
||||||
|
# Create a document with URI and metadata
|
||||||
|
document = Document(
|
||||||
|
content="This is a test document with some content for searching.",
|
||||||
|
uri="https://example.com/test.html",
|
||||||
|
metadata={"title": "Test Document", "author": "Test Author"},
|
||||||
|
)
|
||||||
|
|
||||||
|
created_document = await doc_repo.create(document)
|
||||||
|
|
||||||
|
# Search for chunks
|
||||||
|
results = await chunk_repo.search_chunks_hybrid("test document", limit=1)
|
||||||
|
|
||||||
|
assert len(results) > 0
|
||||||
|
chunk, score = results[0]
|
||||||
|
|
||||||
|
# Verify the chunk includes document information
|
||||||
|
assert chunk.document_uri == "https://example.com/test.html"
|
||||||
|
assert chunk.document_meta == {"title": "Test Document", "author": "Test Author"}
|
||||||
|
assert chunk.document_id == created_document.id
|
||||||
|
|
||||||
|
store.close()
|
||||||
|
|
|
||||||
80
tests/test_settings.py
Normal file
80
tests/test_settings.py
Normal file
|
|
@ -0,0 +1,80 @@
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from haiku.rag.client import HaikuRAG
|
||||||
|
from haiku.rag.config import Config
|
||||||
|
from haiku.rag.store.engine import Store
|
||||||
|
from haiku.rag.store.repositories.settings import (
|
||||||
|
ConfigMismatchError,
|
||||||
|
SettingsRepository,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_settings_table_populated_on_store_init():
|
||||||
|
"""Test that settings table is populated with current config when store is initialized."""
|
||||||
|
|
||||||
|
store = Store(":memory:")
|
||||||
|
settings_repo = SettingsRepository(store)
|
||||||
|
|
||||||
|
db_settings = settings_repo.get()
|
||||||
|
config_dict = Config.model_dump(mode="json")
|
||||||
|
|
||||||
|
assert db_settings == config_dict
|
||||||
|
|
||||||
|
store.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_settings_save_and_retrieve():
|
||||||
|
"""Test saving and retrieving settings after config change."""
|
||||||
|
store = Store(":memory:")
|
||||||
|
settings_repo = SettingsRepository(store)
|
||||||
|
|
||||||
|
original_chunk_size = Config.CHUNK_SIZE
|
||||||
|
Config.CHUNK_SIZE = 2 * original_chunk_size
|
||||||
|
|
||||||
|
settings_repo.save()
|
||||||
|
retrieved_settings = settings_repo.get()
|
||||||
|
assert retrieved_settings["CHUNK_SIZE"] == 2 * original_chunk_size
|
||||||
|
|
||||||
|
Config.CHUNK_SIZE = original_chunk_size
|
||||||
|
store.close()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_config_validation_on_db_load():
|
||||||
|
"""Test that config validation fails when loading db with mismatched settings."""
|
||||||
|
# Create a temporary database file
|
||||||
|
with tempfile.NamedTemporaryFile(suffix=".sqlite") as tmp:
|
||||||
|
db_path = Path(tmp.name)
|
||||||
|
|
||||||
|
# Create store and save settings
|
||||||
|
store1 = Store(db_path)
|
||||||
|
store1.close()
|
||||||
|
|
||||||
|
# Change config
|
||||||
|
original_chunk_size = Config.CHUNK_SIZE
|
||||||
|
Config.CHUNK_SIZE = 999
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Loading the database should raise ConfigMismatchError
|
||||||
|
with pytest.raises(ConfigMismatchError) as exc_info:
|
||||||
|
Store(db_path)
|
||||||
|
|
||||||
|
assert "CHUNK_SIZE" in str(exc_info.value)
|
||||||
|
assert "Consider rebuilding" in str(exc_info.value)
|
||||||
|
|
||||||
|
# Rebuild
|
||||||
|
async with HaikuRAG(db_path=db_path, skip_validation=True) as client:
|
||||||
|
async for _ in client.rebuild_database():
|
||||||
|
pass # Process all documents
|
||||||
|
|
||||||
|
# Verify we can now load the database without exception (settings were updated)
|
||||||
|
store2 = Store(db_path)
|
||||||
|
settings_repo2 = SettingsRepository(store2)
|
||||||
|
db_settings = settings_repo2.get()
|
||||||
|
assert db_settings["CHUNK_SIZE"] == 999
|
||||||
|
store2.close()
|
||||||
|
|
||||||
|
finally:
|
||||||
|
Config.CHUNK_SIZE = original_chunk_size
|
||||||
15
tests/test_utils.py
Normal file
15
tests/test_utils.py
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
from haiku.rag.utils import int_to_semantic_version, semantic_version_to_int
|
||||||
|
|
||||||
|
|
||||||
|
def test_sqlite_user_version():
|
||||||
|
version = "0.1.5"
|
||||||
|
assert semantic_version_to_int(version) == 261
|
||||||
|
assert int_to_semantic_version(261) == version
|
||||||
|
|
||||||
|
version = "0.0.0"
|
||||||
|
assert semantic_version_to_int(version) == 0
|
||||||
|
assert int_to_semantic_version(0) == version
|
||||||
|
|
||||||
|
version = "255.255.255"
|
||||||
|
assert semantic_version_to_int(version) == 16777215
|
||||||
|
assert int_to_semantic_version(16777215) == version
|
||||||
305
uv.lock
305
uv.lock
|
|
@ -133,6 +133,24 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" },
|
{ url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "anthropic"
|
||||||
|
version = "0.56.0"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "anyio" },
|
||||||
|
{ name = "distro" },
|
||||||
|
{ name = "httpx" },
|
||||||
|
{ name = "jiter" },
|
||||||
|
{ name = "pydantic" },
|
||||||
|
{ name = "sniffio" },
|
||||||
|
{ name = "typing-extensions" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/4d/40/0c4eb5728466849803782c8a86eb315af1a6eb0efea6a751de120ab845c9/anthropic-0.56.0.tar.gz", hash = "sha256:56fa9eb61afa004a1664bc85eed071e77b96c579b77395e9cc893097e599f72e", size = 421538, upload-time = "2025-07-01T19:39:10.805Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/e5/90/7f4d4084f9c35c3ea3e784646ec12f9b2c8cf8743b2bb5489252659b5bda/anthropic-0.56.0-py3-none-any.whl", hash = "sha256:91f1f74abdcf0958d3296b657304588cc244b1107b89f973ff6f511afdacfc56", size = 289603, upload-time = "2025-07-01T19:39:08.794Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "anyio"
|
name = "anyio"
|
||||||
version = "4.9.0"
|
version = "4.9.0"
|
||||||
|
|
@ -218,6 +236,29 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/84/29/587c189bbab1ccc8c86a03a5d0e13873df916380ef1be461ebe6acebf48d/authlib-1.6.0-py2.py3-none-any.whl", hash = "sha256:91685589498f79e8655e8a8947431ad6288831d643f11c55c2143ffcc738048d", size = 239981, upload-time = "2025-05-23T00:21:43.075Z" },
|
{ url = "https://files.pythonhosted.org/packages/84/29/587c189bbab1ccc8c86a03a5d0e13873df916380ef1be461ebe6acebf48d/authlib-1.6.0-py2.py3-none-any.whl", hash = "sha256:91685589498f79e8655e8a8947431ad6288831d643f11c55c2143ffcc738048d", size = 239981, upload-time = "2025-05-23T00:21:43.075Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "babel"
|
||||||
|
version = "2.17.0"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/7d/6b/d52e42361e1aa00709585ecc30b3f9684b3ab62530771402248b1b1d6240/babel-2.17.0.tar.gz", hash = "sha256:0c54cffb19f690cdcc52a3b50bcbf71e07a808d1c80d549f2459b9d2cf0afb9d", size = 9951852, upload-time = "2025-02-01T15:17:41.026Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/b7/b8/3fe70c75fe32afc4bb507f75563d39bc5642255d1d94f1f23604725780bf/babel-2.17.0-py3-none-any.whl", hash = "sha256:4d0b53093fdfb4b21c92b5213dba5a1b23885afa8383709427046b21c366e5f2", size = 10182537, upload-time = "2025-02-01T15:17:37.39Z" },
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "backrefs"
|
||||||
|
version = "5.9"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/eb/a7/312f673df6a79003279e1f55619abbe7daebbb87c17c976ddc0345c04c7b/backrefs-5.9.tar.gz", hash = "sha256:808548cb708d66b82ee231f962cb36faaf4f2baab032f2fbb783e9c2fdddaa59", size = 5765857, upload-time = "2025-06-22T19:34:13.97Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/19/4d/798dc1f30468134906575156c089c492cf79b5a5fd373f07fe26c4d046bf/backrefs-5.9-py310-none-any.whl", hash = "sha256:db8e8ba0e9de81fcd635f440deab5ae5f2591b54ac1ebe0550a2ca063488cd9f", size = 380267, upload-time = "2025-06-22T19:34:05.252Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/55/07/f0b3375bf0d06014e9787797e6b7cc02b38ac9ff9726ccfe834d94e9991e/backrefs-5.9-py311-none-any.whl", hash = "sha256:6907635edebbe9b2dc3de3a2befff44d74f30a4562adbb8b36f21252ea19c5cf", size = 392072, upload-time = "2025-06-22T19:34:06.743Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/9d/12/4f345407259dd60a0997107758ba3f221cf89a9b5a0f8ed5b961aef97253/backrefs-5.9-py312-none-any.whl", hash = "sha256:7fdf9771f63e6028d7fee7e0c497c81abda597ea45d6b8f89e8ad76994f5befa", size = 397947, upload-time = "2025-06-22T19:34:08.172Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/10/bf/fa31834dc27a7f05e5290eae47c82690edc3a7b37d58f7fb35a1bdbf355b/backrefs-5.9-py313-none-any.whl", hash = "sha256:cc37b19fa219e93ff825ed1fed8879e47b4d89aa7a1884860e2db64ccd7c676b", size = 399843, upload-time = "2025-06-22T19:34:09.68Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/fc/24/b29af34b2c9c41645a9f4ff117bae860291780d73880f449e0b5d948c070/backrefs-5.9-py314-none-any.whl", hash = "sha256:df5e169836cc8acb5e440ebae9aad4bf9d15e226d3bad049cf3f6a5c20cc8dc9", size = 411762, upload-time = "2025-06-22T19:34:11.037Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/41/ff/392bff89415399a979be4a65357a41d92729ae8580a66073d8ec8d810f98/backrefs-5.9-py39-none-any.whl", hash = "sha256:f48ee18f6252b8f5777a22a00a09a85de0ca931658f1dd96d4406a34f3748c60", size = 380265, upload-time = "2025-06-22T19:34:12.405Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "beautifulsoup4"
|
name = "beautifulsoup4"
|
||||||
version = "4.13.4"
|
version = "4.13.4"
|
||||||
|
|
@ -752,6 +793,18 @@ http = [
|
||||||
{ name = "aiohttp" },
|
{ name = "aiohttp" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "ghp-import"
|
||||||
|
version = "2.1.0"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "python-dateutil" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/d9/29/d40217cbe2f6b1359e00c6c307bb3fc876ba74068cbab3dde77f03ca0dc4/ghp-import-2.1.0.tar.gz", hash = "sha256:9c535c4c61193c2df8871222567d7fd7e5014d835f97dc7b7439069e2413d343", size = 10943, upload-time = "2022-05-02T15:47:16.11Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619", size = 11034, upload-time = "2022-05-02T15:47:14.552Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "h11"
|
name = "h11"
|
||||||
version = "0.16.0"
|
version = "0.16.0"
|
||||||
|
|
@ -763,7 +816,7 @@ wheels = [
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "haiku-rag"
|
name = "haiku-rag"
|
||||||
version = "0.2.0"
|
version = "0.3.4"
|
||||||
source = { editable = "." }
|
source = { editable = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "fastmcp" },
|
{ name = "fastmcp" },
|
||||||
|
|
@ -780,6 +833,9 @@ dependencies = [
|
||||||
]
|
]
|
||||||
|
|
||||||
[package.optional-dependencies]
|
[package.optional-dependencies]
|
||||||
|
anthropic = [
|
||||||
|
{ name = "anthropic" },
|
||||||
|
]
|
||||||
openai = [
|
openai = [
|
||||||
{ name = "openai" },
|
{ name = "openai" },
|
||||||
]
|
]
|
||||||
|
|
@ -790,6 +846,8 @@ voyageai = [
|
||||||
[package.dev-dependencies]
|
[package.dev-dependencies]
|
||||||
dev = [
|
dev = [
|
||||||
{ name = "datasets" },
|
{ name = "datasets" },
|
||||||
|
{ name = "mkdocs" },
|
||||||
|
{ name = "mkdocs-material" },
|
||||||
{ name = "pre-commit" },
|
{ name = "pre-commit" },
|
||||||
{ name = "pyright" },
|
{ name = "pyright" },
|
||||||
{ name = "pytest" },
|
{ name = "pytest" },
|
||||||
|
|
@ -800,6 +858,7 @@ dev = [
|
||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
requires-dist = [
|
requires-dist = [
|
||||||
|
{ name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.56.0" },
|
||||||
{ name = "fastmcp", specifier = ">=2.8.1" },
|
{ name = "fastmcp", specifier = ">=2.8.1" },
|
||||||
{ name = "httpx", specifier = ">=0.28.1" },
|
{ name = "httpx", specifier = ">=0.28.1" },
|
||||||
{ name = "markitdown", extras = ["audio-transcription", "docx", "pdf", "pptx", "xlsx"], specifier = ">=0.1.2" },
|
{ name = "markitdown", extras = ["audio-transcription", "docx", "pdf", "pptx", "xlsx"], specifier = ">=0.1.2" },
|
||||||
|
|
@ -814,13 +873,15 @@ requires-dist = [
|
||||||
{ name = "voyageai", marker = "extra == 'voyageai'", specifier = ">=0.3.2" },
|
{ name = "voyageai", marker = "extra == 'voyageai'", specifier = ">=0.3.2" },
|
||||||
{ name = "watchfiles", specifier = ">=1.1.0" },
|
{ name = "watchfiles", specifier = ">=1.1.0" },
|
||||||
]
|
]
|
||||||
provides-extras = ["voyageai", "openai"]
|
provides-extras = ["voyageai", "openai", "anthropic"]
|
||||||
|
|
||||||
[package.metadata.requires-dev]
|
[package.metadata.requires-dev]
|
||||||
dev = [
|
dev = [
|
||||||
{ name = "datasets", specifier = ">=3.6.0" },
|
{ name = "datasets", specifier = ">=3.6.0" },
|
||||||
|
{ name = "mkdocs", specifier = ">=1.6.1" },
|
||||||
|
{ name = "mkdocs-material", specifier = ">=9.6.14" },
|
||||||
{ name = "pre-commit", specifier = ">=4.2.0" },
|
{ name = "pre-commit", specifier = ">=4.2.0" },
|
||||||
{ name = "pyright", specifier = ">=1.1.402" },
|
{ name = "pyright", specifier = ">=1.1.403" },
|
||||||
{ name = "pytest", specifier = ">=8.4.0" },
|
{ name = "pytest", specifier = ">=8.4.0" },
|
||||||
{ name = "pytest-asyncio", specifier = ">=1.0.0" },
|
{ name = "pytest-asyncio", specifier = ">=1.0.0" },
|
||||||
{ name = "pytest-cov", specifier = ">=6.2.1" },
|
{ name = "pytest-cov", specifier = ">=6.2.1" },
|
||||||
|
|
@ -937,6 +998,18 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/2c/e1/e6716421ea10d38022b952c159d5161ca1193197fb744506875fbb87ea7b/iniconfig-2.1.0-py3-none-any.whl", hash = "sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760", size = 6050, upload-time = "2025-03-19T20:10:01.071Z" },
|
{ url = "https://files.pythonhosted.org/packages/2c/e1/e6716421ea10d38022b952c159d5161ca1193197fb744506875fbb87ea7b/iniconfig-2.1.0-py3-none-any.whl", hash = "sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760", size = 6050, upload-time = "2025-03-19T20:10:01.071Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "jinja2"
|
||||||
|
version = "3.1.6"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "markupsafe" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "jiter"
|
name = "jiter"
|
||||||
version = "0.10.0"
|
version = "0.10.0"
|
||||||
|
|
@ -1122,6 +1195,15 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/be/0c/3153f159b78e368ac473a00e955d69d976e4b69740ed07c76c9f72a161b8/mammoth-1.9.1-py2.py3-none-any.whl", hash = "sha256:f0569bd640cee6c77a07e7c75c5dc10d745dc4dc95d530cfcbb0a5d9536d636c", size = 52991, upload-time = "2025-05-28T19:17:54.62Z" },
|
{ url = "https://files.pythonhosted.org/packages/be/0c/3153f159b78e368ac473a00e955d69d976e4b69740ed07c76c9f72a161b8/mammoth-1.9.1-py2.py3-none-any.whl", hash = "sha256:f0569bd640cee6c77a07e7c75c5dc10d745dc4dc95d530cfcbb0a5d9536d636c", size = 52991, upload-time = "2025-05-28T19:17:54.62Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "markdown"
|
||||||
|
version = "3.8.2"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/d7/c2/4ab49206c17f75cb08d6311171f2d65798988db4360c4d1485bd0eedd67c/markdown-3.8.2.tar.gz", hash = "sha256:247b9a70dd12e27f67431ce62523e675b866d254f900c4fe75ce3dda62237c45", size = 362071, upload-time = "2025-06-19T17:12:44.483Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/96/2b/34cc11786bc00d0f04d0f5fdc3a2b1ae0b6239eef72d3d345805f9ad92a1/markdown-3.8.2-py3-none-any.whl", hash = "sha256:5c83764dbd4e00bdd94d85a19b8d55ccca20fe35b2e678a1422b380324dd5f24", size = 106827, upload-time = "2025-06-19T17:12:42.994Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "markdown-it-py"
|
name = "markdown-it-py"
|
||||||
version = "3.0.0"
|
version = "3.0.0"
|
||||||
|
|
@ -1184,6 +1266,64 @@ xlsx = [
|
||||||
{ name = "pandas" },
|
{ name = "pandas" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "markupsafe"
|
||||||
|
version = "3.0.2"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/b2/97/5d42485e71dfc078108a86d6de8fa46db44a1a9295e89c5d6d4a06e23a62/markupsafe-3.0.2.tar.gz", hash = "sha256:ee55d3edf80167e48ea11a923c7386f4669df67d7994554387f84e7d8b0a2bf0", size = 20537, upload-time = "2024-10-18T15:21:54.129Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/04/90/d08277ce111dd22f77149fd1a5d4653eeb3b3eaacbdfcbae5afb2600eebd/MarkupSafe-3.0.2-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:7e94c425039cde14257288fd61dcfb01963e658efbc0ff54f5306b06054700f8", size = 14357, upload-time = "2024-10-18T15:20:51.44Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/04/e1/6e2194baeae0bca1fae6629dc0cbbb968d4d941469cbab11a3872edff374/MarkupSafe-3.0.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9e2d922824181480953426608b81967de705c3cef4d1af983af849d7bd619158", size = 12393, upload-time = "2024-10-18T15:20:52.426Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/1d/69/35fa85a8ece0a437493dc61ce0bb6d459dcba482c34197e3efc829aa357f/MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:38a9ef736c01fccdd6600705b09dc574584b89bea478200c5fbf112a6b0d5579", size = 21732, upload-time = "2024-10-18T15:20:53.578Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/22/35/137da042dfb4720b638d2937c38a9c2df83fe32d20e8c8f3185dbfef05f7/MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bbcb445fa71794da8f178f0f6d66789a28d7319071af7a496d4d507ed566270d", size = 20866, upload-time = "2024-10-18T15:20:55.06Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/29/28/6d029a903727a1b62edb51863232152fd335d602def598dade38996887f0/MarkupSafe-3.0.2-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:57cb5a3cf367aeb1d316576250f65edec5bb3be939e9247ae594b4bcbc317dfb", size = 20964, upload-time = "2024-10-18T15:20:55.906Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/cc/cd/07438f95f83e8bc028279909d9c9bd39e24149b0d60053a97b2bc4f8aa51/MarkupSafe-3.0.2-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:3809ede931876f5b2ec92eef964286840ed3540dadf803dd570c3b7e13141a3b", size = 21977, upload-time = "2024-10-18T15:20:57.189Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/29/01/84b57395b4cc062f9c4c55ce0df7d3108ca32397299d9df00fedd9117d3d/MarkupSafe-3.0.2-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:e07c3764494e3776c602c1e78e298937c3315ccc9043ead7e685b7f2b8d47b3c", size = 21366, upload-time = "2024-10-18T15:20:58.235Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/bd/6e/61ebf08d8940553afff20d1fb1ba7294b6f8d279df9fd0c0db911b4bbcfd/MarkupSafe-3.0.2-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:b424c77b206d63d500bcb69fa55ed8d0e6a3774056bdc4839fc9298a7edca171", size = 21091, upload-time = "2024-10-18T15:20:59.235Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/11/23/ffbf53694e8c94ebd1e7e491de185124277964344733c45481f32ede2499/MarkupSafe-3.0.2-cp310-cp310-win32.whl", hash = "sha256:fcabf5ff6eea076f859677f5f0b6b5c1a51e70a376b0579e0eadef8db48c6b50", size = 15065, upload-time = "2024-10-18T15:21:00.307Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/44/06/e7175d06dd6e9172d4a69a72592cb3f7a996a9c396eee29082826449bbc3/MarkupSafe-3.0.2-cp310-cp310-win_amd64.whl", hash = "sha256:6af100e168aa82a50e186c82875a5893c5597a0c1ccdb0d8b40240b1f28b969a", size = 15514, upload-time = "2024-10-18T15:21:01.122Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/6b/28/bbf83e3f76936960b850435576dd5e67034e200469571be53f69174a2dfd/MarkupSafe-3.0.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:9025b4018f3a1314059769c7bf15441064b2207cb3f065e6ea1e7359cb46db9d", size = 14353, upload-time = "2024-10-18T15:21:02.187Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/6c/30/316d194b093cde57d448a4c3209f22e3046c5bb2fb0820b118292b334be7/MarkupSafe-3.0.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:93335ca3812df2f366e80509ae119189886b0f3c2b81325d39efdb84a1e2ae93", size = 12392, upload-time = "2024-10-18T15:21:02.941Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f2/96/9cdafba8445d3a53cae530aaf83c38ec64c4d5427d975c974084af5bc5d2/MarkupSafe-3.0.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2cb8438c3cbb25e220c2ab33bb226559e7afb3baec11c4f218ffa7308603c832", size = 23984, upload-time = "2024-10-18T15:21:03.953Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f1/a4/aefb044a2cd8d7334c8a47d3fb2c9f328ac48cb349468cc31c20b539305f/MarkupSafe-3.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a123e330ef0853c6e822384873bef7507557d8e4a082961e1defa947aa59ba84", size = 23120, upload-time = "2024-10-18T15:21:06.495Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/8d/21/5e4851379f88f3fad1de30361db501300d4f07bcad047d3cb0449fc51f8c/MarkupSafe-3.0.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1e084f686b92e5b83186b07e8a17fc09e38fff551f3602b249881fec658d3eca", size = 23032, upload-time = "2024-10-18T15:21:07.295Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/00/7b/e92c64e079b2d0d7ddf69899c98842f3f9a60a1ae72657c89ce2655c999d/MarkupSafe-3.0.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d8213e09c917a951de9d09ecee036d5c7d36cb6cb7dbaece4c71a60d79fb9798", size = 24057, upload-time = "2024-10-18T15:21:08.073Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f9/ac/46f960ca323037caa0a10662ef97d0a4728e890334fc156b9f9e52bcc4ca/MarkupSafe-3.0.2-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:5b02fb34468b6aaa40dfc198d813a641e3a63b98c2b05a16b9f80b7ec314185e", size = 23359, upload-time = "2024-10-18T15:21:09.318Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/69/84/83439e16197337b8b14b6a5b9c2105fff81d42c2a7c5b58ac7b62ee2c3b1/MarkupSafe-3.0.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:0bff5e0ae4ef2e1ae4fdf2dfd5b76c75e5c2fa4132d05fc1b0dabcd20c7e28c4", size = 23306, upload-time = "2024-10-18T15:21:10.185Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/9a/34/a15aa69f01e2181ed8d2b685c0d2f6655d5cca2c4db0ddea775e631918cd/MarkupSafe-3.0.2-cp311-cp311-win32.whl", hash = "sha256:6c89876f41da747c8d3677a2b540fb32ef5715f97b66eeb0c6b66f5e3ef6f59d", size = 15094, upload-time = "2024-10-18T15:21:11.005Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/da/b8/3a3bd761922d416f3dc5d00bfbed11f66b1ab89a0c2b6e887240a30b0f6b/MarkupSafe-3.0.2-cp311-cp311-win_amd64.whl", hash = "sha256:70a87b411535ccad5ef2f1df5136506a10775d267e197e4cf531ced10537bd6b", size = 15521, upload-time = "2024-10-18T15:21:12.911Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/22/09/d1f21434c97fc42f09d290cbb6350d44eb12f09cc62c9476effdb33a18aa/MarkupSafe-3.0.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:9778bd8ab0a994ebf6f84c2b949e65736d5575320a17ae8984a77fab08db94cf", size = 14274, upload-time = "2024-10-18T15:21:13.777Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/6b/b0/18f76bba336fa5aecf79d45dcd6c806c280ec44538b3c13671d49099fdd0/MarkupSafe-3.0.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:846ade7b71e3536c4e56b386c2a47adf5741d2d8b94ec9dc3e92e5e1ee1e2225", size = 12348, upload-time = "2024-10-18T15:21:14.822Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/e0/25/dd5c0f6ac1311e9b40f4af06c78efde0f3b5cbf02502f8ef9501294c425b/MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1c99d261bd2d5f6b59325c92c73df481e05e57f19837bdca8413b9eac4bd8028", size = 24149, upload-time = "2024-10-18T15:21:15.642Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f3/f0/89e7aadfb3749d0f52234a0c8c7867877876e0a20b60e2188e9850794c17/MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e17c96c14e19278594aa4841ec148115f9c7615a47382ecb6b82bd8fea3ab0c8", size = 23118, upload-time = "2024-10-18T15:21:17.133Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/d5/da/f2eeb64c723f5e3777bc081da884b414671982008c47dcc1873d81f625b6/MarkupSafe-3.0.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:88416bd1e65dcea10bc7569faacb2c20ce071dd1f87539ca2ab364bf6231393c", size = 22993, upload-time = "2024-10-18T15:21:18.064Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/da/0e/1f32af846df486dce7c227fe0f2398dc7e2e51d4a370508281f3c1c5cddc/MarkupSafe-3.0.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2181e67807fc2fa785d0592dc2d6206c019b9502410671cc905d132a92866557", size = 24178, upload-time = "2024-10-18T15:21:18.859Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/c4/f6/bb3ca0532de8086cbff5f06d137064c8410d10779c4c127e0e47d17c0b71/MarkupSafe-3.0.2-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:52305740fe773d09cffb16f8ed0427942901f00adedac82ec8b67752f58a1b22", size = 23319, upload-time = "2024-10-18T15:21:19.671Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/a2/82/8be4c96ffee03c5b4a034e60a31294daf481e12c7c43ab8e34a1453ee48b/MarkupSafe-3.0.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ad10d3ded218f1039f11a75f8091880239651b52e9bb592ca27de44eed242a48", size = 23352, upload-time = "2024-10-18T15:21:20.971Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/51/ae/97827349d3fcffee7e184bdf7f41cd6b88d9919c80f0263ba7acd1bbcb18/MarkupSafe-3.0.2-cp312-cp312-win32.whl", hash = "sha256:0f4ca02bea9a23221c0182836703cbf8930c5e9454bacce27e767509fa286a30", size = 15097, upload-time = "2024-10-18T15:21:22.646Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/c1/80/a61f99dc3a936413c3ee4e1eecac96c0da5ed07ad56fd975f1a9da5bc630/MarkupSafe-3.0.2-cp312-cp312-win_amd64.whl", hash = "sha256:8e06879fc22a25ca47312fbe7c8264eb0b662f6db27cb2d3bbbc74b1df4b9b87", size = 15601, upload-time = "2024-10-18T15:21:23.499Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/83/0e/67eb10a7ecc77a0c2bbe2b0235765b98d164d81600746914bebada795e97/MarkupSafe-3.0.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ba9527cdd4c926ed0760bc301f6728ef34d841f405abf9d4f959c478421e4efd", size = 14274, upload-time = "2024-10-18T15:21:24.577Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/2b/6d/9409f3684d3335375d04e5f05744dfe7e9f120062c9857df4ab490a1031a/MarkupSafe-3.0.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f8b3d067f2e40fe93e1ccdd6b2e1d16c43140e76f02fb1319a05cf2b79d99430", size = 12352, upload-time = "2024-10-18T15:21:25.382Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/d2/f5/6eadfcd3885ea85fe2a7c128315cc1bb7241e1987443d78c8fe712d03091/MarkupSafe-3.0.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:569511d3b58c8791ab4c2e1285575265991e6d8f8700c7be0e88f86cb0672094", size = 24122, upload-time = "2024-10-18T15:21:26.199Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/0c/91/96cf928db8236f1bfab6ce15ad070dfdd02ed88261c2afafd4b43575e9e9/MarkupSafe-3.0.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:15ab75ef81add55874e7ab7055e9c397312385bd9ced94920f2802310c930396", size = 23085, upload-time = "2024-10-18T15:21:27.029Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/c2/cf/c9d56af24d56ea04daae7ac0940232d31d5a8354f2b457c6d856b2057d69/MarkupSafe-3.0.2-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f3818cb119498c0678015754eba762e0d61e5b52d34c8b13d770f0719f7b1d79", size = 22978, upload-time = "2024-10-18T15:21:27.846Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/2a/9f/8619835cd6a711d6272d62abb78c033bda638fdc54c4e7f4272cf1c0962b/MarkupSafe-3.0.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:cdb82a876c47801bb54a690c5ae105a46b392ac6099881cdfb9f6e95e4014c6a", size = 24208, upload-time = "2024-10-18T15:21:28.744Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f9/bf/176950a1792b2cd2102b8ffeb5133e1ed984547b75db47c25a67d3359f77/MarkupSafe-3.0.2-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:cabc348d87e913db6ab4aa100f01b08f481097838bdddf7c7a84b7575b7309ca", size = 23357, upload-time = "2024-10-18T15:21:29.545Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/ce/4f/9a02c1d335caabe5c4efb90e1b6e8ee944aa245c1aaaab8e8a618987d816/MarkupSafe-3.0.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:444dcda765c8a838eaae23112db52f1efaf750daddb2d9ca300bcae1039adc5c", size = 23344, upload-time = "2024-10-18T15:21:30.366Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/ee/55/c271b57db36f748f0e04a759ace9f8f759ccf22b4960c270c78a394f58be/MarkupSafe-3.0.2-cp313-cp313-win32.whl", hash = "sha256:bcf3e58998965654fdaff38e58584d8937aa3096ab5354d493c77d1fdd66d7a1", size = 15101, upload-time = "2024-10-18T15:21:31.207Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/29/88/07df22d2dd4df40aba9f3e402e6dc1b8ee86297dddbad4872bd5e7b0094f/MarkupSafe-3.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:e6a2a455bd412959b57a172ce6328d2dd1f01cb2135efda2e4576e8a23fa3b0f", size = 15603, upload-time = "2024-10-18T15:21:32.032Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/62/6a/8b89d24db2d32d433dffcd6a8779159da109842434f1dd2f6e71f32f738c/MarkupSafe-3.0.2-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:b5a6b3ada725cea8a5e634536b1b01c30bcdcd7f9c6fff4151548d5bf6b3a36c", size = 14510, upload-time = "2024-10-18T15:21:33.625Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/7a/06/a10f955f70a2e5a9bf78d11a161029d278eeacbd35ef806c3fd17b13060d/MarkupSafe-3.0.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:a904af0a6162c73e3edcb969eeeb53a63ceeb5d8cf642fade7d39e7963a22ddb", size = 12486, upload-time = "2024-10-18T15:21:34.611Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/34/cf/65d4a571869a1a9078198ca28f39fba5fbb910f952f9dbc5220afff9f5e6/MarkupSafe-3.0.2-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4aa4e5faecf353ed117801a068ebab7b7e09ffb6e1d5e412dc852e0da018126c", size = 25480, upload-time = "2024-10-18T15:21:35.398Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/0c/e3/90e9651924c430b885468b56b3d597cabf6d72be4b24a0acd1fa0e12af67/MarkupSafe-3.0.2-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c0ef13eaeee5b615fb07c9a7dadb38eac06a0608b41570d8ade51c56539e509d", size = 23914, upload-time = "2024-10-18T15:21:36.231Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/66/8c/6c7cf61f95d63bb866db39085150df1f2a5bd3335298f14a66b48e92659c/MarkupSafe-3.0.2-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d16a81a06776313e817c951135cf7340a3e91e8c1ff2fac444cfd75fffa04afe", size = 23796, upload-time = "2024-10-18T15:21:37.073Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/bb/35/cbe9238ec3f47ac9a7c8b3df7a808e7cb50fe149dc7039f5f454b3fba218/MarkupSafe-3.0.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:6381026f158fdb7c72a168278597a5e3a5222e83ea18f543112b2662a9b699c5", size = 25473, upload-time = "2024-10-18T15:21:37.932Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/e6/32/7621a4382488aa283cc05e8984a9c219abad3bca087be9ec77e89939ded9/MarkupSafe-3.0.2-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:3d79d162e7be8f996986c064d1c7c817f6df3a77fe3d6859f6f9e7be4b8c213a", size = 24114, upload-time = "2024-10-18T15:21:39.799Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/0d/80/0985960e4b89922cb5a0bac0ed39c5b96cbc1a536a99f30e8c220a996ed9/MarkupSafe-3.0.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:131a3c7689c85f5ad20f9f6fb1b866f402c445b220c19fe4308c0b147ccd2ad9", size = 24098, upload-time = "2024-10-18T15:21:40.813Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/82/78/fedb03c7d5380df2427038ec8d973587e90561b2d90cd472ce9254cf348b/MarkupSafe-3.0.2-cp313-cp313t-win32.whl", hash = "sha256:ba8062ed2cf21c07a9e295d5b8a2a5ce678b913b45fdf68c32d95d6c1291e0b6", size = 15208, upload-time = "2024-10-18T15:21:41.814Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/4f/65/6079a46068dfceaeabb5dcad6d674f5f5c61a6fa5673746f42a9f4c233b3/MarkupSafe-3.0.2-cp313-cp313t-win_amd64.whl", hash = "sha256:e444a31f8db13eb18ada366ab3cf45fd4b31e4db1236a4448f68778c1d1a5a2f", size = 15739, upload-time = "2024-10-18T15:21:42.784Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "mcp"
|
name = "mcp"
|
||||||
version = "1.9.4"
|
version = "1.9.4"
|
||||||
|
|
@ -1213,6 +1353,84 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
|
{ url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "mergedeep"
|
||||||
|
version = "1.3.4"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/3a/41/580bb4006e3ed0361b8151a01d324fb03f420815446c7def45d02f74c270/mergedeep-1.3.4.tar.gz", hash = "sha256:0096d52e9dad9939c3d975a774666af186eda617e6ca84df4c94dec30004f2a8", size = 4661, upload-time = "2021-02-05T18:55:30.623Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl", hash = "sha256:70775750742b25c0d8f36c55aed03d24c3384d17c951b3175d898bd778ef0307", size = 6354, upload-time = "2021-02-05T18:55:29.583Z" },
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "mkdocs"
|
||||||
|
version = "1.6.1"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "click" },
|
||||||
|
{ name = "colorama", marker = "sys_platform == 'win32'" },
|
||||||
|
{ name = "ghp-import" },
|
||||||
|
{ name = "jinja2" },
|
||||||
|
{ name = "markdown" },
|
||||||
|
{ name = "markupsafe" },
|
||||||
|
{ name = "mergedeep" },
|
||||||
|
{ name = "mkdocs-get-deps" },
|
||||||
|
{ name = "packaging" },
|
||||||
|
{ name = "pathspec" },
|
||||||
|
{ name = "pyyaml" },
|
||||||
|
{ name = "pyyaml-env-tag" },
|
||||||
|
{ name = "watchdog" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/bc/c6/bbd4f061bd16b378247f12953ffcb04786a618ce5e904b8c5a01a0309061/mkdocs-1.6.1.tar.gz", hash = "sha256:7b432f01d928c084353ab39c57282f29f92136665bdd6abf7c1ec8d822ef86f2", size = 3889159, upload-time = "2024-08-30T12:24:06.899Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/22/5b/dbc6a8cddc9cfa9c4971d59fb12bb8d42e161b7e7f8cc89e49137c5b279c/mkdocs-1.6.1-py3-none-any.whl", hash = "sha256:db91759624d1647f3f34aa0c3f327dd2601beae39a366d6e064c03468d35c20e", size = 3864451, upload-time = "2024-08-30T12:24:05.054Z" },
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "mkdocs-get-deps"
|
||||||
|
version = "0.2.0"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "mergedeep" },
|
||||||
|
{ name = "platformdirs" },
|
||||||
|
{ name = "pyyaml" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/98/f5/ed29cd50067784976f25ed0ed6fcd3c2ce9eb90650aa3b2796ddf7b6870b/mkdocs_get_deps-0.2.0.tar.gz", hash = "sha256:162b3d129c7fad9b19abfdcb9c1458a651628e4b1dea628ac68790fb3061c60c", size = 10239, upload-time = "2023-11-20T17:51:09.981Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/9f/d4/029f984e8d3f3b6b726bd33cafc473b75e9e44c0f7e80a5b29abc466bdea/mkdocs_get_deps-0.2.0-py3-none-any.whl", hash = "sha256:2bf11d0b133e77a0dd036abeeb06dec8775e46efa526dc70667d8863eefc6134", size = 9521, upload-time = "2023-11-20T17:51:08.587Z" },
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "mkdocs-material"
|
||||||
|
version = "9.6.14"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "babel" },
|
||||||
|
{ name = "backrefs" },
|
||||||
|
{ name = "colorama" },
|
||||||
|
{ name = "jinja2" },
|
||||||
|
{ name = "markdown" },
|
||||||
|
{ name = "mkdocs" },
|
||||||
|
{ name = "mkdocs-material-extensions" },
|
||||||
|
{ name = "paginate" },
|
||||||
|
{ name = "pygments" },
|
||||||
|
{ name = "pymdown-extensions" },
|
||||||
|
{ name = "requests" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/b3/fa/0101de32af88f87cf5cc23ad5f2e2030d00995f74e616306513431b8ab4b/mkdocs_material-9.6.14.tar.gz", hash = "sha256:39d795e90dce6b531387c255bd07e866e027828b7346d3eba5ac3de265053754", size = 3951707, upload-time = "2025-05-13T13:27:57.173Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/3a/a1/7fdb959ad592e013c01558822fd3c22931a95a0f08cf0a7c36da13a5b2b5/mkdocs_material-9.6.14-py3-none-any.whl", hash = "sha256:3b9cee6d3688551bf7a8e8f41afda97a3c39a12f0325436d76c86706114b721b", size = 8703767, upload-time = "2025-05-13T13:27:54.089Z" },
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "mkdocs-material-extensions"
|
||||||
|
version = "1.3.1"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/79/9b/9b4c96d6593b2a541e1cb8b34899a6d021d208bb357042823d4d2cabdbe7/mkdocs_material_extensions-1.3.1.tar.gz", hash = "sha256:10c9511cea88f568257f960358a467d12b970e1f7b2c0e5fb2bb48cab1928443", size = 11847, upload-time = "2023-11-22T19:09:45.208Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/5b/54/662a4743aa81d9582ee9339d4ffa3c8fd40a4965e033d77b9da9774d3960/mkdocs_material_extensions-1.3.1-py3-none-any.whl", hash = "sha256:adff8b62700b25cb77b53358dad940f3ef973dd6db797907c49e3c2ef3ab4e31", size = 8728, upload-time = "2023-11-22T19:09:43.465Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "mpmath"
|
name = "mpmath"
|
||||||
version = "1.3.0"
|
version = "1.3.0"
|
||||||
|
|
@ -1573,6 +1791,15 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" },
|
{ url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "paginate"
|
||||||
|
version = "0.5.7"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/ec/46/68dde5b6bc00c1296ec6466ab27dddede6aec9af1b99090e1107091b3b84/paginate-0.5.7.tar.gz", hash = "sha256:22bd083ab41e1a8b4f3690544afb2c60c25e5c9a63a30fa2f483f6c60c8e5945", size = 19252, upload-time = "2024-08-25T14:17:24.139Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl", hash = "sha256:b885e2af73abcf01d9559fd5216b57ef722f8c42affbb63942377668e35c7591", size = 13746, upload-time = "2024-08-25T14:17:22.55Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "pandas"
|
name = "pandas"
|
||||||
version = "2.3.0"
|
version = "2.3.0"
|
||||||
|
|
@ -1622,6 +1849,15 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/39/c2/646d2e93e0af70f4e5359d870a63584dacbc324b54d73e6b3267920ff117/pandas-2.3.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:bb3be958022198531eb7ec2008cfc78c5b1eed51af8600c6c5d9160d89d8d249", size = 13231847, upload-time = "2025-06-05T03:27:51.465Z" },
|
{ url = "https://files.pythonhosted.org/packages/39/c2/646d2e93e0af70f4e5359d870a63584dacbc324b54d73e6b3267920ff117/pandas-2.3.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:bb3be958022198531eb7ec2008cfc78c5b1eed51af8600c6c5d9160d89d8d249", size = 13231847, upload-time = "2025-06-05T03:27:51.465Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "pathspec"
|
||||||
|
version = "0.12.1"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/ca/bc/f35b8446f4531a7cb215605d100cd88b7ac6f44ab3fc94870c120ab3adbf/pathspec-0.12.1.tar.gz", hash = "sha256:a482d51503a1ab33b1c67a6c3813a26953dbdc71c31dacaef9a838c4e29f5712", size = 51043, upload-time = "2023-12-10T22:30:45Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/cc/20/ff623b09d963f88bfde16306a54e12ee5ea43e9b597108672ff3a408aad6/pathspec-0.12.1-py3-none-any.whl", hash = "sha256:a0d503e138a4c123b27490a4f7beda6a01c6f288df0e4a8b79c7eb0dc7b4cc08", size = 31191, upload-time = "2023-12-10T22:30:43.14Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "pdfminer-six"
|
name = "pdfminer-six"
|
||||||
version = "20250506"
|
version = "20250506"
|
||||||
|
|
@ -2045,6 +2281,19 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/8a/0b/9fcc47d19c48b59121088dd6da2488a49d5f72dacf8262e2790a1d2c7d15/pygments-2.19.1-py3-none-any.whl", hash = "sha256:9ea1544ad55cecf4b8242fab6dd35a93bbce657034b0611ee383099054ab6d8c", size = 1225293, upload-time = "2025-01-06T17:26:25.553Z" },
|
{ url = "https://files.pythonhosted.org/packages/8a/0b/9fcc47d19c48b59121088dd6da2488a49d5f72dacf8262e2790a1d2c7d15/pygments-2.19.1-py3-none-any.whl", hash = "sha256:9ea1544ad55cecf4b8242fab6dd35a93bbce657034b0611ee383099054ab6d8c", size = 1225293, upload-time = "2025-01-06T17:26:25.553Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "pymdown-extensions"
|
||||||
|
version = "10.16"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "markdown" },
|
||||||
|
{ name = "pyyaml" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/1a/0a/c06b542ac108bfc73200677309cd9188a3a01b127a63f20cadc18d873d88/pymdown_extensions-10.16.tar.gz", hash = "sha256:71dac4fca63fabeffd3eb9038b756161a33ec6e8d230853d3cecf562155ab3de", size = 853197, upload-time = "2025-06-21T17:56:36.974Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/98/d4/10bb14004d3c792811e05e21b5e5dcae805aacb739bd12a0540967b99592/pymdown_extensions-10.16-py3-none-any.whl", hash = "sha256:f5dd064a4db588cb2d95229fc4ee63a1b16cc8b4d0e6145c0899ed8723da1df2", size = 266143, upload-time = "2025-06-21T17:56:35.356Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "pyreadline3"
|
name = "pyreadline3"
|
||||||
version = "3.5.4"
|
version = "3.5.4"
|
||||||
|
|
@ -2056,15 +2305,15 @@ wheels = [
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "pyright"
|
name = "pyright"
|
||||||
version = "1.1.402"
|
version = "1.1.403"
|
||||||
source = { registry = "https://pypi.org/simple" }
|
source = { registry = "https://pypi.org/simple" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "nodeenv" },
|
{ name = "nodeenv" },
|
||||||
{ name = "typing-extensions" },
|
{ name = "typing-extensions" },
|
||||||
]
|
]
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/aa/04/ce0c132d00e20f2d2fb3b3e7c125264ca8b909e693841210534b1ea1752f/pyright-1.1.402.tar.gz", hash = "sha256:85a33c2d40cd4439c66aa946fd4ce71ab2f3f5b8c22ce36a623f59ac22937683", size = 3888207, upload-time = "2025-06-11T08:48:35.759Z" }
|
sdist = { url = "https://files.pythonhosted.org/packages/fe/f6/35f885264ff08c960b23d1542038d8da86971c5d8c955cfab195a4f672d7/pyright-1.1.403.tar.gz", hash = "sha256:3ab69b9f41c67fb5bbb4d7a36243256f0d549ed3608678d381d5f51863921104", size = 3913526, upload-time = "2025-07-09T07:15:52.882Z" }
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/fe/37/1a1c62d955e82adae588be8e374c7f77b165b6cb4203f7d581269959abbc/pyright-1.1.402-py3-none-any.whl", hash = "sha256:2c721f11869baac1884e846232800fe021c33f1b4acb3929cff321f7ea4e2982", size = 5624004, upload-time = "2025-06-11T08:48:33.998Z" },
|
{ url = "https://files.pythonhosted.org/packages/49/b6/b04e5c2f41a5ccad74a1a4759da41adb20b4bc9d59a5e08d29ba60084d07/pyright-1.1.403-py3-none-any.whl", hash = "sha256:c0eeca5aa76cbef3fcc271259bbd785753c7ad7bcac99a9162b4c4c7daed23b3", size = 5684504, upload-time = "2025-07-09T07:15:50.958Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
|
|
@ -2209,6 +2458,18 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/fa/de/02b54f42487e3d3c6efb3f89428677074ca7bf43aae402517bc7cca949f3/PyYAML-6.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:8388ee1976c416731879ac16da0aff3f63b286ffdd57cdeb95f3f2e085687563", size = 156446, upload-time = "2024-08-06T20:33:04.33Z" },
|
{ url = "https://files.pythonhosted.org/packages/fa/de/02b54f42487e3d3c6efb3f89428677074ca7bf43aae402517bc7cca949f3/PyYAML-6.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:8388ee1976c416731879ac16da0aff3f63b286ffdd57cdeb95f3f2e085687563", size = 156446, upload-time = "2024-08-06T20:33:04.33Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "pyyaml-env-tag"
|
||||||
|
version = "1.1"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "pyyaml" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/eb/2e/79c822141bfd05a853236b504869ebc6b70159afc570e1d5a20641782eaa/pyyaml_env_tag-1.1.tar.gz", hash = "sha256:2eb38b75a2d21ee0475d6d97ec19c63287a7e140231e4214969d0eac923cd7ff", size = 5737, upload-time = "2025-05-13T15:24:01.64Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/04/11/432f32f8097b03e3cd5fe57e88efb685d964e2e5178a48ed61e841f7fdce/pyyaml_env_tag-1.1-py3-none-any.whl", hash = "sha256:17109e1a528561e32f026364712fee1264bc2ea6715120891174ed1b980d2e04", size = 4722, upload-time = "2025-05-13T15:23:59.629Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "regex"
|
name = "regex"
|
||||||
version = "2024.11.6"
|
version = "2024.11.6"
|
||||||
|
|
@ -2675,6 +2936,38 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/86/e1/0b2defa3a83aabe67db05d5f494d617dd4764b2a043d83ddc26be5e6e0db/voyageai-0.3.2-py3-none-any.whl", hash = "sha256:1398d6c6bfb1dd3b484f400713e538f00ce8a335250442b0902c21116d9705a8", size = 25518, upload-time = "2024-12-03T00:33:51.927Z" },
|
{ url = "https://files.pythonhosted.org/packages/86/e1/0b2defa3a83aabe67db05d5f494d617dd4764b2a043d83ddc26be5e6e0db/voyageai-0.3.2-py3-none-any.whl", hash = "sha256:1398d6c6bfb1dd3b484f400713e538f00ce8a335250442b0902c21116d9705a8", size = 25518, upload-time = "2024-12-03T00:33:51.927Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "watchdog"
|
||||||
|
version = "6.0.0"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/db/7d/7f3d619e951c88ed75c6037b246ddcf2d322812ee8ea189be89511721d54/watchdog-6.0.0.tar.gz", hash = "sha256:9ddf7c82fda3ae8e24decda1338ede66e1c99883db93711d8fb941eaa2d8c282", size = 131220, upload-time = "2024-11-01T14:07:13.037Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/0c/56/90994d789c61df619bfc5ce2ecdabd5eeff564e1eb47512bd01b5e019569/watchdog-6.0.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:d1cdb490583ebd691c012b3d6dae011000fe42edb7a82ece80965b42abd61f26", size = 96390, upload-time = "2024-11-01T14:06:24.793Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/55/46/9a67ee697342ddf3c6daa97e3a587a56d6c4052f881ed926a849fcf7371c/watchdog-6.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:bc64ab3bdb6a04d69d4023b29422170b74681784ffb9463ed4870cf2f3e66112", size = 88389, upload-time = "2024-11-01T14:06:27.112Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/44/65/91b0985747c52064d8701e1075eb96f8c40a79df889e59a399453adfb882/watchdog-6.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c897ac1b55c5a1461e16dae288d22bb2e412ba9807df8397a635d88f671d36c3", size = 89020, upload-time = "2024-11-01T14:06:29.876Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/e0/24/d9be5cd6642a6aa68352ded4b4b10fb0d7889cb7f45814fb92cecd35f101/watchdog-6.0.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6eb11feb5a0d452ee41f824e271ca311a09e250441c262ca2fd7ebcf2461a06c", size = 96393, upload-time = "2024-11-01T14:06:31.756Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/63/7a/6013b0d8dbc56adca7fdd4f0beed381c59f6752341b12fa0886fa7afc78b/watchdog-6.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ef810fbf7b781a5a593894e4f439773830bdecb885e6880d957d5b9382a960d2", size = 88392, upload-time = "2024-11-01T14:06:32.99Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/d1/40/b75381494851556de56281e053700e46bff5b37bf4c7267e858640af5a7f/watchdog-6.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:afd0fe1b2270917c5e23c2a65ce50c2a4abb63daafb0d419fde368e272a76b7c", size = 89019, upload-time = "2024-11-01T14:06:34.963Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/39/ea/3930d07dafc9e286ed356a679aa02d777c06e9bfd1164fa7c19c288a5483/watchdog-6.0.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:bdd4e6f14b8b18c334febb9c4425a878a2ac20efd1e0b231978e7b150f92a948", size = 96471, upload-time = "2024-11-01T14:06:37.745Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/12/87/48361531f70b1f87928b045df868a9fd4e253d9ae087fa4cf3f7113be363/watchdog-6.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c7c15dda13c4eb00d6fb6fc508b3c0ed88b9d5d374056b239c4ad1611125c860", size = 88449, upload-time = "2024-11-01T14:06:39.748Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/5b/7e/8f322f5e600812e6f9a31b75d242631068ca8f4ef0582dd3ae6e72daecc8/watchdog-6.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6f10cb2d5902447c7d0da897e2c6768bca89174d0c6e1e30abec5421af97a5b0", size = 89054, upload-time = "2024-11-01T14:06:41.009Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/68/98/b0345cabdce2041a01293ba483333582891a3bd5769b08eceb0d406056ef/watchdog-6.0.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:490ab2ef84f11129844c23fb14ecf30ef3d8a6abafd3754a6f75ca1e6654136c", size = 96480, upload-time = "2024-11-01T14:06:42.952Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/85/83/cdf13902c626b28eedef7ec4f10745c52aad8a8fe7eb04ed7b1f111ca20e/watchdog-6.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:76aae96b00ae814b181bb25b1b98076d5fc84e8a53cd8885a318b42b6d3a5134", size = 88451, upload-time = "2024-11-01T14:06:45.084Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/fe/c4/225c87bae08c8b9ec99030cd48ae9c4eca050a59bf5c2255853e18c87b50/watchdog-6.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a175f755fc2279e0b7312c0035d52e27211a5bc39719dd529625b1930917345b", size = 89057, upload-time = "2024-11-01T14:06:47.324Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/30/ad/d17b5d42e28a8b91f8ed01cb949da092827afb9995d4559fd448d0472763/watchdog-6.0.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:c7ac31a19f4545dd92fc25d200694098f42c9a8e391bc00bdd362c5736dbf881", size = 87902, upload-time = "2024-11-01T14:06:53.119Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/5c/ca/c3649991d140ff6ab67bfc85ab42b165ead119c9e12211e08089d763ece5/watchdog-6.0.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:9513f27a1a582d9808cf21a07dae516f0fab1cf2d7683a742c498b93eedabb11", size = 88380, upload-time = "2024-11-01T14:06:55.19Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/a9/c7/ca4bf3e518cb57a686b2feb4f55a1892fd9a3dd13f470fca14e00f80ea36/watchdog-6.0.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:7607498efa04a3542ae3e05e64da8202e58159aa1fa4acddf7678d34a35d4f13", size = 79079, upload-time = "2024-11-01T14:06:59.472Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/5c/51/d46dc9332f9a647593c947b4b88e2381c8dfc0942d15b8edc0310fa4abb1/watchdog-6.0.0-py3-none-manylinux2014_armv7l.whl", hash = "sha256:9041567ee8953024c83343288ccc458fd0a2d811d6a0fd68c4c22609e3490379", size = 79078, upload-time = "2024-11-01T14:07:01.431Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/d4/57/04edbf5e169cd318d5f07b4766fee38e825d64b6913ca157ca32d1a42267/watchdog-6.0.0-py3-none-manylinux2014_i686.whl", hash = "sha256:82dc3e3143c7e38ec49d61af98d6558288c415eac98486a5c581726e0737c00e", size = 79076, upload-time = "2024-11-01T14:07:02.568Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/ab/cc/da8422b300e13cb187d2203f20b9253e91058aaf7db65b74142013478e66/watchdog-6.0.0-py3-none-manylinux2014_ppc64.whl", hash = "sha256:212ac9b8bf1161dc91bd09c048048a95ca3a4c4f5e5d4a7d1b1a7d5752a7f96f", size = 79077, upload-time = "2024-11-01T14:07:03.893Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/2c/3b/b8964e04ae1a025c44ba8e4291f86e97fac443bca31de8bd98d3263d2fcf/watchdog-6.0.0-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:e3df4cbb9a450c6d49318f6d14f4bbc80d763fa587ba46ec86f99f9e6876bb26", size = 79078, upload-time = "2024-11-01T14:07:05.189Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/62/ae/a696eb424bedff7407801c257d4b1afda455fe40821a2be430e173660e81/watchdog-6.0.0-py3-none-manylinux2014_s390x.whl", hash = "sha256:2cce7cfc2008eb51feb6aab51251fd79b85d9894e98ba847408f662b3395ca3c", size = 79077, upload-time = "2024-11-01T14:07:06.376Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/b5/e8/dbf020b4d98251a9860752a094d09a65e1b436ad181faf929983f697048f/watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:20ffe5b202af80ab4266dcd3e91aae72bf2da48c0d33bdb15c66658e685e94e2", size = 79078, upload-time = "2024-11-01T14:07:07.547Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/07/f6/d0e5b343768e8bcb4cda79f0f2f55051bf26177ecd5651f84c07567461cf/watchdog-6.0.0-py3-none-win32.whl", hash = "sha256:07df1fdd701c5d4c8e55ef6cf55b8f0120fe1aef7ef39a1c6fc6bc2e606d517a", size = 79065, upload-time = "2024-11-01T14:07:09.525Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/db/d9/c495884c6e548fce18a8f40568ff120bc3a4b7b99813081c8ac0c936fa64/watchdog-6.0.0-py3-none-win_amd64.whl", hash = "sha256:cbafb470cf848d93b5d013e2ecb245d4aa1c8fd0504e863ccefa32445359d680", size = 79070, upload-time = "2024-11-01T14:07:10.686Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/33/e8/e40370e6d74ddba47f002a32919d91310d6074130fe4e17dabcafc15cbf1/watchdog-6.0.0-py3-none-win_ia64.whl", hash = "sha256:a1914259fa9e1454315171103c6a30961236f508b9b623eae470268bbcc6a22f", size = 79067, upload-time = "2024-11-01T14:07:11.845Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "watchfiles"
|
name = "watchfiles"
|
||||||
version = "1.1.0"
|
version = "1.1.0"
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue