Merge pull request #360 from ggozad/feat/s3-monitor
S3/object-storage monitoring and s3:// document sources
This commit is contained in:
Yiorgis Gozadinos
GitHub
commit
c34a9dd466
24 changed files with 1543 additions and 16 deletions
12
CHANGELOG.md
12
CHANGELOG.md
|
|
@ -1,6 +1,18 @@
|
|||
# Changelog
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **`s3://` is a first-class document source.** `create_document_from_source`, the CLI `haiku-rag add-src`, and the MCP `add_document_from_url` tool all dispatch on the `s3` URL scheme. Two-stage change detection keeps `metadata["md5"]` semantically uniform across all sources: HEAD ETag matching the stored `metadata["etag"]` short-circuits without GET; if ETag differs but bytes hash to the same MD5 (multipart re-upload, server-side `CopyObject`, SSE mode change), only the etag refreshes — no re-chunk or re-embed. Closes #357.
|
||||
- **S3 / object-storage monitoring.** `monitor.s3: list[S3MonitorEntry]` adds a polling watcher per bucket prefix alongside the existing local-directory watcher. Each entry has its own `poll_interval`, `include_patterns`, `ignore_patterns`, `delete_orphans`, and `storage_options`. The same `serve --monitor` flag enables both. Orphan deletion is per-entry (scoped via `uri LIKE 's3://bucket/prefix/%'`); other buckets and prefixes are never touched.
|
||||
- **`[s3]` optional extra** (`obstore>=0.9`). Required for `s3://` sources and the S3 watcher. Uses obstore — the Python binding to the same Rust `object_store` crate that LanceDB uses internally — so `monitor.s3[*].storage_options` accepts the same dict shape as `lancedb.storage_options`. Empty/missing options fall back to the AWS default credential chain.
|
||||
- **`scripts/run-integration-tests.sh`** — wraps `docker compose up --wait`, `pytest -m integration`, and tear-down so the SeaweedFS-backed integration suite is a one-liner.
|
||||
|
||||
### Documentation
|
||||
|
||||
- New "S3 / Object Storage Monitoring" section in `docs/server.md` and `docs/configuration/processing.md` covering the `[s3]` extra, polling cadence, ETag semantics, credentials, and CLI usage.
|
||||
- New "Deployment Pattern: One Writer, Many Readers" subsection in `docs/configuration/storage.md` documenting the recommended IAM split (one ingestion process + N read-only consumers).
|
||||
|
||||
## [0.45.0] - 2026-05-08
|
||||
|
||||
### Added
|
||||
|
|
|
|||
11
docs/cli.md
11
docs/cli.md
|
|
@ -75,6 +75,17 @@ From directory (recursively adds all supported files):
|
|||
haiku-rag add-src /path/to/documents/
|
||||
```
|
||||
|
||||
From an S3 bucket (requires the `[s3]` extra — see [Server Mode → S3 / Object Storage Monitoring](server.md#s3-object-storage-monitoring)):
|
||||
```bash
|
||||
# AWS S3 with credentials in the default chain (env vars, IAM role, AWS profile)
|
||||
haiku-rag add-src s3://my-bucket/path/to/document.pdf
|
||||
|
||||
# S3-compatible endpoint (SeaweedFS, MinIO, Cloudflare R2, etc.)
|
||||
AWS_ACCESS_KEY_ID=key AWS_SECRET_ACCESS_KEY=secret AWS_REGION=us-east-1 \
|
||||
AWS_ENDPOINT_URL=http://localhost:8333 \
|
||||
haiku-rag add-src s3://my-bucket/path/to/document.pdf
|
||||
```
|
||||
|
||||
!!! note
|
||||
When adding a directory, the same content filters configured for [file monitoring](configuration/processing.md#filtering-monitored-files) are applied. This means `ignore_patterns` and `include_patterns` from your configuration will be used to filter which files are added.
|
||||
|
||||
|
|
|
|||
|
|
@ -322,3 +322,29 @@ Patterns follow [gitignore syntax](https://git-scm.com/docs/gitignore#_pattern_f
|
|||
- `**` matches zero or more directories
|
||||
- `?` matches any single character
|
||||
- `[abc]` matches any character in the set
|
||||
|
||||
### S3 / Object Storage Sources
|
||||
|
||||
In addition to local directories, the watcher can poll S3-compatible buckets (AWS S3, SeaweedFS, MinIO, Cloudflare R2, etc.). Install the `[s3]` extra and configure one or more entries under `monitor.s3`:
|
||||
|
||||
```yaml
|
||||
monitor:
|
||||
s3:
|
||||
- uri: s3://my-bucket/incoming/
|
||||
poll_interval: 300 # seconds between sweeps; default 300
|
||||
include_patterns: ["*.pdf", "*.md"]
|
||||
ignore_patterns: ["draft*"]
|
||||
delete_orphans: true
|
||||
storage_options:
|
||||
endpoint: http://seaweed:8333
|
||||
aws_access_key_id: ${AWS_KEY}
|
||||
aws_secret_access_key: ${AWS_SECRET}
|
||||
region: us-east-1
|
||||
allow_http: "true"
|
||||
```
|
||||
|
||||
Each entry is independent — own poll interval, own include/ignore patterns, own `delete_orphans` setting, own credentials. Omit `storage_options` to fall back to the AWS default credential chain (env vars, IAM role, AWS profile).
|
||||
|
||||
The dict shape matches `lancedb.storage_options` — the same Rust `object_store` library is used by both, so credentials configured for the LanceDB backend can be copy-pasted here.
|
||||
|
||||
See [Server Mode → S3 / Object Storage Monitoring](../server.md#s3-object-storage-monitoring) for behaviour details (ETag-based change detection, orphan-deletion scope, CLI `add-src s3://…`).
|
||||
|
|
|
|||
|
|
@ -74,6 +74,17 @@ The `storage_options` keys are case-insensitive and passed directly to the under
|
|||
|
||||
**Note:** Table optimization is automatically handled by LanceDB Cloud (`db://` URIs) and is disabled for better performance. For object storage backends (S3, Azure, GCS), optimization and vector indexing are still performed normally.
|
||||
|
||||
### Deployment Pattern: One Writer, Many Readers
|
||||
|
||||
LanceDB on S3 supports **exactly one writer + N readers** per database URI. Multiple writers against the same URI can race on the manifest commit and corrupt state — this is a LanceDB property, not something `haiku.rag` enforces.
|
||||
|
||||
The recommended layout for production is "different buckets, same account, separate IAM roles per process":
|
||||
|
||||
- **Ingestion process** — IAM role with `s3:Get/List` on the documents bucket and `s3:Get/Put/Delete` on the LanceDB bucket. Runs `haiku-rag serve --monitor` (with `monitor.s3` entries pointing at the documents bucket). Exactly one such process per LanceDB URI.
|
||||
- **Consumer processes** (1..N) — IAM role with `s3:Get/List` on the LanceDB bucket only. Run `haiku-rag serve --read-only --mcp`, the chat TUI, etc. They never see the documents bucket.
|
||||
|
||||
Each process picks up its own credentials from the AWS default chain (env vars, IAM instance role, AWS profile), so no credentials are hard-coded in the configuration files.
|
||||
|
||||
## Database Creation
|
||||
|
||||
Databases must be explicitly created before use:
|
||||
|
|
|
|||
|
|
@ -120,3 +120,53 @@ The file monitor processes documents using [Docling](https://github.com/DS4SD/do
|
|||
- RST (`.rst`)
|
||||
|
||||
URLs are also supported - the content is fetched and converted to markdown.
|
||||
|
||||
## S3 / Object Storage Monitoring
|
||||
|
||||
The server can also poll S3-compatible object storage (AWS S3, SeaweedFS, MinIO, Cloudflare R2, etc.) for new, modified, and deleted objects, treating each one as a document source.
|
||||
|
||||
Install the optional `[s3]` extra:
|
||||
|
||||
```bash
|
||||
pip install haiku.rag-slim[s3]
|
||||
# or, for the full package:
|
||||
pip install haiku.rag[s3]
|
||||
```
|
||||
|
||||
Configure one or more S3 sources under `monitor.s3` in `haiku.rag.yaml`:
|
||||
|
||||
```yaml
|
||||
monitor:
|
||||
s3:
|
||||
- uri: s3://my-bucket/incoming/
|
||||
poll_interval: 300 # seconds between sweeps; default 300
|
||||
include_patterns: ["*.pdf", "*.md"]
|
||||
ignore_patterns: ["draft*"]
|
||||
delete_orphans: true
|
||||
storage_options:
|
||||
endpoint: http://seaweed:8333
|
||||
aws_access_key_id: ${AWS_KEY}
|
||||
aws_secret_access_key: ${AWS_SECRET}
|
||||
region: us-east-1
|
||||
allow_http: "true"
|
||||
```
|
||||
|
||||
Then start the server with `--monitor` — the same flag enables both local-directory and S3 watchers:
|
||||
|
||||
```bash
|
||||
haiku-rag serve --monitor
|
||||
```
|
||||
|
||||
Each entry in `monitor.s3` runs as its own polling task. On every sweep the watcher lists all objects under the configured prefix, compares each object's S3 ETag against the document's stored `metadata["etag"]`, and only re-fetches keys whose ETag has changed. When the bytes turn out to match the stored MD5 (e.g. the same file was re-uploaded with a different multipart chunk size), the watcher refreshes the etag and skips re-chunking. Otherwise the document is downloaded, chunked, and re-embedded.
|
||||
|
||||
### Credentials
|
||||
|
||||
`storage_options` follows the same convention as `lancedb.storage_options` — the dict is passed straight to obstore (the same Rust `object_store` library LanceDB uses internally), so any keys you've configured there work here too. When `storage_options` is omitted, the watcher falls back to the AWS default credential chain (environment variables, IAM instance role, AWS profile).
|
||||
|
||||
### Orphan deletion scope
|
||||
|
||||
`delete_orphans: true` is per-entry: a watcher only removes documents whose URI starts with that entry's `s3://bucket/prefix/`. Documents from other buckets, prefixes, or local-file sources are never touched.
|
||||
|
||||
### One-off ingestion
|
||||
|
||||
`s3://` URIs are also a first-class source for `haiku-rag add-src` and the MCP `add_document_from_url` tool — see [CLI → Add Documents](cli.md#add-documents).
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ from rich.syntax import Syntax
|
|||
from haiku.rag.client import HaikuRAG, RebuildMode
|
||||
from haiku.rag.config import AppConfig, Config
|
||||
from haiku.rag.mcp import create_mcp_server
|
||||
from haiku.rag.monitor import FileWatcher
|
||||
from haiku.rag.monitor import FileWatcher, S3Watcher
|
||||
from haiku.rag.store.models.document import Document
|
||||
|
||||
if TYPE_CHECKING:
|
||||
|
|
@ -821,6 +821,20 @@ class HaikuRAGApp: # pragma: no cover
|
|||
monitor_task = asyncio.create_task(monitor.observe())
|
||||
tasks.append(monitor_task)
|
||||
|
||||
if self.config.monitor.s3:
|
||||
from haiku.rag.converters import get_converter
|
||||
|
||||
supported_extensions = get_converter(
|
||||
self.config
|
||||
).supported_extensions
|
||||
for entry in self.config.monitor.s3:
|
||||
s3_watcher = S3Watcher(
|
||||
client=client,
|
||||
entry=entry,
|
||||
supported_extensions=supported_extensions,
|
||||
)
|
||||
tasks.append(asyncio.create_task(s3_watcher.observe()))
|
||||
|
||||
# Start MCP server if enabled
|
||||
if enable_mcp:
|
||||
server = create_mcp_server(
|
||||
|
|
|
|||
|
|
@ -210,10 +210,18 @@ class HaikuRAG:
|
|||
title: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
uri: str | None = None,
|
||||
storage_options: dict[str, str] | None = None,
|
||||
) -> Document | list[Document]:
|
||||
from haiku.rag.client.documents import create_document_from_source
|
||||
|
||||
return await create_document_from_source(self, source, title, metadata, uri=uri)
|
||||
return await create_document_from_source(
|
||||
self,
|
||||
source,
|
||||
title,
|
||||
metadata,
|
||||
uri=uri,
|
||||
storage_options=storage_options,
|
||||
)
|
||||
|
||||
async def update_document(
|
||||
self,
|
||||
|
|
|
|||
|
|
@ -197,6 +197,7 @@ async def create_document_from_source(
|
|||
title: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
uri: str | None = None,
|
||||
storage_options: dict[str, str] | None = None,
|
||||
) -> Document | list[Document]:
|
||||
"""Create or update document(s) from a file path, directory, or URL.
|
||||
|
||||
|
|
@ -221,6 +222,15 @@ async def create_document_from_source(
|
|||
return await _create_or_update_document_from_url(
|
||||
client, source_str, title=title, metadata=metadata, uri=uri
|
||||
)
|
||||
elif parsed_url.scheme == "s3":
|
||||
return await _create_or_update_document_from_s3(
|
||||
client,
|
||||
source_str,
|
||||
storage_options=storage_options,
|
||||
title=title,
|
||||
metadata=metadata,
|
||||
uri=uri,
|
||||
)
|
||||
elif parsed_url.scheme == "file":
|
||||
source_path = Path(parsed_url.path)
|
||||
else:
|
||||
|
|
@ -441,6 +451,140 @@ async def _create_or_update_document_from_url(
|
|||
)
|
||||
|
||||
|
||||
async def _create_or_update_document_from_s3(
|
||||
client: "HaikuRAG",
|
||||
url: str,
|
||||
*,
|
||||
storage_options: dict[str, str] | None = None,
|
||||
title: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
uri: str | None = None,
|
||||
) -> Document:
|
||||
"""Create or update a document from an s3:// URL.
|
||||
|
||||
Two-stage change detection:
|
||||
- HEAD ETag matches stored metadata["etag"] → skip GET and re-chunk.
|
||||
- ETag differs but content MD5 matches → refresh etag only, no re-chunk.
|
||||
- Otherwise download, convert, chunk, embed.
|
||||
|
||||
``uri`` overrides the s3:// URL as the stored document identifier.
|
||||
"""
|
||||
import obstore # type: ignore[import-not-found]
|
||||
|
||||
from haiku.rag.client.processing import get_extension_from_content_type_or_url
|
||||
from haiku.rag.embeddings import embed_chunks
|
||||
from haiku.rag.s3 import make_s3_store
|
||||
|
||||
metadata = metadata or {}
|
||||
stored_uri = uri if uri is not None else url
|
||||
|
||||
parsed = urlparse(url)
|
||||
bucket = parsed.netloc
|
||||
key = parsed.path.lstrip("/")
|
||||
if not bucket or not key:
|
||||
raise ValueError(f"Invalid S3 URI: {url}")
|
||||
|
||||
converter = get_converter(client._config)
|
||||
supported_extensions = converter.supported_extensions
|
||||
|
||||
store = make_s3_store(bucket, storage_options)
|
||||
|
||||
head = await obstore.head_async(store, key)
|
||||
etag = (head.get("e_tag") or "").strip('"')
|
||||
|
||||
existing_doc = await client.get_document_by_uri(stored_uri)
|
||||
if existing_doc and existing_doc.metadata.get("etag") == etag:
|
||||
updated = False
|
||||
if title is not None and title != existing_doc.title:
|
||||
existing_doc.title = title
|
||||
updated = True
|
||||
|
||||
merged_metadata = {**(existing_doc.metadata or {}), **metadata}
|
||||
if merged_metadata != existing_doc.metadata:
|
||||
existing_doc.metadata = merged_metadata
|
||||
updated = True
|
||||
|
||||
if updated:
|
||||
return await client.document_repository.update(existing_doc)
|
||||
return existing_doc
|
||||
|
||||
content_type, _ = mimetypes.guess_type(key)
|
||||
if not content_type:
|
||||
content_type = "application/octet-stream"
|
||||
|
||||
file_extension = get_extension_from_content_type_or_url(url, content_type)
|
||||
if file_extension not in supported_extensions:
|
||||
raise ValueError(
|
||||
f"Unsupported content type/extension: {content_type}/{file_extension}"
|
||||
)
|
||||
|
||||
get_resp = await obstore.get_async(store, key)
|
||||
body = await get_resp.bytes_async()
|
||||
|
||||
md5_hash = hashlib.md5(body, usedforsecurity=False).hexdigest()
|
||||
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="wb", suffix=file_extension, delete=False
|
||||
) as temp_file:
|
||||
temp_file.write(body)
|
||||
temp_file.flush()
|
||||
temp_path = Path(temp_file.name)
|
||||
|
||||
metadata.update({"contentType": content_type, "md5": md5_hash, "etag": etag})
|
||||
|
||||
if existing_doc and existing_doc.metadata.get("md5") == md5_hash:
|
||||
temp_path.unlink(missing_ok=True)
|
||||
merged_metadata = {**(existing_doc.metadata or {}), **metadata}
|
||||
updated = False
|
||||
if merged_metadata != existing_doc.metadata:
|
||||
existing_doc.metadata = merged_metadata
|
||||
updated = True
|
||||
if title is not None and title != existing_doc.title:
|
||||
existing_doc.title = title
|
||||
updated = True
|
||||
if updated:
|
||||
return await client.document_repository.update(existing_doc)
|
||||
return existing_doc
|
||||
|
||||
try:
|
||||
docling_document = await client.convert(temp_path)
|
||||
chunks = await client.chunk(docling_document)
|
||||
embedded_chunks = await embed_chunks(chunks, client._config)
|
||||
finally:
|
||||
temp_path.unlink(missing_ok=True)
|
||||
|
||||
stored_content = docling_document.export_to_markdown()
|
||||
|
||||
if existing_doc:
|
||||
existing_doc.content = stored_content
|
||||
existing_doc.metadata = metadata
|
||||
existing_doc.set_docling(docling_document)
|
||||
if title is not None:
|
||||
existing_doc.title = title
|
||||
elif existing_doc.title is None:
|
||||
existing_doc.title = await resolve_title(
|
||||
client._config, docling_document, stored_content
|
||||
)
|
||||
return await _update_document_with_chunks(
|
||||
client, existing_doc, embedded_chunks, docling_document
|
||||
)
|
||||
else:
|
||||
if title is None:
|
||||
title = await resolve_title(
|
||||
client._config, docling_document, stored_content
|
||||
)
|
||||
document = Document(
|
||||
content=stored_content,
|
||||
uri=stored_uri,
|
||||
title=title,
|
||||
metadata=metadata,
|
||||
)
|
||||
document.set_docling(docling_document)
|
||||
return await _store_document_with_chunks(
|
||||
client, document, embedded_chunks, docling_document
|
||||
)
|
||||
|
||||
|
||||
async def update_document(
|
||||
client: "HaikuRAG",
|
||||
document_id: str,
|
||||
|
|
@ -525,7 +669,7 @@ def check_source_accessible(uri: str) -> bool:
|
|||
try:
|
||||
if parsed_url.scheme == "file":
|
||||
return Path(parsed_url.path).exists()
|
||||
elif parsed_url.scheme in ("http", "https"):
|
||||
elif parsed_url.scheme in ("http", "https", "s3"):
|
||||
return True
|
||||
return False
|
||||
except Exception:
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ from haiku.rag.config.models import (
|
|||
QAConfig,
|
||||
RerankingConfig,
|
||||
ResearchConfig,
|
||||
S3MonitorEntry,
|
||||
StorageConfig,
|
||||
)
|
||||
|
||||
|
|
@ -37,6 +38,7 @@ __all__ = [
|
|||
"QAConfig",
|
||||
"RerankingConfig",
|
||||
"ResearchConfig",
|
||||
"S3MonitorEntry",
|
||||
"StorageConfig",
|
||||
"find_config_file",
|
||||
"generate_default_config",
|
||||
|
|
|
|||
|
|
@ -51,11 +51,21 @@ class StorageConfig(BaseModel):
|
|||
vacuum_retention_seconds: int = 86400
|
||||
|
||||
|
||||
class S3MonitorEntry(BaseModel):
|
||||
uri: str
|
||||
storage_options: dict[str, str] = Field(default_factory=dict)
|
||||
poll_interval: int = 300
|
||||
ignore_patterns: list[str] = []
|
||||
include_patterns: list[str] = []
|
||||
delete_orphans: bool = False
|
||||
|
||||
|
||||
class MonitorConfig(BaseModel):
|
||||
directories: list[Path] = []
|
||||
ignore_patterns: list[str] = []
|
||||
include_patterns: list[str] = []
|
||||
delete_orphans: bool = False
|
||||
s3: list[S3MonitorEntry] = []
|
||||
|
||||
|
||||
class LanceDBConfig(BaseModel):
|
||||
|
|
|
|||
|
|
@ -2,13 +2,15 @@ import asyncio
|
|||
import logging
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import pathspec
|
||||
from watchfiles import Change, DefaultFilter, awatch
|
||||
|
||||
from haiku.rag.client import HaikuRAG
|
||||
from haiku.rag.config import AppConfig, Config
|
||||
from haiku.rag.config import AppConfig, Config, S3MonitorEntry
|
||||
from haiku.rag.store.models.document import Document
|
||||
from haiku.rag.utils import escape_sql_string
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
|
@ -215,3 +217,104 @@ class FileWatcher:
|
|||
logger.info(f"Deleted document {existing_doc.id} for {file}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete document for {file}: {e}")
|
||||
|
||||
|
||||
class S3Watcher:
|
||||
"""Polls an S3 prefix and keeps documents in sync with the index.
|
||||
|
||||
Uses ListObjectsV2 ETags as the cheap-skip key. When a key's listing
|
||||
ETag differs from the stored `metadata["etag"]`, delegates to
|
||||
`client.create_document_from_source` which performs the full
|
||||
HeadObject + GetObject + MD5 compare two-stage detection.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
client: HaikuRAG,
|
||||
entry: S3MonitorEntry,
|
||||
supported_extensions: list[str],
|
||||
) -> None:
|
||||
from haiku.rag.s3 import make_s3_store
|
||||
|
||||
parsed = urlparse(entry.uri)
|
||||
if not parsed.netloc:
|
||||
raise ValueError(f"Invalid S3 monitor URI: {entry.uri}")
|
||||
|
||||
self.client = client
|
||||
self.entry = entry
|
||||
self.bucket = parsed.netloc
|
||||
self.prefix = parsed.path.lstrip("/")
|
||||
self.uri_prefix = f"s3://{self.bucket}/{self.prefix}"
|
||||
self._make_s3_store = make_s3_store
|
||||
self.filter = FileFilter(
|
||||
ignore_patterns=entry.ignore_patterns or None,
|
||||
include_patterns=entry.include_patterns or None,
|
||||
supported_extensions=supported_extensions,
|
||||
)
|
||||
|
||||
async def observe(self) -> None:
|
||||
logger.info(
|
||||
f"Watching S3 {self.entry.uri} (poll_interval={self.entry.poll_interval}s)"
|
||||
)
|
||||
await self.refresh()
|
||||
while True:
|
||||
await asyncio.sleep(self.entry.poll_interval)
|
||||
try:
|
||||
await self.refresh()
|
||||
except Exception as e:
|
||||
logger.error(f"S3 watcher refresh failed for {self.entry.uri}: {e}")
|
||||
|
||||
async def refresh(self) -> None:
|
||||
import obstore # type: ignore[import-not-found]
|
||||
|
||||
uris_seen: dict[str, str] = {}
|
||||
store = self._make_s3_store(self.bucket, self.entry.storage_options)
|
||||
|
||||
async for batch in obstore.list(store, prefix=self.prefix or None):
|
||||
for obj in batch:
|
||||
key = obj["path"]
|
||||
if not self.filter.include_file(key):
|
||||
continue
|
||||
uri = f"s3://{self.bucket}/{key}"
|
||||
uris_seen[uri] = (obj.get("e_tag") or "").strip('"')
|
||||
|
||||
existing_etags = await self._existing_etags_under_prefix()
|
||||
|
||||
for uri, etag in uris_seen.items():
|
||||
if existing_etags.get(uri) == etag:
|
||||
continue
|
||||
await self._upsert_object(uri)
|
||||
|
||||
if self.entry.delete_orphans:
|
||||
await self._delete_orphans(set(uris_seen.keys()), existing_etags)
|
||||
|
||||
async def _existing_etags_under_prefix(self) -> dict[str, str]:
|
||||
safe_prefix = escape_sql_string(self.uri_prefix)
|
||||
docs = await self.client.list_documents(filter=f"uri LIKE '{safe_prefix}%'")
|
||||
return {
|
||||
doc.uri: (doc.metadata or {}).get("etag", "") for doc in docs if doc.uri
|
||||
}
|
||||
|
||||
async def _upsert_object(self, uri: str) -> Document | None:
|
||||
try:
|
||||
result = await self.client.create_document_from_source(
|
||||
uri, storage_options=self.entry.storage_options
|
||||
)
|
||||
doc = result if isinstance(result, Document) else result[0]
|
||||
logger.info(f"Upserted document {doc.id} from {uri}")
|
||||
return doc
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to upsert document from {uri}: {e}")
|
||||
return None
|
||||
|
||||
async def _delete_orphans(
|
||||
self, uris_seen: set[str], existing_etags: dict[str, str]
|
||||
) -> None:
|
||||
for uri in existing_etags.keys() - uris_seen:
|
||||
try:
|
||||
doc = await self.client.get_document_by_uri(uri)
|
||||
if doc and doc.id:
|
||||
await self.client.delete_document(doc.id)
|
||||
logger.info(f"Deleted orphaned document {doc.id} for {uri}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete orphan {uri}: {e}")
|
||||
|
|
|
|||
39
haiku_rag_slim/haiku/rag/s3.py
Normal file
39
haiku_rag_slim/haiku/rag/s3.py
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
from typing import Any
|
||||
|
||||
|
||||
def make_s3_store(bucket: str, storage_options: dict[str, str] | None) -> Any:
|
||||
"""Build an obstore `S3Store` from LanceDB-style storage_options.
|
||||
|
||||
Accepts the same dict shape as `LanceDBConfig.storage_options` (the same
|
||||
Rust `object_store` crate is used by both LanceDB and obstore, so the keys
|
||||
line up). Recognized keys include aws_access_key_id, aws_secret_access_key,
|
||||
aws_session_token, region (or aws_region), endpoint (or aws_endpoint),
|
||||
allow_http. Empty/missing keys fall back to the AWS default credential
|
||||
chain (environment variables, IAM role, AWS profile).
|
||||
"""
|
||||
try:
|
||||
from obstore.store import (
|
||||
S3Store, # type: ignore[import-not-found]
|
||||
)
|
||||
except ImportError as e:
|
||||
raise ImportError(
|
||||
"obstore is required for s3:// sources. "
|
||||
"Install with: pip install haiku.rag-slim[s3]"
|
||||
) from e
|
||||
|
||||
options = dict(storage_options or {})
|
||||
allow_http = str(options.pop("allow_http", "")).lower() == "true"
|
||||
|
||||
# Custom endpoints (SeaweedFS, MinIO, etc.) need path-style requests; AWS
|
||||
# accepts virtual-hosted-style by default.
|
||||
has_custom_endpoint = bool(options.get("endpoint") or options.get("aws_endpoint"))
|
||||
|
||||
kwargs: dict[str, Any] = {}
|
||||
if options:
|
||||
kwargs["config"] = options
|
||||
if allow_http:
|
||||
kwargs["client_options"] = {"allow_http": True}
|
||||
if has_custom_endpoint:
|
||||
kwargs["virtual_hosted_style_request"] = False
|
||||
|
||||
return S3Store(bucket, **kwargs)
|
||||
|
|
@ -43,6 +43,8 @@ dependencies = [
|
|||
[project.optional-dependencies]
|
||||
# Document processing
|
||||
docling = ["docling>=2.91.0", "opencv-python-headless>=4.13.0.92"]
|
||||
# S3 / object-storage monitoring
|
||||
s3 = ["obstore>=0.9,<0.10"]
|
||||
# Embedding providers
|
||||
voyageai = ["pydantic-ai-slim[voyageai]"]
|
||||
# Rerankers
|
||||
|
|
|
|||
|
|
@ -38,6 +38,7 @@ haiku-rag = "haiku.rag.cli:cli"
|
|||
|
||||
[project.optional-dependencies]
|
||||
tui = ["textual>=1.0.0"]
|
||||
s3 = ["haiku.rag-slim[s3]==0.45.0"]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
|
|
@ -68,6 +69,7 @@ members = ["haiku_rag_slim", "evaluations"]
|
|||
dev = [
|
||||
"haiku.rag-evals",
|
||||
"datasets>=4.8.4",
|
||||
"obstore>=0.9,<0.10",
|
||||
"mkdocs>=1.6.1",
|
||||
"mkdocs-material>=9.7.6",
|
||||
"pre-commit>=4.5.1",
|
||||
|
|
|
|||
24
scripts/run-integration-tests.sh
Executable file
24
scripts/run-integration-tests.sh
Executable file
|
|
@ -0,0 +1,24 @@
|
|||
#!/bin/bash
|
||||
# Run integration tests against a local SeaweedFS instance.
|
||||
#
|
||||
# Brings up SeaweedFS via docker compose, runs every test marked `integration`,
|
||||
# and tears down on exit. Extra args are forwarded to pytest:
|
||||
#
|
||||
# ./scripts/run-integration-tests.sh
|
||||
# ./scripts/run-integration-tests.sh tests/test_s3_integration.py::test_s3_watcher_initial_sweep
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
COMPOSE_FILE="$REPO_ROOT/tests/docker/docker-compose.s3.yml"
|
||||
|
||||
cleanup() {
|
||||
echo "==> Tearing down SeaweedFS"
|
||||
docker compose -f "$COMPOSE_FILE" down -v >/dev/null 2>&1 || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "==> Bringing up SeaweedFS"
|
||||
docker compose -f "$COMPOSE_FILE" up -d --wait
|
||||
|
||||
echo "==> Running integration tests"
|
||||
uv run pytest -m integration "$@"
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -3,6 +3,7 @@
|
|||
# Stop after:
|
||||
# docker compose -f tests/docker/docker-compose.s3.yml down -v
|
||||
|
||||
import importlib.util
|
||||
import socket
|
||||
from uuid import uuid4
|
||||
|
||||
|
|
@ -10,9 +11,11 @@ import pytest
|
|||
|
||||
from haiku.rag.app import HaikuRAGApp
|
||||
from haiku.rag.client import HaikuRAG
|
||||
from haiku.rag.config.models import AppConfig, LanceDBConfig
|
||||
from haiku.rag.config.models import AppConfig, LanceDBConfig, S3MonitorEntry
|
||||
from haiku.rag.store.engine import Store
|
||||
|
||||
HAS_OBSTORE = importlib.util.find_spec("obstore") is not None
|
||||
|
||||
S3_ENDPOINT = "http://localhost:8333"
|
||||
S3_BUCKET = "test-bucket"
|
||||
S3_STORAGE_OPTIONS = {
|
||||
|
|
@ -149,3 +152,151 @@ async def test_app_info_empty_db(tmp_path, capsys):
|
|||
|
||||
out = capsys.readouterr().out
|
||||
assert "Database is empty" in out
|
||||
|
||||
|
||||
# ----------------------- S3 watcher integration tests ----------------------- #
|
||||
# These exercise S3Watcher against the live SeaweedFS instance. Documents are
|
||||
# uploaded as raw S3 objects under a unique per-test prefix; the watcher's
|
||||
# refresh() is invoked directly so tests stay deterministic. LanceDB stays
|
||||
# local — these tests verify the watcher path, not LanceDB-on-S3.
|
||||
|
||||
|
||||
_obstore_required = pytest.mark.skipif(
|
||||
not HAS_OBSTORE,
|
||||
reason="obstore not installed (uv sync --extra s3)",
|
||||
)
|
||||
|
||||
|
||||
def _watcher_store():
|
||||
from haiku.rag.s3 import make_s3_store
|
||||
|
||||
return make_s3_store(S3_BUCKET, S3_STORAGE_OPTIONS)
|
||||
|
||||
|
||||
async def _put_object(prefix: str, key: str, body: bytes) -> None:
|
||||
import obstore
|
||||
|
||||
await obstore.put_async(_watcher_store(), f"{prefix}/{key}", body)
|
||||
|
||||
|
||||
async def _delete_object(prefix: str, key: str) -> None:
|
||||
import obstore
|
||||
|
||||
await obstore.delete_async(_watcher_store(), f"{prefix}/{key}")
|
||||
|
||||
|
||||
def _watcher_entry(prefix: str, **overrides) -> S3MonitorEntry:
|
||||
return S3MonitorEntry(
|
||||
uri=overrides.pop("uri", f"s3://{S3_BUCKET}/{prefix}/"),
|
||||
storage_options=overrides.pop("storage_options", S3_STORAGE_OPTIONS),
|
||||
include_patterns=overrides.pop("include_patterns", ["*.txt"]),
|
||||
delete_orphans=overrides.pop("delete_orphans", False),
|
||||
poll_interval=overrides.pop("poll_interval", 60),
|
||||
**overrides,
|
||||
)
|
||||
|
||||
|
||||
@_obstore_required
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_initial_sweep(tmp_path):
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
prefix = f"watcher-init-{uuid4().hex[:8]}"
|
||||
await _put_object(prefix, "alpha.txt", b"alpha content")
|
||||
await _put_object(prefix, "beta.txt", b"beta content")
|
||||
|
||||
async with HaikuRAG(tmp_path / "db.lancedb", create=True) as rag:
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_watcher_entry(prefix),
|
||||
supported_extensions=[".txt", ".md", ".pdf"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
|
||||
docs = await rag.list_documents()
|
||||
uris = sorted(d.uri or "" for d in docs)
|
||||
|
||||
assert uris == [
|
||||
f"s3://{S3_BUCKET}/{prefix}/alpha.txt",
|
||||
f"s3://{S3_BUCKET}/{prefix}/beta.txt",
|
||||
]
|
||||
|
||||
|
||||
@_obstore_required
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_detects_new_object(tmp_path):
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
prefix = f"watcher-new-{uuid4().hex[:8]}"
|
||||
await _put_object(prefix, "first.txt", b"first content")
|
||||
|
||||
async with HaikuRAG(tmp_path / "db.lancedb", create=True) as rag:
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_watcher_entry(prefix),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
assert await rag.count_documents() == 1
|
||||
|
||||
await _put_object(prefix, "second.txt", b"second content")
|
||||
await watcher.refresh()
|
||||
assert await rag.count_documents() == 2
|
||||
|
||||
|
||||
@_obstore_required
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_detects_modified_object(tmp_path):
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
prefix = f"watcher-mod-{uuid4().hex[:8]}"
|
||||
uri = f"s3://{S3_BUCKET}/{prefix}/file.txt"
|
||||
|
||||
await _put_object(prefix, "file.txt", b"original content")
|
||||
|
||||
async with HaikuRAG(tmp_path / "db.lancedb", create=True) as rag:
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_watcher_entry(prefix),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
|
||||
first = await rag.get_document_by_uri(uri)
|
||||
assert first is not None
|
||||
first_md5 = first.metadata["md5"]
|
||||
|
||||
await _put_object(prefix, "file.txt", b"new content body")
|
||||
await watcher.refresh()
|
||||
|
||||
second = await rag.get_document_by_uri(uri)
|
||||
assert second is not None
|
||||
assert second.id == first.id
|
||||
assert second.metadata["md5"] != first_md5
|
||||
assert "new content body" in second.content
|
||||
|
||||
|
||||
@_obstore_required
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_orphan_deletion(tmp_path):
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
prefix = f"watcher-orphan-{uuid4().hex[:8]}"
|
||||
await _put_object(prefix, "kept.txt", b"keep me")
|
||||
await _put_object(prefix, "doomed.txt", b"will be deleted")
|
||||
|
||||
async with HaikuRAG(tmp_path / "db.lancedb", create=True) as rag:
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_watcher_entry(prefix, delete_orphans=True),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
assert await rag.count_documents() == 2
|
||||
|
||||
await _delete_object(prefix, "doomed.txt")
|
||||
await watcher.refresh()
|
||||
|
||||
docs = await rag.list_documents()
|
||||
assert len(docs) == 1
|
||||
assert docs[0].uri == f"s3://{S3_BUCKET}/{prefix}/kept.txt"
|
||||
|
|
|
|||
403
tests/test_s3_monitor.py
Normal file
403
tests/test_s3_monitor.py
Normal file
|
|
@ -0,0 +1,403 @@
|
|||
import asyncio
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from haiku.rag.client import HaikuRAG
|
||||
from haiku.rag.config import AppConfig, MonitorConfig, S3MonitorEntry
|
||||
from haiku.rag.store.models.document import Document
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def s3_listing(monkeypatch):
|
||||
"""Patch `obstore.list_obs` with an async-iterator returning controllable batches.
|
||||
|
||||
Returns `(set_batches, list_mock)`. `set_batches([[meta, ...], ...])`
|
||||
seeds the next call's pages.
|
||||
"""
|
||||
import obstore
|
||||
|
||||
batches: list[list[MagicMock]] = []
|
||||
|
||||
def list_obs(_store, *_, **__):
|
||||
async def _iter():
|
||||
for batch in batches:
|
||||
yield batch
|
||||
|
||||
return _iter()
|
||||
|
||||
list_mock = MagicMock(side_effect=list_obs)
|
||||
monkeypatch.setattr(obstore, "list", list_mock)
|
||||
|
||||
def set_batches(new_batches):
|
||||
batches.clear()
|
||||
batches.extend(new_batches)
|
||||
|
||||
return set_batches, list_mock
|
||||
|
||||
|
||||
def _meta(path: str, etag: str) -> dict:
|
||||
# Real obstore ObjectMeta is a TypedDict; raw S3 ETags include quotes.
|
||||
return {
|
||||
"path": path,
|
||||
"e_tag": f'"{etag}"',
|
||||
"size": 0,
|
||||
"last_modified": None,
|
||||
}
|
||||
|
||||
|
||||
def _entry(**kwargs) -> S3MonitorEntry:
|
||||
return S3MonitorEntry(
|
||||
uri=kwargs.pop("uri", "s3://my-bucket/incoming/"),
|
||||
poll_interval=kwargs.pop("poll_interval", 60),
|
||||
delete_orphans=kwargs.pop("delete_orphans", False),
|
||||
ignore_patterns=kwargs.pop("ignore_patterns", []),
|
||||
include_patterns=kwargs.pop("include_patterns", []),
|
||||
storage_options=kwargs.pop("storage_options", {}),
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
def _doc(uri: str, etag: str, doc_id: str | None = None) -> Document:
|
||||
return Document(
|
||||
id=doc_id or uri,
|
||||
content="...",
|
||||
uri=uri,
|
||||
metadata={"etag": etag, "md5": "deadbeef"},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_refresh_upserts_new_objects(s3_listing):
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[_meta("incoming/a.txt", "abc"), _meta("incoming/b.txt", "def")]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = []
|
||||
rag.create_document_from_source.return_value = Document(
|
||||
id="x", content="...", uri="s3://my-bucket/incoming/a.txt"
|
||||
)
|
||||
|
||||
watcher = S3Watcher(
|
||||
client=rag, entry=_entry(), supported_extensions=[".txt", ".md", ".pdf"]
|
||||
)
|
||||
await watcher.refresh()
|
||||
|
||||
assert rag.create_document_from_source.await_count == 2
|
||||
called_uris = {c.args[0] for c in rag.create_document_from_source.await_args_list}
|
||||
assert called_uris == {
|
||||
"s3://my-bucket/incoming/a.txt",
|
||||
"s3://my-bucket/incoming/b.txt",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_skips_unchanged_etag(s3_listing):
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[_meta("incoming/a.txt", "abc")]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = [_doc("s3://my-bucket/incoming/a.txt", "abc")]
|
||||
|
||||
watcher = S3Watcher(client=rag, entry=_entry(), supported_extensions=[".txt"])
|
||||
await watcher.refresh()
|
||||
|
||||
rag.create_document_from_source.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_upserts_when_etag_differs(s3_listing):
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[_meta("incoming/a.txt", "new")]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = [_doc("s3://my-bucket/incoming/a.txt", "old")]
|
||||
rag.create_document_from_source.return_value = Document(
|
||||
id="x", content="...", uri="s3://my-bucket/incoming/a.txt"
|
||||
)
|
||||
|
||||
watcher = S3Watcher(client=rag, entry=_entry(), supported_extensions=[".txt"])
|
||||
await watcher.refresh()
|
||||
|
||||
rag.create_document_from_source.assert_awaited_once_with(
|
||||
"s3://my-bucket/incoming/a.txt", storage_options={}
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_strips_etag_quotes(s3_listing):
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[_meta("incoming/a.txt", "abc")]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = [
|
||||
_doc("s3://my-bucket/incoming/a.txt", "abc") # already stripped in storage
|
||||
]
|
||||
|
||||
watcher = S3Watcher(client=rag, entry=_entry(), supported_extensions=[".txt"])
|
||||
await watcher.refresh()
|
||||
|
||||
rag.create_document_from_source.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_deletes_orphans_when_enabled(s3_listing):
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[_meta("incoming/a.txt", "abc")]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
a_doc = _doc("s3://my-bucket/incoming/a.txt", "abc", doc_id="a-id")
|
||||
orphan = _doc("s3://my-bucket/incoming/old.txt", "stale", doc_id="orphan-id")
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = [a_doc, orphan]
|
||||
rag.get_document_by_uri.return_value = orphan
|
||||
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_entry(delete_orphans=True),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
|
||||
rag.delete_document.assert_awaited_once_with("orphan-id")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_does_not_delete_orphans_when_disabled(s3_listing):
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
orphan = _doc("s3://my-bucket/incoming/old.txt", "stale", doc_id="orphan-id")
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = [orphan]
|
||||
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_entry(delete_orphans=False),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
|
||||
rag.delete_document.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_orphan_scope_is_per_entry(s3_listing):
|
||||
"""A doc under a different bucket prefix must not be touched."""
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = [] # filter scopes to my-bucket
|
||||
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_entry(delete_orphans=True),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
|
||||
rag.list_documents.assert_awaited_once()
|
||||
filter_kwarg = rag.list_documents.await_args.kwargs["filter"]
|
||||
assert filter_kwarg == "uri LIKE 's3://my-bucket/incoming/%'"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_applies_include_and_ignore_patterns(s3_listing):
|
||||
set_batches, _ = s3_listing
|
||||
set_batches(
|
||||
[
|
||||
[
|
||||
_meta("incoming/keep.md", "1"),
|
||||
_meta("incoming/draft.md", "2"),
|
||||
_meta("incoming/skip.txt", "3"),
|
||||
]
|
||||
]
|
||||
)
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = []
|
||||
rag.create_document_from_source.return_value = Document(
|
||||
id="x", content="...", uri="s3://my-bucket/incoming/keep.md"
|
||||
)
|
||||
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_entry(include_patterns=["*.md"], ignore_patterns=["draft*"]),
|
||||
supported_extensions=[".md", ".txt"],
|
||||
)
|
||||
await watcher.refresh()
|
||||
|
||||
assert rag.create_document_from_source.await_count == 1
|
||||
assert (
|
||||
rag.create_document_from_source.await_args.args[0]
|
||||
== "s3://my-bucket/incoming/keep.md"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_observe_survives_transient_list_failure(s3_listing):
|
||||
"""First refresh succeeds; second refresh raises; loop survives and recovers."""
|
||||
set_batches, list_mock = s3_listing
|
||||
|
||||
pages_initial = [[_meta("incoming/a.txt", "abc")]]
|
||||
pages_after = [[_meta("incoming/a.txt", "abc")]]
|
||||
|
||||
paginate_calls = {"n": 0}
|
||||
|
||||
def list_obs_side_effect(_store, *_, **__):
|
||||
paginate_calls["n"] += 1
|
||||
if paginate_calls["n"] == 2:
|
||||
raise RuntimeError("transient list failure")
|
||||
|
||||
async def _iter():
|
||||
for batch in pages_after if paginate_calls["n"] > 1 else pages_initial:
|
||||
yield batch
|
||||
|
||||
return _iter()
|
||||
|
||||
list_mock.side_effect = list_obs_side_effect
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = []
|
||||
rag.create_document_from_source.return_value = Document(
|
||||
id="x", content="...", uri="s3://my-bucket/incoming/a.txt"
|
||||
)
|
||||
|
||||
watcher = S3Watcher(
|
||||
client=rag,
|
||||
entry=_entry(poll_interval=0),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
task = asyncio.create_task(watcher.observe())
|
||||
|
||||
for _ in range(20):
|
||||
await asyncio.sleep(0)
|
||||
if paginate_calls["n"] >= 3:
|
||||
break
|
||||
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
assert paginate_calls["n"] >= 3 # loop kept going past the failure
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_invalid_uri_rejected():
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
with pytest.raises(ValueError, match="Invalid S3 monitor URI"):
|
||||
S3Watcher(
|
||||
client=rag,
|
||||
entry=S3MonitorEntry(uri="s3://"),
|
||||
supported_extensions=[".txt"],
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_s3_watcher_upsert_failure_does_not_abort_sweep(s3_listing):
|
||||
"""A failing upsert doesn't propagate; the refresh keeps processing siblings."""
|
||||
set_batches, _ = s3_listing
|
||||
set_batches([[_meta("incoming/bad.txt", "abc"), _meta("incoming/good.txt", "def")]])
|
||||
|
||||
from haiku.rag.monitor import S3Watcher
|
||||
|
||||
rag = AsyncMock(spec=HaikuRAG)
|
||||
rag.list_documents.return_value = []
|
||||
|
||||
good_doc = Document(
|
||||
id="good-id", content="...", uri="s3://my-bucket/incoming/good.txt"
|
||||
)
|
||||
|
||||
async def maybe_fail(uri, **_):
|
||||
if uri.endswith("bad.txt"):
|
||||
raise RuntimeError("boom")
|
||||
return good_doc
|
||||
|
||||
rag.create_document_from_source.side_effect = maybe_fail
|
||||
|
||||
watcher = S3Watcher(client=rag, entry=_entry(), supported_extensions=[".txt"])
|
||||
|
||||
# The failing upsert must not propagate out of refresh().
|
||||
await watcher.refresh()
|
||||
|
||||
# Both objects were attempted — the first failure didn't abort the sibling.
|
||||
assert rag.create_document_from_source.await_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_serve_starts_one_s3_task_per_entry(monkeypatch, s3_listing):
|
||||
"""`serve` wires one S3Watcher task per MonitorConfig.s3 entry."""
|
||||
from haiku.rag import app as app_module
|
||||
|
||||
original_create_task = asyncio.create_task
|
||||
|
||||
def tracking_create_task(coro, *args, **kwargs):
|
||||
return original_create_task(coro, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(app_module.asyncio, "create_task", tracking_create_task)
|
||||
|
||||
config = AppConfig(
|
||||
monitor=MonitorConfig(
|
||||
s3=[
|
||||
S3MonitorEntry(uri="s3://bucket-a/x/"),
|
||||
S3MonitorEntry(uri="s3://bucket-b/y/"),
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
fw_observe_calls = {"n": 0}
|
||||
|
||||
async def fake_fw_observe(self):
|
||||
fw_observe_calls["n"] += 1
|
||||
|
||||
monkeypatch.setattr(app_module.FileWatcher, "observe", fake_fw_observe)
|
||||
|
||||
s3_observe_calls = {"n": 0}
|
||||
|
||||
async def fake_s3_observe(self):
|
||||
s3_observe_calls["n"] += 1
|
||||
|
||||
monkeypatch.setattr(app_module.S3Watcher, "observe", fake_s3_observe)
|
||||
|
||||
class _Conv:
|
||||
supported_extensions = [".txt"]
|
||||
|
||||
monkeypatch.setattr("haiku.rag.converters.get_converter", lambda cfg: _Conv())
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
db_path = Path(tmp) / "db.lancedb"
|
||||
app = app_module.HaikuRAGApp(db_path=db_path, config=config)
|
||||
async with HaikuRAG(db_path, config=config, create=True):
|
||||
pass # create the database
|
||||
|
||||
await app.serve(enable_monitor=True, enable_mcp=False)
|
||||
|
||||
assert fw_observe_calls["n"] == 1
|
||||
assert s3_observe_calls["n"] == 2
|
||||
205
tests/test_s3_source.py
Normal file
205
tests/test_s3_source.py
Normal file
|
|
@ -0,0 +1,205 @@
|
|||
import sys
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from haiku.rag.client import HaikuRAG
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_obstore_io(monkeypatch):
|
||||
"""Patch `obstore.head_async` and `obstore.get_async` with controllable mocks.
|
||||
|
||||
Returns a tuple `(head_async, get_async)`. Tests assign `.return_value` to
|
||||
each to seed responses. Real `obstore.store.S3Store` is left intact —
|
||||
constructing the store still exercises the storage_options path.
|
||||
"""
|
||||
import obstore
|
||||
|
||||
head_async = AsyncMock()
|
||||
get_async = AsyncMock()
|
||||
monkeypatch.setattr(obstore, "head_async", head_async)
|
||||
monkeypatch.setattr(obstore, "get_async", get_async)
|
||||
return head_async, get_async
|
||||
|
||||
|
||||
def _meta(etag: str) -> dict:
|
||||
return {"e_tag": etag, "path": "ignored", "size": 0, "last_modified": None}
|
||||
|
||||
|
||||
def _get_result(data: bytes) -> MagicMock:
|
||||
result = MagicMock()
|
||||
result.bytes_async = AsyncMock(return_value=data)
|
||||
return result
|
||||
|
||||
|
||||
def test_make_s3_store_accepts_lancedb_keys():
|
||||
from haiku.rag.s3 import make_s3_store
|
||||
|
||||
store = make_s3_store(
|
||||
"my-bucket",
|
||||
{
|
||||
"endpoint": "http://seaweed:8333",
|
||||
"region": "us-east-1",
|
||||
"aws_access_key_id": "AKIA",
|
||||
"aws_secret_access_key": "secret",
|
||||
"allow_http": "true",
|
||||
},
|
||||
)
|
||||
assert store is not None # construction must not raise
|
||||
|
||||
|
||||
def test_make_s3_store_no_options_uses_default_chain():
|
||||
from haiku.rag.s3 import make_s3_store
|
||||
|
||||
store = make_s3_store("my-bucket", None)
|
||||
assert store is not None
|
||||
|
||||
|
||||
def test_make_s3_store_missing_obstore_raises_actionable_error(monkeypatch):
|
||||
monkeypatch.setitem(sys.modules, "obstore.store", None)
|
||||
|
||||
from haiku.rag.s3 import make_s3_store
|
||||
|
||||
with pytest.raises(ImportError, match=r"haiku\.rag-slim\[s3\]"):
|
||||
make_s3_store("my-bucket", {})
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.vcr()
|
||||
async def test_create_document_from_s3_new(fake_obstore_io, temp_db_path):
|
||||
head_async, get_async = fake_obstore_io
|
||||
text = b"S3 hosted content"
|
||||
head_async.return_value = _meta('"abc123"')
|
||||
get_async.return_value = _get_result(text)
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||
doc = await client.create_document_from_source("s3://my-bucket/folder/file.txt")
|
||||
|
||||
assert doc.uri == "s3://my-bucket/folder/file.txt"
|
||||
assert doc.metadata["etag"] == "abc123" # quotes stripped
|
||||
assert doc.metadata["md5"] # real content MD5
|
||||
assert doc.metadata["md5"] != "abc123"
|
||||
head_async.assert_awaited_once()
|
||||
get_async.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.vcr()
|
||||
async def test_create_document_from_s3_skips_when_etag_unchanged(
|
||||
fake_obstore_io, temp_db_path
|
||||
):
|
||||
head_async, get_async = fake_obstore_io
|
||||
text = b"S3 hosted content"
|
||||
head_async.return_value = _meta('"abc123"')
|
||||
get_async.return_value = _get_result(text)
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||
first = await client.create_document_from_source("s3://my-bucket/file.txt")
|
||||
|
||||
# Second call with the same ETag must not GET.
|
||||
get_async.reset_mock()
|
||||
get_async.return_value = _get_result(text) # re-arm just in case
|
||||
second = await client.create_document_from_source("s3://my-bucket/file.txt")
|
||||
|
||||
assert second.id == first.id
|
||||
assert head_async.await_count == 2
|
||||
get_async.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.vcr()
|
||||
async def test_create_document_from_s3_etag_changed_md5_same_skips_rechunk(
|
||||
fake_obstore_io, temp_db_path
|
||||
):
|
||||
"""Multipart re-upload of same content: etag changes, MD5 doesn't.
|
||||
|
||||
Expected: GET runs to verify, but no re-chunk; only metadata.etag updates.
|
||||
"""
|
||||
head_async, get_async = fake_obstore_io
|
||||
text = b"S3 hosted content"
|
||||
head_async.return_value = _meta('"abc123"')
|
||||
get_async.return_value = _get_result(text)
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||
first = await client.create_document_from_source("s3://my-bucket/file.txt")
|
||||
original_md5 = first.metadata["md5"]
|
||||
original_updated_at = first.updated_at
|
||||
|
||||
# Same bytes, different ETag (multipart re-upload).
|
||||
head_async.return_value = _meta('"def456-2"')
|
||||
get_async.return_value = _get_result(text)
|
||||
|
||||
second = await client.create_document_from_source("s3://my-bucket/file.txt")
|
||||
|
||||
assert second.id == first.id
|
||||
assert second.metadata["md5"] == original_md5
|
||||
assert second.metadata["etag"] == "def456-2"
|
||||
assert second.updated_at >= original_updated_at
|
||||
assert get_async.await_count == 2 # initial create + etag-changed compare
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.vcr()
|
||||
async def test_create_document_from_s3_etag_changed_md5_changed_rechunks(
|
||||
fake_obstore_io, temp_db_path
|
||||
):
|
||||
head_async, get_async = fake_obstore_io
|
||||
head_async.return_value = _meta('"abc123"')
|
||||
get_async.return_value = _get_result(b"original text")
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||
first = await client.create_document_from_source("s3://my-bucket/file.txt")
|
||||
|
||||
head_async.return_value = _meta('"new999"')
|
||||
get_async.return_value = _get_result(b"different text now")
|
||||
|
||||
second = await client.create_document_from_source("s3://my-bucket/file.txt")
|
||||
|
||||
assert second.id == first.id
|
||||
assert second.metadata["md5"] != first.metadata["md5"]
|
||||
assert second.metadata["etag"] == "new999"
|
||||
assert "different text now" in second.content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_document_from_s3_rejects_invalid_uri(
|
||||
fake_obstore_io, temp_db_path
|
||||
):
|
||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||
with pytest.raises(ValueError, match="Invalid S3 URI"):
|
||||
await client.create_document_from_source("s3://only-bucket-no-key")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_document_from_s3_rejects_unsupported_extension(
|
||||
fake_obstore_io, temp_db_path
|
||||
):
|
||||
head_async, _ = fake_obstore_io
|
||||
head_async.return_value = _meta('"abc"')
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||
with pytest.raises(ValueError, match="Unsupported content type"):
|
||||
await client.create_document_from_source("s3://my-bucket/file.unsupported")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.vcr()
|
||||
async def test_create_document_from_s3_uri_override(fake_obstore_io, temp_db_path):
|
||||
"""`uri=` kwarg overrides the s3:// URL as the stored document identifier."""
|
||||
head_async, get_async = fake_obstore_io
|
||||
head_async.return_value = _meta('"abc"')
|
||||
get_async.return_value = _get_result(b"override target content")
|
||||
|
||||
async with HaikuRAG(temp_db_path, create=True) as client:
|
||||
doc = await client.create_document_from_source(
|
||||
"s3://my-bucket/file.txt", uri="arxiv:2401.00001"
|
||||
)
|
||||
|
||||
assert doc.uri == "arxiv:2401.00001"
|
||||
# The override is the canonical identifier — lookup by it must hit the doc.
|
||||
looked_up = await client.get_document_by_uri("arxiv:2401.00001")
|
||||
assert looked_up is not None
|
||||
assert looked_up.id == doc.id
|
||||
# The s3:// URL is NOT a stored identifier.
|
||||
assert await client.get_document_by_uri("s3://my-bucket/file.txt") is None
|
||||
82
uv.lock
82
uv.lock
|
|
@ -872,14 +872,14 @@ wheels = [
|
|||
|
||||
[[package]]
|
||||
name = "docling"
|
||||
version = "2.92.0"
|
||||
version = "2.93.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "docling-slim", extra = ["standard"] },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/e9/87/343d558da23e0626ba027d2e27da25f1c9022964807bfe80778205550a65/docling-2.92.0.tar.gz", hash = "sha256:e8e393ce1a9520c6a61acc67977a6c2d7c2588d98c7446a935b01f9877af9f93", size = 8727, upload-time = "2026-04-29T07:41:25.438Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/df/e7/41a7ef58935b914332a7924e0a9d817d50214201aeca7507944404229161/docling-2.93.0.tar.gz", hash = "sha256:c8b4455466d9a6314c27cf84debfdf491532298f65364ec3449acaad517bfea6", size = 8726, upload-time = "2026-05-07T11:55:38.745Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b1/ad/e3c776dac9639ba7a9fff4c3fe3687258b6eb651d83c4885fad71b3b6aa4/docling-2.92.0-py3-none-any.whl", hash = "sha256:bd85e102e34bb5a90d3be5c2179855fedbdeb9f45318dfba340bf54f1b3268b2", size = 4829, upload-time = "2026-04-29T07:41:23.992Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/47/6f/3befa6e4cd42f3a9cf284165d18ca1039ab676ab36da327242aa701e6b63/docling-2.93.0-py3-none-any.whl", hash = "sha256:30a2dc2db733d6a24095e624cc133ce67e9edc46e778f982cf719e137bcee200", size = 4827, upload-time = "2026-05-07T11:55:37.457Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -969,7 +969,7 @@ wheels = [
|
|||
|
||||
[[package]]
|
||||
name = "docling-slim"
|
||||
version = "2.92.0"
|
||||
version = "2.93.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "certifi" },
|
||||
|
|
@ -981,9 +981,9 @@ dependencies = [
|
|||
{ name = "requests" },
|
||||
{ name = "tqdm" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/3c/45/ff4d565ccf69694b2ab3b0fcc7cd403243b8650c735bb1a42a48ba82bcaf/docling_slim-2.92.0.tar.gz", hash = "sha256:f54a2159a46cf00f4738888594c5a81372048d8a0d1a15dd279a7390641a04fa", size = 387431, upload-time = "2026-04-29T07:40:05.913Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c4/51/2ee874abcd62b990f0a86abec2e3b87b4cc00731b675ed446fefff6199d9/docling_slim-2.93.0.tar.gz", hash = "sha256:2962f4fc5bdf9dd6d67d6f36f09334f7187039985c0fd4b2d4b1d375e4799157", size = 390036, upload-time = "2026-05-07T11:54:14.562Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/91/12/bdf2579d85e43e5944a146c1c4eab708103784a043fb12d793b524c1cd02/docling_slim-2.92.0-py3-none-any.whl", hash = "sha256:e6b7ea5b955c5c47e9bc36f828268b2b78c32f544842036d43d529746783e380", size = 503467, upload-time = "2026-04-29T07:40:03.714Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f3/3f/f2195f79a62fd6cd10c768c04c758c9d6bd0c804a46175d0f3f10a784fca/docling_slim-2.93.0-py3-none-any.whl", hash = "sha256:98e3db67f7976f051f132e6a6e04f73e7fcd4017877eddf9e849e0969c39fbe7", size = 506046, upload-time = "2026-05-07T11:54:10.884Z" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
|
|
@ -1445,6 +1445,9 @@ dependencies = [
|
|||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
s3 = [
|
||||
{ name = "haiku-rag-slim", extra = ["s3"] },
|
||||
]
|
||||
tui = [
|
||||
{ name = "textual" },
|
||||
]
|
||||
|
|
@ -1455,6 +1458,7 @@ dev = [
|
|||
{ name = "haiku-rag-evals" },
|
||||
{ name = "mkdocs" },
|
||||
{ name = "mkdocs-material" },
|
||||
{ name = "obstore" },
|
||||
{ name = "pre-commit" },
|
||||
{ name = "pydantic-ai-slim", extra = ["anthropic", "bedrock", "google", "groq"] },
|
||||
{ name = "pytest" },
|
||||
|
|
@ -1469,9 +1473,10 @@ dev = [
|
|||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "haiku-rag-slim", extras = ["docling", "voyageai", "mxbai", "cohere", "zeroentropy", "tui"], editable = "haiku_rag_slim" },
|
||||
{ name = "haiku-rag-slim", extras = ["s3"], marker = "extra == 's3'", editable = "haiku_rag_slim" },
|
||||
{ name = "textual", marker = "extra == 'tui'", specifier = ">=1.0.0" },
|
||||
]
|
||||
provides-extras = ["tui"]
|
||||
provides-extras = ["tui", "s3"]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [
|
||||
|
|
@ -1479,6 +1484,7 @@ dev = [
|
|||
{ name = "haiku-rag-evals", editable = "evaluations" },
|
||||
{ name = "mkdocs", specifier = ">=1.6.1" },
|
||||
{ name = "mkdocs-material", specifier = ">=9.7.6" },
|
||||
{ name = "obstore", specifier = ">=0.9,<0.10" },
|
||||
{ name = "pre-commit", specifier = ">=4.5.1" },
|
||||
{ name = "pydantic-ai-slim", extras = ["anthropic"] },
|
||||
{ name = "pydantic-ai-slim", extras = ["bedrock"] },
|
||||
|
|
@ -1571,6 +1577,9 @@ mistral = [
|
|||
mxbai = [
|
||||
{ name = "mxbai-rerank" },
|
||||
]
|
||||
s3 = [
|
||||
{ name = "obstore" },
|
||||
]
|
||||
tui = [
|
||||
{ name = "textual" },
|
||||
{ name = "textual-image" },
|
||||
|
|
@ -1596,6 +1605,7 @@ requires-dist = [
|
|||
{ name = "jsonpatch", specifier = ">=1.33" },
|
||||
{ name = "lancedb", specifier = "==0.30.2" },
|
||||
{ name = "mxbai-rerank", marker = "extra == 'mxbai'", specifier = ">=0.1.6" },
|
||||
{ name = "obstore", marker = "extra == 's3'", specifier = ">=0.9,<0.10" },
|
||||
{ name = "opencv-python-headless", marker = "extra == 'docling'", specifier = ">=4.13.0.92" },
|
||||
{ name = "pathspec", specifier = ">=1.0.4" },
|
||||
{ name = "pydantic", specifier = ">=2.12.5" },
|
||||
|
|
@ -1620,7 +1630,7 @@ requires-dist = [
|
|||
{ name = "zeroentropy", marker = "extra == 'zeroentropy'", specifier = ">=0.1.0a11" },
|
||||
{ name = "zstandard", marker = "python_full_version < '3.14'", specifier = ">=0.23.0" },
|
||||
]
|
||||
provides-extras = ["docling", "voyageai", "mxbai", "cohere", "zeroentropy", "jina", "tui", "anthropic", "groq", "google", "mistral", "bedrock", "vertexai"]
|
||||
provides-extras = ["docling", "s3", "voyageai", "mxbai", "cohere", "zeroentropy", "jina", "tui", "anthropic", "groq", "google", "mistral", "bedrock", "vertexai"]
|
||||
|
||||
[[package]]
|
||||
name = "haiku-skills"
|
||||
|
|
@ -2878,6 +2888,56 @@ wheels = [
|
|||
{ url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "obstore"
|
||||
version = "0.9.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "typing-extensions", marker = "python_full_version < '3.13'" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d3/7a/3a37b0bf0da898478029fcc511a0d2a7252689b1f29e46db7ae74a219c74/obstore-0.9.4.tar.gz", hash = "sha256:e2b93f1372c59da2c7e74122fc6dc4b713d84fd4528b5b500ef7f548425496b5", size = 124167, upload-time = "2026-04-22T19:51:05.261Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/ac/25/4449a0066796b91e282d7604a66387bba399b14752598c748ea9557c4c32/obstore-0.9.4-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:0d17cd04e7f22960050a85f8daa6e274d693e8fb3b97b81eeaa293c6f9e62eb4", size = 4090743, upload-time = "2026-04-22T19:49:26.461Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/93/91/639fe5f5644593b9f4bea66f8f29c7bfd4de3b3381fb74b4f7df678f505f/obstore-0.9.4-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:d4beec92710fb8826fb357baf28fb79a91ee07dcdfe73777207aa762164aaa35", size = 3876313, upload-time = "2026-04-22T19:49:28.107Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ce/71/d6675f845ebe1e3927f2dce6a2a4d5a393359274762ee00c5e6855d5f468/obstore-0.9.4-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d523c8c365ab60afb8d232614a00a92bea439a9f5c55b92486c23a47af038a1e", size = 4029950, upload-time = "2026-04-22T19:49:30.279Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0e/3a/5915a173f5c6a95f9ec186a7e29b0ce6a23bd9b04c2b0b29a351dbe2baf6/obstore-0.9.4-cp311-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ee0483619088337ee365cb344fceee337e2670ec4de2a1da92ac7f6b2220f18e", size = 4129455, upload-time = "2026-04-22T19:49:31.934Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b5/a9/63c31d2d436c06c4d39ed5cb154fe54202b303854532ec09537c4ce0755b/obstore-0.9.4-cp311-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:83da348bf0a7dd84e5839c0cd54d79dcd08e0729c394e566f73a605b93b9e998", size = 4416727, upload-time = "2026-04-22T19:49:34.016Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7f/fa/23c5c6db02be0e13abcbe01c1ca94c5f7876e8c58e74cb9ac2b57b068866/obstore-0.9.4-cp311-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f282a17200bcc37b8d7a1d02a146ed41812eb6e76fd0a4c9a154f02da1b8031f", size = 4311520, upload-time = "2026-04-22T19:49:35.905Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/86/f0/49f6b02dab9c05e3fd79d6129e4d9e7e9874d6e5e05369ca3b3b80a48aaa/obstore-0.9.4-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d29dcfceaa0a205ded2263d29a2a3aa206819d549e0325c1f2106f79e2658584", size = 4220536, upload-time = "2026-04-22T19:49:38.343Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/50/ab/d0bfd6d68422e7d8f2204d91736c7e62767e0576ad749da442a71e7773b2/obstore-0.9.4-cp311-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:caecb912723ab8e9da8da26def249d66da4318959df2bafc0a55af64f3255902", size = 4105099, upload-time = "2026-04-22T19:49:40.384Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/66/3b/f595d0ee354f9daa69438991f8818602f34bc59498c8468456a02d45fb27/obstore-0.9.4-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0c1c06fec8837595a2829b5f7536d0d01e940ce10b07ad2a8594fec1cfd0b7d5", size = 4294206, upload-time = "2026-04-22T19:49:42.016Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/60/54/3c5af2d59258aaa9e5bef05320658ea6e9b1f3897a3a977bf7f54a0b6ec1/obstore-0.9.4-cp311-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:c132795a789ec5ade31bf4d5b55ed321fb41d9749e9145520bf19063e1da5f7b", size = 4265047, upload-time = "2026-04-22T19:49:43.983Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fb/af/a8ba1feb81b9833b253147839da40405ec6bfa51feb3abfe909c800208a5/obstore-0.9.4-cp311-abi3-musllinux_1_2_i686.whl", hash = "sha256:c6e342360a5d0ae71486bc5f8311778aa144ec1a905c23593f8ef57b5bceae24", size = 4255361, upload-time = "2026-04-22T19:49:45.864Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/15/f7/3ccc0288111e057f8ba3d99bee14f95d9e9bb00acaf6e9700e0eb4cd82c3/obstore-0.9.4-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:aeb6f7e7e862550f5020a10692ef6f02d5ba4912dba08942eb59bb7d73f93fe0", size = 4439378, upload-time = "2026-04-22T19:49:47.581Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c4/b1/3ac8b5772743c60064f3c7e02d27f346dbb58feaa99a49ee09798d1cfb00/obstore-0.9.4-cp311-abi3-win_amd64.whl", hash = "sha256:a58ef942292841f99d69ac11d19d05544c835447c8c09dacbfb7409c6374c4a1", size = 4191594, upload-time = "2026-04-22T19:49:49.308Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9d/81/8f6b6509f8df603261cdb5ddb521c49891457775669c6ad857812bf4a7c1/obstore-0.9.4-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:fff17f59390ed307afcd1fb18c56076c1f911dd9f5c2636b7d7133c4d07f8c3f", size = 4071300, upload-time = "2026-04-22T19:49:51.386Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ab/fe/0c74ddf3ab9b24ef356925bfb613bc7846f869220361a784b63f754d8563/obstore-0.9.4-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4527c4c7889f1bd1f1952017d74774870e14e199d6b50b9e72f291f9498d898c", size = 3870593, upload-time = "2026-04-22T19:49:53.481Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/73/fa/260ec94f9a7b4f4c8afbdd016710bed0736615488d3ac0c5620f9179bfcd/obstore-0.9.4-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a57c2016e3e569de35050f95c679ffe61813c4e3cb6d6028c4c3f57231021eb4", size = 4023990, upload-time = "2026-04-22T19:49:55.644Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8d/84/5b8e2b9607fb93c96a39a4cfa6d37bd3049ebf7265d0e9f8afa938bf32fe/obstore-0.9.4-cp313-cp313t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bd5327cee4fb3578b51beb1c92915cc3a05ffe794be40f50bd68d27e97d78c5c", size = 4119971, upload-time = "2026-04-22T19:49:57.745Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3c/2b/e6c093acb7e62009d5b1678d82839903287c29d4a6e1dfbea8fbf41313d5/obstore-0.9.4-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:12b1e6105eafe02d8973dbeb2d274eeac2271c67f1126ffa16f18ddea8dd5443", size = 4407147, upload-time = "2026-04-22T19:49:59.928Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ce/3d/5c93a9adee8f045b89d5f21b337f53667499db770bda129f805723ab14e4/obstore-0.9.4-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0b0d378248fda4e36652808d73eaaeb7e67154427e6c724248c9b0b9b03e70a6", size = 4312215, upload-time = "2026-04-22T19:50:01.534Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9a/de/507f60b4e6a8c0cad9f93a51a7b28132c9db49e20aadbcd542fa2abc57c4/obstore-0.9.4-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a78fb77c346abd2bcdfa071d7166be2bdc38c28573ae5a230746df6158a5593e", size = 4216936, upload-time = "2026-04-22T19:50:03.244Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/ff/612bd5f8258349bfe9e8c349d184b5ea3333038d4cce0d003eefafb2160c/obstore-0.9.4-cp313-cp313t-manylinux_2_24_aarch64.whl", hash = "sha256:f4e5a6dfe6877fb599868d560d6fcf4d7416cadbdf3bd947254b53830c2f11c0", size = 4105091, upload-time = "2026-04-22T19:50:05.038Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e3/73/b083b99e7bc0b529bee7b4437cafd7cc7d9f59c10995a48b6c26447fdf7f/obstore-0.9.4-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:f8114a2b84268c991232d89b105d9239299b6afb56e4941a61c09f3a89033022", size = 4292570, upload-time = "2026-04-22T19:50:06.823Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8b/cd/3c4555f98db9a49432bc0afa68bfc33dd47bdfa3699c915b4b0e887577e3/obstore-0.9.4-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:9d7b959f5f74532a142fb449c0bef5814dfe3fa5c43c31ac4284a15221a75aaf", size = 4261946, upload-time = "2026-04-22T19:50:08.789Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/96/f8/bdc66df3d0dfdcfb3931a585a7fb3b74336619baf6d3540b1425b424232b/obstore-0.9.4-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:a8e9101fc2659dd938e7ae06512075bc0a8f02ab28d2ee438d6fca8b4f3bdfba", size = 4245595, upload-time = "2026-04-22T19:50:10.765Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d7/22/1aa58ea676293e5b888391c8433ff6ab8f66622aae30427287f9daac6d46/obstore-0.9.4-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:538384255545b5c575497fcab26389c8f01707402b6ddcdd73b769b66311635d", size = 4436599, upload-time = "2026-04-22T19:50:12.585Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/1d/9e/b52f2c97be27952d488cf1980af0c635f9947003e5744e3e1dc6252f0040/obstore-0.9.4-cp313-cp313t-win_amd64.whl", hash = "sha256:eef1c772657bb1293adad0d671ca1ff1e1dcae84ec4dfbf1a34e47c2a1f134ac", size = 4180463, upload-time = "2026-04-22T19:50:14.288Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/19/76/c53583f95c6811057abd3116756dca46785318d564a0e99c207cbb2d8938/obstore-0.9.4-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:e009e7437770c85beae4c32cb79f662f0a9922676ef127e943d107a5c082d38d", size = 4071302, upload-time = "2026-04-22T19:50:15.967Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2f/23/ac3b9c05a09b3d5f178ed6f288c5d6913df8f7386059590194e0fee65d15/obstore-0.9.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ac5f3ad314bd4592fe484b79c229518be7bb5f6218bed33c20742026d5caf860", size = 3870813, upload-time = "2026-04-22T19:50:17.62Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f7/ec/c3458e0f24d2d1a4f185f541905b07e51c91b3fec589b1600c77d511e585/obstore-0.9.4-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:db79d5ebc4177360565ffcec4abd49930cf052cdbeb94e3a3ece2e2d08f087d0", size = 4024237, upload-time = "2026-04-22T19:50:19.81Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a7/eb/6cf468a200e491fdc6c04075e2fbbac1707bbecd243f0f56ae1e75d052ed/obstore-0.9.4-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:05b565d89c3115fb74385852dd628e12f6645a1bba97523dceae016b538a3f33", size = 4119635, upload-time = "2026-04-22T19:50:21.605Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/81/fb/b44d002767fa5af95ab4ca8e16c3a9057fc11f13de03f498b99adf0c4e50/obstore-0.9.4-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7dfc4fc98403d8fbb316eb04257c8122b6f1dda37e80869491fdacf60a815e4c", size = 4406906, upload-time = "2026-04-22T19:50:23.654Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4b/18/9a75ad5082cd581c4a55f0e62bedf4b030a8b53824976fc1f030eff225b3/obstore-0.9.4-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c69af620fd3d06a8cfb62d25faf1adb6ccc97cc572f47ee04dddcde5a5e5444e", size = 4311826, upload-time = "2026-04-22T19:50:25.458Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8d/03/b0f945b31f40364a7ed4dbc5677abc66331fcf478732f4d643e17e56bb13/obstore-0.9.4-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fa78c0230e0b9d49b25ed18980e1751331ddfe05782d6ce97579a9ccda8229ea", size = 4217086, upload-time = "2026-04-22T19:50:27.266Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/1a/26/bdd85264c806802086f21d73cc7c95a5baca5feeeac4bce8acb97142163f/obstore-0.9.4-cp314-cp314t-manylinux_2_24_aarch64.whl", hash = "sha256:c828719f0bb310a9cf0e0f08cb62a0b8cc550138617cb03ac897900aec9d3d47", size = 4105560, upload-time = "2026-04-22T19:50:29.324Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6a/36/4a4a6a398e5f145edd1886388ebe5e6f6bbaf74950a5dea1a6ceae63e6b5/obstore-0.9.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:49a0455519f284b6bc2e0694298114926aff1d1f3d5d344e9163e03b446826cc", size = 4292582, upload-time = "2026-04-22T19:50:31.028Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/39/4c/9caa197cd2eba726e9a5285db34027049b9527a23e1a7e08479678ad6a4a/obstore-0.9.4-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cf437309fc0fe852591ae50405300490229f876ea06574651fd753ca3fd23f25", size = 4261613, upload-time = "2026-04-22T19:50:32.868Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b8/94/a3fbe6fb3ee1c57fd4943ddbb21848eea3925b77e0789614c857d86b795e/obstore-0.9.4-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:d83dbd20b6a5d42e35794ef64046de39040854829ec4f1eb2f6dfb54df48cc3d", size = 4245638, upload-time = "2026-04-22T19:50:35.009Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/56/a7/d18e168f318327d63512dfa7cf3b5e89ed9bfba6d6a8917ad7d4700b8657/obstore-0.9.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5a0c337f37f30a2d66555d69bf3abd840457a279c57ede93bd02e014721ed364", size = 4437226, upload-time = "2026-04-22T19:50:36.635Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f4/ce/66aadd155db1e273c6ec2236c0fb904666d10c2e3b791b40624c272e586c/obstore-0.9.4-cp314-cp314t-win_amd64.whl", hash = "sha256:24e37a1c713c95a964e119f8ef879415a495432162e74e80ed29d645aeeca114", size = 4180746, upload-time = "2026-04-22T19:50:38.396Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "omegaconf"
|
||||
version = "2.3.0"
|
||||
|
|
@ -3835,16 +3895,16 @@ wheels = [
|
|||
|
||||
[[package]]
|
||||
name = "pydantic-settings"
|
||||
version = "2.14.0"
|
||||
version = "2.14.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pydantic" },
|
||||
{ name = "python-dotenv" },
|
||||
{ name = "typing-inspection" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/42/98/c8345dccdc31de4228c039a98f6467a941e39558da41c1744fbe29fa5666/pydantic_settings-2.14.0.tar.gz", hash = "sha256:24285fd4b0e0c06507dd9fdfd331ee23794305352aaec8fc4eb92d4047aeb67d", size = 235709, upload-time = "2026-04-20T13:37:40.293Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/07/60/1d1e59c9c90d54591469ada7d268251f71c24bdb765f1a8a832cee8c6653/pydantic_settings-2.14.1.tar.gz", hash = "sha256:e874d3bec7e787b0c9958277956ed9b4dd5de6a80e162188fdaff7c5e26fd5fa", size = 235551, upload-time = "2026-05-08T13:40:06.542Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/01/dd/bebff3040138f00ae8a102d426b27349b9a49acc310fcae7f92112d867e3/pydantic_settings-2.14.0-py3-none-any.whl", hash = "sha256:fc8d5d692eb7092e43c8647c1c35a3ecd00e040fcf02ed86f4cb5458ca62182e", size = 60940, upload-time = "2026-04-20T13:37:38.586Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ae/8d/f1af3832f5e6eb13ba94ee809e72b8ecb5eef226d27ee0bef7d963d943c7/pydantic_settings-2.14.1-py3-none-any.whl", hash = "sha256:6e3c7edfd8277687cdc598f56e5cff0e9bfff0910a3749deaa8d4401c3a2b9de", size = 60964, upload-time = "2026-05-08T13:40:04.958Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
|
|||
Loading…
Reference in a new issue