4c30e5eb8f
Introduces the data layer for multi-store ingestion. Documents can now live in multiple stores, each with its own Ingested/Stale/Failed state. The doc-level lifecycle (#202) becomes the aggregate over all per-store links, computed by a pure domain function. Domain - Store entity (name, slug, kind, embedder, config, is_default) - DocumentStoreLink entity with mark_ingested / mark_stale / mark_failed helpers - StoreKind and DocumentStoreLinkState enums - aggregate_lifecycle(): pure function — Failed > Stale > Ingested > fallback (the doc's pre-link Uploaded/Parsed/Chunked state) Persistence - New tables 'stores' and 'document_store_links' with the right indexes (doc_id, store_id, state) and a UNIQUE (doc, store) on the link - Default 'opensearch' store seeded idempotently in init_db, embedder pulled from DEFAULT_EMBEDDER (fallback bge-m3) - SqliteStoreRepository (find_by_slug, find_by_id, get_default, …) - SqliteDocumentStoreLinkRepository with ON CONFLICT … DO UPDATE upsert Ports - StoreRepository and DocumentStoreLinkRepository protocols added Tests - 14 new tests: seed idempotency, insert/find round-trips, UNIQUE constraint, cascade delete with the document, every link state round-trips, aggregation rule with all branches Refs #203
58 lines
2 KiB
Python
58 lines
2 KiB
Python
"""Aggregate the per-(document, store) link states into a single
|
|
document-level lifecycle state.
|
|
|
|
The doc lifecycle column is the materialized result of this rule. It is
|
|
recomputed any time a link write happens. Read paths use the stored
|
|
column directly (cheap) — they do not call this rule on every GET.
|
|
|
|
The rule prefers "more concerning" states first:
|
|
|
|
any link FAILED -> Document FAILED
|
|
any link STALE -> Document STALE
|
|
any link INGESTED -> Document INGESTED
|
|
no links -> keep the document's current pre-link state
|
|
(Uploaded / Parsed / Chunked) — the caller is
|
|
responsible for not overwriting that.
|
|
|
|
If you find yourself wanting a fourth case, you probably want a new
|
|
link state, not a new aggregation branch.
|
|
|
|
This module is pure: no I/O, no datetime — just data in / data out.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import TYPE_CHECKING
|
|
|
|
from domain.value_objects import DocumentLifecycleState, DocumentStoreLinkState
|
|
|
|
if TYPE_CHECKING:
|
|
from collections.abc import Iterable
|
|
|
|
from domain.models import DocumentStoreLink
|
|
|
|
|
|
def aggregate_lifecycle(
|
|
links: Iterable[DocumentStoreLink],
|
|
*,
|
|
fallback: DocumentLifecycleState,
|
|
) -> DocumentLifecycleState:
|
|
"""Compute the aggregate document lifecycle state.
|
|
|
|
Args:
|
|
links: every `DocumentStoreLink` for the document. May be empty.
|
|
fallback: the lifecycle state to return when there are no links —
|
|
typically the document's current pre-link state (`Uploaded`,
|
|
`Parsed`, or `Chunked`).
|
|
|
|
Returns:
|
|
The aggregate `DocumentLifecycleState`.
|
|
"""
|
|
states = {link.state for link in links}
|
|
if DocumentStoreLinkState.FAILED in states:
|
|
return DocumentLifecycleState.FAILED
|
|
if DocumentStoreLinkState.STALE in states:
|
|
return DocumentLifecycleState.STALE
|
|
if DocumentStoreLinkState.INGESTED in states:
|
|
return DocumentLifecycleState.INGESTED
|
|
return fallback
|