feat(neo4j): Day 1 — compose service, driver, schema bootstrap
Add Neo4j as an optional graph-native storage layer (ingestion profile). Introduces infra/neo4j with a singleton async driver wrapper and an idempotent bootstrap of constraints + indexes, wired into the FastAPI lifespan. Integration tests skip when no live Neo4j is reachable. Refs #186
This commit is contained in:
parent
358e575f0f
commit
dfbca40730
15 changed files with 752 additions and 1 deletions
|
|
@ -48,3 +48,8 @@
|
|||
|
||||
# Embedding vector dimension (default: 384 for Granite Embedding 30M / all-MiniLM-L6-v2)
|
||||
# EMBEDDING_DIMENSION=384
|
||||
|
||||
# Neo4j — graph-native document structure (used by docker-compose.dev.yml)
|
||||
# NEO4J_URI=bolt://neo4j:7687
|
||||
# NEO4J_USER=neo4j
|
||||
# NEO4J_PASSWORD=changeme
|
||||
|
|
|
|||
|
|
@ -9,6 +9,26 @@
|
|||
# =============================================================================
|
||||
|
||||
services:
|
||||
# --- Neo4j (graph-native document structure) ---
|
||||
neo4j:
|
||||
image: neo4j:5.15-community
|
||||
environment:
|
||||
NEO4J_AUTH: ${NEO4J_USER:-neo4j}/${NEO4J_PASSWORD:-changeme}
|
||||
NEO4J_server_memory_heap_initial__size: 512m
|
||||
NEO4J_server_memory_heap_max__size: 1g
|
||||
ports:
|
||||
- "7474:7474"
|
||||
- "7687:7687"
|
||||
volumes:
|
||||
- neo4j_data:/data
|
||||
- neo4j_logs:/logs
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "cypher-shell -u $${NEO4J_USER:-neo4j} -p $${NEO4J_PASSWORD:-changeme} 'RETURN 1' || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
|
||||
# --- OpenSearch (single-node, security disabled for local dev) ---
|
||||
opensearch:
|
||||
image: opensearchproject/opensearch:2
|
||||
|
|
@ -77,12 +97,17 @@ services:
|
|||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
|
||||
OPENSEARCH_URL: http://opensearch:9200
|
||||
EMBEDDING_URL: http://embedding:8001
|
||||
NEO4J_URI: bolt://neo4j:7687
|
||||
NEO4J_USER: ${NEO4J_USER:-neo4j}
|
||||
NEO4J_PASSWORD: ${NEO4J_PASSWORD:-changeme}
|
||||
command: ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||
depends_on:
|
||||
opensearch:
|
||||
condition: service_healthy
|
||||
embedding:
|
||||
condition: service_healthy
|
||||
neo4j:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
|
|
@ -105,6 +130,8 @@ services:
|
|||
|
||||
volumes:
|
||||
opensearch_data:
|
||||
neo4j_data:
|
||||
neo4j_logs:
|
||||
uploads_data:
|
||||
db_data:
|
||||
frontend_node_modules:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,22 @@
|
|||
services:
|
||||
# --- Neo4j (graph-native document structure) ---
|
||||
neo4j:
|
||||
profiles: ["ingestion"]
|
||||
image: neo4j:5.15-community
|
||||
environment:
|
||||
NEO4J_AUTH: ${NEO4J_USER:-neo4j}/${NEO4J_PASSWORD:-changeme}
|
||||
NEO4J_server_memory_heap_initial__size: 512m
|
||||
NEO4J_server_memory_heap_max__size: 1g
|
||||
volumes:
|
||||
- neo4j_data:/data
|
||||
- neo4j_logs:/logs
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "cypher-shell -u $${NEO4J_USER:-neo4j} -p $${NEO4J_PASSWORD:-changeme} 'RETURN 1' || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
|
||||
# --- OpenSearch (single-node, security disabled) ---
|
||||
opensearch:
|
||||
profiles: ["ingestion"]
|
||||
|
|
@ -53,6 +71,9 @@ services:
|
|||
BATCH_PAGE_SIZE: ${BATCH_PAGE_SIZE:-10}
|
||||
OPENSEARCH_URL: ${OPENSEARCH_URL:-}
|
||||
EMBEDDING_URL: ${EMBEDDING_URL:-}
|
||||
NEO4J_URI: ${NEO4J_URI:-}
|
||||
NEO4J_USER: ${NEO4J_USER:-neo4j}
|
||||
NEO4J_PASSWORD: ${NEO4J_PASSWORD:-changeme}
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
|
|
@ -69,5 +90,7 @@ services:
|
|||
|
||||
volumes:
|
||||
opensearch_data:
|
||||
neo4j_data:
|
||||
neo4j_logs:
|
||||
uploads_data:
|
||||
db_data:
|
||||
|
|
|
|||
435
docs/design/neo4j-integration.md
Normal file
435
docs/design/neo4j-integration.md
Normal file
|
|
@ -0,0 +1,435 @@
|
|||
# Neo4j integration — Docling-Studio v0.5.0
|
||||
|
||||
Design doc for Neo4j integration targeting release 0.5.0.
|
||||
Target: Hackernoon hackathon demo (Neo4j partner).
|
||||
|
||||
---
|
||||
|
||||
## 1. Context and goals
|
||||
|
||||
### Already in Docling-Studio
|
||||
- Ingestion pipeline: Docling parser → chunking (HybridChunker) → embedding → OpenSearch (vector index)
|
||||
- Vue 3 + FastAPI UI
|
||||
- Debug view to inspect/edit chunks before retrieval
|
||||
- Docker compose with existing services
|
||||
|
||||
### What we add in v0.5.0
|
||||
- Neo4j as **graph-native storage** of the document structure
|
||||
- A new ingestion layer that stores the DoclingDocument tree faithfully as a graph
|
||||
- Minimal UI to visualize the graph (demo value to the judges)
|
||||
- Compose pipeline with Neo4j
|
||||
|
||||
### Why graph-native (hackathon positioning)
|
||||
> Most document AI tools store parsed content as flat chunks in a vector DB.
|
||||
> Docling-Studio v0.5 introduces a graph-native storage layer on top of Neo4j,
|
||||
> preserving the full hierarchical structure of documents as first-class citizens.
|
||||
> This unlocks hybrid retrieval, agentic navigation, and structural debugging —
|
||||
> impossible with chunk-only stores.
|
||||
|
||||
### Out of scope for v0.5.0 (roadmap mention only)
|
||||
- EnrichmentWriter (entities / summaries / keywords via docling-agent) — v0.6.0
|
||||
- Agent reasoning trace viewer — v0.6.0
|
||||
- RAG hybrid (graph traversal + vector) — v0.7.0
|
||||
- Document versioning — v0.7.0+
|
||||
|
||||
---
|
||||
|
||||
## 2. Architectural principles
|
||||
|
||||
### Port & adapter, with nuance
|
||||
|
||||
**Write side**: one `Writer` port, **composable stages** (not alternative adapters).
|
||||
Pipelines A and B are additive, not exclusive.
|
||||
|
||||
```
|
||||
CORE (always) Pipeline A (RAG) Pipeline B (agent-ready, v0.6+)
|
||||
┌─────────────┐ ┌────────────────┐ ┌───────────────────┐
|
||||
│ TreeWriter │ ─────▶ │ ChunkWriter │ │ EnrichmentWriter │
|
||||
│ │ │ (existing │ │ (via docling- │
|
||||
│ │ │ OpenSearch + │ │ agent, v0.6+) │
|
||||
│ │ │ adds chunks │ │ │
|
||||
│ │ │ to Neo4j) │ │ │
|
||||
└─────────────┘ └────────────────┘ └───────────────────┘
|
||||
```
|
||||
|
||||
```python
|
||||
# docling_studio/ingestion/pipeline.py
|
||||
class Writer(Protocol):
|
||||
def write(self, doc: DoclingDocument, ctx: IngestionContext) -> None: ...
|
||||
|
||||
# Explicit composition per use case
|
||||
def build_pipeline(config: PipelineConfig) -> list[Writer]:
|
||||
writers = [TreeWriter(neo4j_driver)]
|
||||
if config.rag_enabled:
|
||||
writers.append(ChunkWriter(neo4j_driver, chunker, embedder, opensearch))
|
||||
if config.enrichment_enabled: # v0.6.0+
|
||||
writers.append(EnrichmentWriter(neo4j_driver, docling_agent))
|
||||
return writers
|
||||
```
|
||||
|
||||
**Read side**: two distinct ports (same Neo4j backend, different queries).
|
||||
|
||||
```python
|
||||
class RAGRetrievalPort(Protocol):
|
||||
def search(self, query: str, k: int) -> list[Chunk]: ...
|
||||
def similar(self, chunk_id: str, k: int) -> list[Chunk]: ...
|
||||
|
||||
class TreeNavigationPort(Protocol): # v0.6.0+
|
||||
def get_outline(self, doc_id: str) -> Tree: ...
|
||||
def read_node(self, ref: str) -> Element: ...
|
||||
def list_children(self, ref: str) -> list[Element]: ...
|
||||
def walk(self, ref: str, depth: int) -> SubTree: ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Neo4j schema
|
||||
|
||||
### Constraints & indexes (created at boot)
|
||||
|
||||
```cypher
|
||||
// Uniqueness
|
||||
CREATE CONSTRAINT document_id IF NOT EXISTS
|
||||
FOR (d:Document) REQUIRE d.id IS UNIQUE;
|
||||
|
||||
CREATE CONSTRAINT element_composite IF NOT EXISTS
|
||||
FOR (e:Element) REQUIRE (e.doc_id, e.self_ref) IS UNIQUE;
|
||||
|
||||
CREATE CONSTRAINT page_composite IF NOT EXISTS
|
||||
FOR (p:Page) REQUIRE (p.doc_id, p.page_no) IS UNIQUE;
|
||||
|
||||
CREATE CONSTRAINT chunk_id IF NOT EXISTS
|
||||
FOR (c:Chunk) REQUIRE c.id IS UNIQUE;
|
||||
|
||||
// Full-text index (element text search)
|
||||
CREATE FULLTEXT INDEX element_text IF NOT EXISTS
|
||||
FOR (e:Element) ON EACH [e.text];
|
||||
|
||||
// Simple indexes for per-doc queries
|
||||
CREATE INDEX element_doc IF NOT EXISTS FOR (e:Element) ON (e.doc_id);
|
||||
CREATE INDEX chunk_doc IF NOT EXISTS FOR (c:Chunk) ON (c.doc_id);
|
||||
```
|
||||
|
||||
### Data model
|
||||
|
||||
```cypher
|
||||
// Root document
|
||||
(:Document {
|
||||
id: string, // UUID or PDF hash
|
||||
title: string,
|
||||
source_uri: string, // path or S3
|
||||
ingested_at: datetime,
|
||||
docling_version: string,
|
||||
stages_applied: list<string>, // ["tree", "chunks"] etc.
|
||||
last_tree_write: datetime,
|
||||
last_chunk_write: datetime,
|
||||
tenant_id: string // simple multi-tenancy
|
||||
})
|
||||
|
||||
// All tree elements (shared :Element label + specific label)
|
||||
(:Element:SectionHeader {doc_id, self_ref, text, level, prov_page, prov_bbox})
|
||||
(:Element:Paragraph {doc_id, self_ref, text, prov_page, prov_bbox})
|
||||
(:Element:Table {doc_id, self_ref, caption, cells_json, prov_page, prov_bbox})
|
||||
(:Element:Figure {doc_id, self_ref, caption, image_uri, prov_page, prov_bbox})
|
||||
(:Element:ListItem {doc_id, self_ref, text, marker, prov_page, prov_bbox})
|
||||
(:Element:Formula {doc_id, self_ref, latex, text, prov_page, prov_bbox})
|
||||
|
||||
// Page for layout provenance
|
||||
(:Page {doc_id, page_no, width, height})
|
||||
|
||||
// Chunks (Pipeline A)
|
||||
(:Chunk {
|
||||
id, doc_id,
|
||||
text,
|
||||
chunk_index,
|
||||
embedding_ref, // id in OpenSearch (no inline duplication)
|
||||
token_count
|
||||
})
|
||||
```
|
||||
|
||||
### Relations
|
||||
|
||||
```cypher
|
||||
// Hierarchical structure
|
||||
(:Document)-[:HAS_ROOT]->(:Element)
|
||||
(:Element)-[:PARENT_OF {order: int}]->(:Element) // order preserves sequence
|
||||
(:Element)-[:NEXT]->(:Element) // DFS pre-order reading
|
||||
|
||||
// Layout
|
||||
(:Element)-[:ON_PAGE]->(:Page)
|
||||
|
||||
// Pipeline A (chunking)
|
||||
(:Document)-[:HAS_CHUNK]->(:Chunk)
|
||||
(:Chunk)-[:DERIVED_FROM]->(:Element) // back-reference; a chunk can span multiple elements
|
||||
```
|
||||
|
||||
### Decisions
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|-------|---------------|
|
||||
| Element composite key | `(doc_id, self_ref)` | self_ref not unique across docs |
|
||||
| Multi-tenancy | `tenant_id` property on Document | Simple, filterable, migrable to multi-db later |
|
||||
| Table cells | `cells_json` property | v0.5 KISS. May model `(Table)-[:HAS_CELL]->(Cell)` in v0.6+ |
|
||||
| Reading order | `[:NEXT]` chain + `{order}` on `PARENT_OF` | Both views useful |
|
||||
| Versioning | None (replace strategy on re-upload) | v0.5 KISS |
|
||||
| APOC | Not required | Pure Cypher is sufficient for v0.5 |
|
||||
|
||||
### Re-ingestion strategy
|
||||
|
||||
```cypher
|
||||
// Before ingesting, wipe existing
|
||||
MATCH (d:Document {id: $doc_id})
|
||||
OPTIONAL MATCH (d)-[:HAS_ROOT|HAS_CHUNK]->()
|
||||
DETACH DELETE d
|
||||
// Then re-walk cleanly
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation plan (3 days)
|
||||
|
||||
### Day 1 — Infra + schema
|
||||
|
||||
- [ ] Add `neo4j` service to `docker-compose.yml` (`neo4j:5.15-community`, persistent volume, healthcheck)
|
||||
- [ ] Add env vars (`NEO4J_URI`, `NEO4J_USER`, `NEO4J_PASSWORD`) to `.env.example`
|
||||
- [ ] Create module `docling_studio/storage/neo4j/`:
|
||||
- `driver.py` — neo4j-python driver wrapper (connection pool, context manager)
|
||||
- `schema.py` — idempotent `bootstrap_schema()` (CREATE CONSTRAINT / INDEX at startup)
|
||||
- `__init__.py` with exports
|
||||
- [ ] Hook `bootstrap_schema()` in FastAPI startup
|
||||
- [ ] Basic integration tests:
|
||||
- Driver connection
|
||||
- Schema bootstrap (idempotence verified)
|
||||
- Simple round-trip: write Document, read Document, delete Document
|
||||
|
||||
**Deliverable:** docker compose boots with healthy Neo4j, schema in place at init.
|
||||
|
||||
### Day 2 — TreeWriter (write + read)
|
||||
|
||||
- [ ] `storage/neo4j/tree_writer.py` — `DoclingDocument → Neo4j` walker
|
||||
- `write_document(doc, tenant_id, driver)` in a transaction
|
||||
- DFS pre-order, batched `MERGE` for perf
|
||||
- Pages first, then Elements, then `PARENT_OF` / `NEXT` / `ON_PAGE` relations
|
||||
- Dynamic labels based on `node.label` (`SectionHeader`, `Paragraph`, …)
|
||||
- [ ] `storage/neo4j/tree_reader.py` — inverse walker `Neo4j → DoclingDocument`
|
||||
- `read_document(doc_id, driver) -> DoclingDocument`
|
||||
- Loads all Elements + Pages, rebuilds the Pydantic structure
|
||||
- Prerequisite for v0.6 (feeding docling-agent from Neo4j)
|
||||
- [ ] Integrate into existing ingestion pipeline:
|
||||
- Add TreeWriter as first stage of `IngestionPipeline`
|
||||
- `neo4j_enabled: bool` config toggle
|
||||
- [ ] Round-trip tests:
|
||||
- 3–4 varied PDFs (academic, invoice, report)
|
||||
- Assertion: `doc_original == read_document(write_document(doc_original))`
|
||||
- Beware dates, bbox floats (tolerance)
|
||||
|
||||
**Deliverable:** A PDF uploaded to Docling-Studio is fully present in Neo4j and rebuildable.
|
||||
|
||||
### Day 3 — UI + ChunkWriter + packaging
|
||||
|
||||
- [ ] `storage/neo4j/chunk_writer.py`:
|
||||
- After existing chunking, push each Chunk to Neo4j
|
||||
- Create `(:Chunk)-[:DERIVED_FROM]->(:Element)` via source element `self_ref`
|
||||
- Do NOT duplicate embeddings (stay in OpenSearch, keep `embedding_ref`)
|
||||
- [ ] Frontend: new "Graph view" tab in debug panel
|
||||
- Vue component with `cytoscape` (lighter, better layout API)
|
||||
- FastAPI endpoint `/api/documents/{doc_id}/graph` returns nodes + edges around a scope (whole doc or subtree)
|
||||
- View: vertical tree, colors per node type, click-to-zoom, hover details
|
||||
- [ ] Per-document "Graph-ready" / "RAG-ready" badge in list
|
||||
- [ ] README update:
|
||||
- "Graph storage with Neo4j" section
|
||||
- Schema diagram (Mermaid or image)
|
||||
- 2–3 Cypher examples like "find all paragraphs under section X that mention Y"
|
||||
- Neo4j badge in features list
|
||||
- [ ] (bonus if time) "Query explorer" dev tab for live demo: Cypher editor + results
|
||||
|
||||
**Deliverable:** release 0.5.0 with Neo4j visible, functional, documented.
|
||||
|
||||
---
|
||||
|
||||
## 5. Proposed code structure
|
||||
|
||||
```
|
||||
docling_studio/
|
||||
├── storage/
|
||||
│ ├── neo4j/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── driver.py # connection management
|
||||
│ │ ├── schema.py # bootstrap_schema()
|
||||
│ │ ├── tree_writer.py # DoclingDocument -> Neo4j
|
||||
│ │ ├── tree_reader.py # Neo4j -> DoclingDocument
|
||||
│ │ ├── chunk_writer.py # Chunks -> Neo4j
|
||||
│ │ └── queries.py # shared Cypher queries
|
||||
│ ├── opensearch/ # (existing)
|
||||
│ └── ports.py # Writer, RAGRetrievalPort protocols
|
||||
├── ingestion/
|
||||
│ └── pipeline.py # IngestionPipeline composing Writers
|
||||
├── api/
|
||||
│ └── graph.py # /api/documents/{id}/graph
|
||||
└── frontend/
|
||||
└── components/
|
||||
└── GraphView.vue # cytoscape + graph API fetch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Docker compose (added excerpt)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
neo4j:
|
||||
image: neo4j:5.15-community
|
||||
environment:
|
||||
NEO4J_AUTH: ${NEO4J_USER:-neo4j}/${NEO4J_PASSWORD:-changeme}
|
||||
NEO4J_PLUGINS: '["apoc"]'
|
||||
NEO4J_server_memory_heap_initial__size: 512m
|
||||
NEO4J_server_memory_heap_max__size: 1g
|
||||
ports:
|
||||
- "7474:7474" # Browser UI (demo)
|
||||
- "7687:7687" # Bolt protocol
|
||||
volumes:
|
||||
- neo4j_data:/data
|
||||
- neo4j_logs:/logs
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "cypher-shell -u neo4j -p $${NEO4J_PASSWORD:-changeme} 'RETURN 1' || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
docling-studio-backend:
|
||||
depends_on:
|
||||
neo4j:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
NEO4J_URI: bolt://neo4j:7687
|
||||
NEO4J_USER: neo4j
|
||||
NEO4J_PASSWORD: ${NEO4J_PASSWORD:-changeme}
|
||||
|
||||
volumes:
|
||||
neo4j_data:
|
||||
neo4j_logs:
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Tests
|
||||
|
||||
### Unit tests
|
||||
- `tests/storage/neo4j/test_schema.py` — bootstrap is idempotent
|
||||
- `tests/storage/neo4j/test_tree_writer.py` — round-trip on synthetic DoclingDocument
|
||||
- `tests/storage/neo4j/test_chunk_writer.py` — chunks written with correct `DERIVED_FROM`
|
||||
|
||||
### Integration tests
|
||||
- `tests/integration/test_ingestion_pipeline.py` — full pipeline on a real PDF
|
||||
- PDF fixtures: 1 academic (complex heading hierarchy), 1 invoice (tables), 1 report (lists)
|
||||
|
||||
### E2E (bonus)
|
||||
- Upload PDF via UI → check structure in Neo4j Browser
|
||||
|
||||
---
|
||||
|
||||
## 8. Open decisions to settle before coding
|
||||
|
||||
1. **Neo4j edition**: Community (free) or AuraDB (managed) ?
|
||||
- Rec: Community in Docker for v0.5.0 dev/demo. AuraDB mentioned as prod option.
|
||||
|
||||
2. **Chunks: duplicate embeddings in Neo4j or OpenSearch ref ?**
|
||||
- Rec: OpenSearch ref (avoid duplication; OpenSearch remains source of truth for vectors). In v0.6+, consider native Neo4j vector index.
|
||||
|
||||
3. **Graph view UI: cytoscape or vis-network ?**
|
||||
- Rec: **cytoscape** — lighter, better layout API, used by Neo4j itself.
|
||||
|
||||
4. **Graph endpoint: return full doc or paginate ?**
|
||||
- Rec: full doc for v0.5 (reasonable cap at 200 pages). Pagination in v0.6.
|
||||
|
||||
5. **Error strategy**: if Neo4j is down at ingestion, fail or degrade gracefully ?
|
||||
- Rec: **fail fast** for v0.5 (avoid silent inconsistencies). `neo4j_required: bool` config option.
|
||||
|
||||
---
|
||||
|
||||
## 9. Hooks for later (v0.6.0+ — don't implement but prepare)
|
||||
|
||||
**EnrichmentWriter (v0.6)** — will need:
|
||||
- The reader (Neo4j → DoclingDocument) to re-materialize the doc, feed docling-agent, re-patch enrichments
|
||||
- A stage addable to `IngestionPipeline` without touching other stages
|
||||
- An `:Entity` label (not created in v0.5 but schema-compatible)
|
||||
|
||||
**Agent reasoning trace viewer (v0.6)** — will need:
|
||||
- An event stream (WebSocket) that v0.5 already prepares via the reactive UI
|
||||
- A node_ref ↔ Element correlation in Neo4j (our composite `self_ref` key is enough)
|
||||
|
||||
**TreeNavigationPort (v0.7)** — will need:
|
||||
- Optimized Cypher queries for descendant/ancestor walk (indexes already provisioned)
|
||||
|
||||
---
|
||||
|
||||
## 10. v0.5.0 success criteria
|
||||
|
||||
**Must have:**
|
||||
- [ ] A PDF uploaded to Docling-Studio is in Neo4j with structure preserved
|
||||
- [ ] Neo4j Browser shows the graph and is manually explorable
|
||||
- [ ] A graph visual in the Docling-Studio UI works
|
||||
- [ ] `docker compose up` works zero-config
|
||||
- [ ] README mentions Neo4j and describes the schema
|
||||
|
||||
**Nice to have (decreasing priority):**
|
||||
- [ ] Graph-ready / RAG-ready badge per doc
|
||||
- [ ] Live query explorer in the UI
|
||||
- [ ] 2–3 example queries in README that do something impossible with vector-only
|
||||
|
||||
**For the hackathon (post-release):**
|
||||
- [ ] 60s video: upload PDF → structure in Neo4j → cross-doc query impossible in vector-only
|
||||
- [ ] HackerNoon post explaining "graph-native documents" positioning
|
||||
- [ ] Explicit Neo4j partnership mention
|
||||
|
||||
---
|
||||
|
||||
## 11. Fundamental architectural decisions recap
|
||||
|
||||
| Question | Answer |
|
||||
|----------|---------|
|
||||
| Is Neo4j source of truth or cache ? | **Source of truth** for structure. OpenSearch remains source of truth for embeddings. |
|
||||
| Does chunking go away ? | No, v0.5.0 keeps existing chunking. "Chunkless" is Pipeline B, v0.6+. |
|
||||
| Can it be toggled per doc ? | Yes — `stages_applied` on Document + pipeline config |
|
||||
| What about OpenSearch ? | Stays, stores vectors. Neo4j tracks `(:Chunk)-[:DERIVED_FROM]->(:Element)` links. |
|
||||
| Multi-tenancy ? | `tenant_id` property on Document, Cypher filter |
|
||||
| Versioning ? | None for v0.5.0 — replace strategy on re-upload |
|
||||
|
||||
---
|
||||
|
||||
## Appendix — Demo queries
|
||||
|
||||
### Query 1 — All "Methods" sections across documents
|
||||
```cypher
|
||||
MATCH (d:Document)-[:HAS_ROOT]->(:Element)-[:PARENT_OF*]->(s:SectionHeader)
|
||||
WHERE toLower(s.text) CONTAINS 'method'
|
||||
RETURN d.title, s.text, s.level
|
||||
```
|
||||
|
||||
### Query 2 — Context of a chunk (parent + siblings)
|
||||
```cypher
|
||||
MATCH (c:Chunk {id: $chunk_id})-[:DERIVED_FROM]->(e:Element)
|
||||
MATCH (e)<-[:PARENT_OF]-(parent:Element)
|
||||
MATCH (parent)-[:PARENT_OF]->(sibling:Element)
|
||||
RETURN parent, collect(sibling) AS siblings
|
||||
```
|
||||
|
||||
### Query 3 — All tables from a document type
|
||||
```cypher
|
||||
MATCH (d:Document)-[:HAS_ROOT]->(:Element)-[:PARENT_OF*]->(t:Table)
|
||||
WHERE d.source_uri CONTAINS 'invoices/'
|
||||
RETURN d.title, t.caption, t.cells_json
|
||||
```
|
||||
|
||||
### Query 4 — Direct children of a section (ordered)
|
||||
```cypher
|
||||
MATCH (s:Element {doc_id: $doc_id, self_ref: $section_ref})
|
||||
MATCH (s)-[pof:PARENT_OF]->(child)
|
||||
RETURN child
|
||||
ORDER BY pof.order
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Single reference doc for Neo4j v0.5.0 implementation.
|
||||
Read this first in the implementation thread.*
|
||||
10
document-parser/infra/neo4j/__init__.py
Normal file
10
document-parser/infra/neo4j/__init__.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
"""Neo4j storage adapter — graph-native document structure.
|
||||
|
||||
Provides a thin driver wrapper, idempotent schema bootstrap, and
|
||||
walkers between DoclingDocument and the graph model.
|
||||
"""
|
||||
|
||||
from infra.neo4j.driver import Neo4jDriver, close_driver, get_driver
|
||||
from infra.neo4j.schema import bootstrap_schema
|
||||
|
||||
__all__ = ["Neo4jDriver", "bootstrap_schema", "close_driver", "get_driver"]
|
||||
48
document-parser/infra/neo4j/driver.py
Normal file
48
document-parser/infra/neo4j/driver.py
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
"""Async Neo4j driver wrapper.
|
||||
|
||||
Owns a single `AsyncDriver` per process. Callers acquire it via
|
||||
`get_driver()` and must call `close_driver()` at shutdown.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
from neo4j import AsyncDriver, AsyncGraphDatabase
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Neo4jDriver:
|
||||
driver: AsyncDriver
|
||||
database: str = "neo4j"
|
||||
|
||||
|
||||
_instance: Neo4jDriver | None = None
|
||||
|
||||
|
||||
async def get_driver(uri: str, user: str, password: str, database: str = "neo4j") -> Neo4jDriver:
|
||||
"""Return the process-wide driver, creating it on first call.
|
||||
|
||||
Verifies connectivity once at creation — raises if the server is unreachable.
|
||||
"""
|
||||
global _instance
|
||||
if _instance is not None:
|
||||
return _instance
|
||||
|
||||
driver = AsyncGraphDatabase.driver(uri, auth=(user, password))
|
||||
await driver.verify_connectivity()
|
||||
logger.info("Neo4j driver connected to %s (db=%s)", uri, database)
|
||||
_instance = Neo4jDriver(driver=driver, database=database)
|
||||
return _instance
|
||||
|
||||
|
||||
async def close_driver() -> None:
|
||||
global _instance
|
||||
if _instance is None:
|
||||
return
|
||||
await _instance.driver.close()
|
||||
_instance = None
|
||||
logger.info("Neo4j driver closed")
|
||||
51
document-parser/infra/neo4j/schema.py
Normal file
51
document-parser/infra/neo4j/schema.py
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
"""Idempotent Neo4j schema bootstrap.
|
||||
|
||||
Runs at backend startup. All statements use `IF NOT EXISTS`, so calling
|
||||
this multiple times is safe — it's the contract integration tests rely on.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from infra.neo4j.driver import Neo4jDriver
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
CONSTRAINTS: tuple[str, ...] = (
|
||||
"CREATE CONSTRAINT document_id IF NOT EXISTS "
|
||||
"FOR (d:Document) REQUIRE d.id IS UNIQUE",
|
||||
"CREATE CONSTRAINT element_composite IF NOT EXISTS "
|
||||
"FOR (e:Element) REQUIRE (e.doc_id, e.self_ref) IS UNIQUE",
|
||||
"CREATE CONSTRAINT page_composite IF NOT EXISTS "
|
||||
"FOR (p:Page) REQUIRE (p.doc_id, p.page_no) IS UNIQUE",
|
||||
"CREATE CONSTRAINT chunk_id IF NOT EXISTS "
|
||||
"FOR (c:Chunk) REQUIRE c.id IS UNIQUE",
|
||||
)
|
||||
|
||||
INDEXES: tuple[str, ...] = (
|
||||
"CREATE INDEX element_doc IF NOT EXISTS FOR (e:Element) ON (e.doc_id)",
|
||||
"CREATE INDEX chunk_doc IF NOT EXISTS FOR (c:Chunk) ON (c.doc_id)",
|
||||
)
|
||||
|
||||
FULLTEXT_INDEXES: tuple[str, ...] = (
|
||||
"CREATE FULLTEXT INDEX element_text IF NOT EXISTS "
|
||||
"FOR (e:Element) ON EACH [e.text]",
|
||||
)
|
||||
|
||||
|
||||
async def bootstrap_schema(neo: Neo4jDriver) -> None:
|
||||
"""Create constraints and indexes required by the graph model.
|
||||
|
||||
Idempotent: safe to call on every startup.
|
||||
"""
|
||||
async with neo.driver.session(database=neo.database) as session:
|
||||
for stmt in (*CONSTRAINTS, *INDEXES, *FULLTEXT_INDEXES):
|
||||
await session.run(stmt)
|
||||
logger.info(
|
||||
"Neo4j schema bootstrapped (%d constraints, %d indexes, %d fulltext)",
|
||||
len(CONSTRAINTS),
|
||||
len(INDEXES),
|
||||
len(FULLTEXT_INDEXES),
|
||||
)
|
||||
|
|
@ -25,6 +25,9 @@ class Settings:
|
|||
batch_page_size: int = 0 # 0 = disabled, > 0 = pages per batch
|
||||
opensearch_url: str = "" # empty = disabled
|
||||
embedding_url: str = "" # empty = disabled (e.g. http://localhost:8001)
|
||||
neo4j_uri: str = "" # empty = disabled (e.g. bolt://neo4j:7687)
|
||||
neo4j_user: str = "neo4j"
|
||||
neo4j_password: str = "changeme"
|
||||
opensearch_default_limit: int = 1000 # max chunks returned by get_chunks
|
||||
embedding_dimension: int = 384 # Granite Embedding 30M / all-MiniLM-L6-v2
|
||||
upload_dir: str = "./uploads"
|
||||
|
|
@ -102,6 +105,9 @@ class Settings:
|
|||
batch_page_size=int(os.environ.get("BATCH_PAGE_SIZE", "10")),
|
||||
opensearch_url=os.environ.get("OPENSEARCH_URL", ""),
|
||||
embedding_url=os.environ.get("EMBEDDING_URL", ""),
|
||||
neo4j_uri=os.environ.get("NEO4J_URI", ""),
|
||||
neo4j_user=os.environ.get("NEO4J_USER", "neo4j"),
|
||||
neo4j_password=os.environ.get("NEO4J_PASSWORD", "changeme"),
|
||||
opensearch_default_limit=int(os.environ.get("OPENSEARCH_DEFAULT_LIMIT", "1000")),
|
||||
embedding_dimension=int(os.environ.get("EMBEDDING_DIMENSION", "384")),
|
||||
upload_dir=os.environ.get("UPLOAD_DIR", "./uploads"),
|
||||
|
|
|
|||
|
|
@ -93,6 +93,28 @@ def _build_analysis_service(
|
|||
)
|
||||
|
||||
|
||||
async def _init_neo4j():
|
||||
"""Initialize the Neo4j driver and bootstrap schema — skip if not configured."""
|
||||
if not settings.neo4j_uri:
|
||||
logger.info("Neo4j disabled (NEO4J_URI not set)")
|
||||
return None
|
||||
|
||||
from infra.neo4j import Neo4jDriver, bootstrap_schema, get_driver
|
||||
|
||||
try:
|
||||
neo = await get_driver(
|
||||
settings.neo4j_uri,
|
||||
settings.neo4j_user,
|
||||
settings.neo4j_password,
|
||||
)
|
||||
await bootstrap_schema(neo)
|
||||
logger.info("Neo4j ready (uri=%s)", settings.neo4j_uri)
|
||||
return neo
|
||||
except Exception:
|
||||
logger.exception("Neo4j init failed — continuing without graph storage")
|
||||
return None
|
||||
|
||||
|
||||
def _build_ingestion_service() -> IngestionService | None:
|
||||
"""Build the ingestion service — only if embedding + opensearch are configured."""
|
||||
if not settings.embedding_url or not settings.opensearch_url:
|
||||
|
|
@ -150,8 +172,15 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
|||
if ingestion_service is not None:
|
||||
app.include_router(ingestion_router)
|
||||
logger.info("Ingestion router mounted")
|
||||
app.state.neo4j = await _init_neo4j()
|
||||
logger.info("Docling Studio backend ready (engine=%s)", settings.conversion_engine)
|
||||
yield
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
if app.state.neo4j is not None:
|
||||
from infra.neo4j import close_driver
|
||||
|
||||
await close_driver()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
|
|
|
|||
|
|
@ -8,3 +8,4 @@ aiosqlite>=0.20.0,<1.0.0
|
|||
httpx>=0.27.0,<1.0.0
|
||||
pypdfium2>=4.0.0,<5.0.0
|
||||
opensearch-py[async]>=2.6.0,<3.0.0
|
||||
neo4j>=5.15.0,<6.0.0
|
||||
|
|
|
|||
0
document-parser/tests/neo4j/__init__.py
Normal file
0
document-parser/tests/neo4j/__init__.py
Normal file
36
document-parser/tests/neo4j/conftest.py
Normal file
36
document-parser/tests/neo4j/conftest.py
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
"""Shared fixtures for Neo4j integration tests.
|
||||
|
||||
These tests are skipped unless a live Neo4j is reachable via NEO4J_TEST_URI
|
||||
(defaulting to bolt://localhost:7687). CI spins up `neo4j:5.15-community`
|
||||
alongside the job; locally run `docker compose -f docker-compose.dev.yml up neo4j`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from infra.neo4j import close_driver, get_driver
|
||||
|
||||
|
||||
def _cfg() -> tuple[str, str, str]:
|
||||
return (
|
||||
os.environ.get("NEO4J_TEST_URI", "bolt://localhost:7687"),
|
||||
os.environ.get("NEO4J_TEST_USER", "neo4j"),
|
||||
os.environ.get("NEO4J_TEST_PASSWORD", "changeme"),
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def neo4j_driver():
|
||||
uri, user, password = _cfg()
|
||||
try:
|
||||
neo = await get_driver(uri, user, password)
|
||||
except Exception as exc:
|
||||
pytest.skip(f"Neo4j not reachable at {uri}: {exc}")
|
||||
# Wipe DB before each test — integration tests assume an empty graph.
|
||||
async with neo.driver.session(database=neo.database) as session:
|
||||
await session.run("MATCH (n) DETACH DELETE n")
|
||||
yield neo
|
||||
await close_driver()
|
||||
32
document-parser/tests/neo4j/test_document_roundtrip.py
Normal file
32
document-parser/tests/neo4j/test_document_roundtrip.py
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
"""Minimal Document node round-trip — validates the driver + schema end-to-end."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from infra.neo4j.schema import bootstrap_schema
|
||||
|
||||
|
||||
async def test_document_write_read_delete(neo4j_driver):
|
||||
await bootstrap_schema(neo4j_driver)
|
||||
|
||||
async with neo4j_driver.driver.session(database=neo4j_driver.database) as session:
|
||||
await session.run(
|
||||
"CREATE (d:Document {id: $id, title: $title, tenant_id: $tenant})",
|
||||
id="doc-42",
|
||||
title="Round-trip fixture",
|
||||
tenant="default",
|
||||
)
|
||||
|
||||
result = await session.run(
|
||||
"MATCH (d:Document {id: $id}) RETURN d.title AS title, d.tenant_id AS tenant",
|
||||
id="doc-42",
|
||||
)
|
||||
record = await result.single()
|
||||
assert record is not None
|
||||
assert record["title"] == "Round-trip fixture"
|
||||
assert record["tenant"] == "default"
|
||||
|
||||
await session.run("MATCH (d:Document {id: $id}) DETACH DELETE d", id="doc-42")
|
||||
gone = await (
|
||||
await session.run("MATCH (d:Document {id: $id}) RETURN d", id="doc-42")
|
||||
).single()
|
||||
assert gone is None
|
||||
10
document-parser/tests/neo4j/test_driver.py
Normal file
10
document-parser/tests/neo4j/test_driver.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
"""Neo4j driver connectivity smoke test."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
async def test_driver_connects_and_runs_cypher(neo4j_driver):
|
||||
async with neo4j_driver.driver.session(database=neo4j_driver.database) as session:
|
||||
result = await session.run("RETURN 1 AS x")
|
||||
record = await result.single()
|
||||
assert record["x"] == 1
|
||||
38
document-parser/tests/neo4j/test_schema.py
Normal file
38
document-parser/tests/neo4j/test_schema.py
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
"""Schema bootstrap is idempotent and produces the expected constraints/indexes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from infra.neo4j.schema import CONSTRAINTS, FULLTEXT_INDEXES, INDEXES, bootstrap_schema
|
||||
|
||||
|
||||
async def _count_schema(neo4j_driver) -> tuple[int, int]:
|
||||
async with neo4j_driver.driver.session(database=neo4j_driver.database) as session:
|
||||
constraints = await (await session.run("SHOW CONSTRAINTS")).data()
|
||||
indexes = await (await session.run("SHOW INDEXES")).data()
|
||||
return len(constraints), len(indexes)
|
||||
|
||||
|
||||
async def test_bootstrap_is_idempotent(neo4j_driver):
|
||||
await bootstrap_schema(neo4j_driver)
|
||||
first = await _count_schema(neo4j_driver)
|
||||
|
||||
# Running a second time must not duplicate anything.
|
||||
await bootstrap_schema(neo4j_driver)
|
||||
second = await _count_schema(neo4j_driver)
|
||||
|
||||
assert first == second
|
||||
# Sanity: we created at least what we declared.
|
||||
assert first[0] >= len(CONSTRAINTS)
|
||||
assert first[1] >= len(INDEXES) + len(FULLTEXT_INDEXES)
|
||||
|
||||
|
||||
async def test_document_id_is_unique(neo4j_driver):
|
||||
await bootstrap_schema(neo4j_driver)
|
||||
async with neo4j_driver.driver.session(database=neo4j_driver.database) as session:
|
||||
await session.run("CREATE (d:Document {id: 'doc-1', title: 'first'})")
|
||||
with_err: Exception | None = None
|
||||
try:
|
||||
await session.run("CREATE (d:Document {id: 'doc-1', title: 'dup'})")
|
||||
except Exception as exc:
|
||||
with_err = exc
|
||||
assert with_err is not None, "unique constraint on Document.id must reject duplicates"
|
||||
Loading…
Reference in a new issue