Human-in-the-loop in ag-ui-example
This commit is contained in:
parent
9818cac4eb
commit
ff13be1210
8 changed files with 443 additions and 92 deletions
|
|
@ -12,6 +12,8 @@
|
||||||
- New `human_decide` graph node emits AG-UI tool calls (`TOOL_CALL_START/ARGS/END`) for frontend integration
|
- New `human_decide` graph node emits AG-UI tool calls (`TOOL_CALL_START/ARGS/END`) for frontend integration
|
||||||
- New `emit_tool_call_start()`, `emit_tool_call_args()`, `emit_tool_call_end()` AG-UI event helpers
|
- New `emit_tool_call_start()`, `emit_tool_call_args()`, `emit_tool_call_end()` AG-UI event helpers
|
||||||
- New `AGUIEmitter.emit()` method for direct event emission
|
- New `AGUIEmitter.emit()` method for direct event emission
|
||||||
|
- **AG-UI Research Example**: Updated with interactive decision UI
|
||||||
|
- Decision panel with question editing (add/remove) at each decision point
|
||||||
- **HotpotQA Evaluation**: Added HotpotQA dataset adapter for multi-hop QA benchmarks
|
- **HotpotQA Evaluation**: Added HotpotQA dataset adapter for multi-hop QA benchmarks
|
||||||
- Extracts unique documents from validation set context paragraphs
|
- Extracts unique documents from validation set context paragraphs
|
||||||
- Uses MAP for retrieval evaluation (multiple supporting documents per question)
|
- Uses MAP for retrieval evaluation (multiple supporting documents per question)
|
||||||
|
|
|
||||||
|
|
@ -219,4 +219,8 @@ In interactive mode, you can:
|
||||||
- Execute searches and review collected answers
|
- Execute searches and review collected answers
|
||||||
- Continue researching or synthesize when ready
|
- Continue researching or synthesize when ready
|
||||||
|
|
||||||
For a web-based interactive experience, see the [AG-UI Research Example](https://github.com/ggozad/haiku.rag/tree/main/examples/ag-ui-research).
|
For a web-based interactive experience with visual decision UI, see the [AG-UI Research Example](https://github.com/ggozad/haiku.rag/tree/main/examples/ag-ui-research). The web interface provides:
|
||||||
|
|
||||||
|
- Question editing panel to add/remove sub-questions at decision points
|
||||||
|
- Search and Generate Report buttons for controlling research flow
|
||||||
|
- Live state display showing answers, confidence, and progress
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,13 @@
|
||||||
# Interactive Research Assistant
|
# Interactive Research Assistant
|
||||||
|
|
||||||
Research assistant powered by [haiku.rag](https://ggozad.github.io/haiku.rag/), [Pydantic Graph](https://ai.pydantic.dev/graph/), and [AG-UI](https://docs.ag-ui.com/). Ask complex questions and watch the research process unfold in real-time.
|
Research assistant powered by [haiku.rag](https://ggozad.github.io/haiku.rag/), [Pydantic Graph](https://ai.pydantic.dev/graph/), and [AG-UI](https://docs.ag-ui.com/). Ask complex questions and watch the research process unfold in real-time with human-in-the-loop control.
|
||||||
|
|
||||||
[Watch demo video](https://vimeo.com/1128874386)
|
[Watch demo video](https://vimeo.com/1128874386)
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- **Multi-iteration research graph**: Automated question decomposition and search
|
- **Human-in-the-loop research**: Review and modify questions at decision points, then continue searching or generate report
|
||||||
- **Intelligent evaluation**: Confidence-based decision making with automatic iteration until sufficient information is gathered
|
- **Multi-iteration research graph**: Automated question decomposition and parallel search
|
||||||
- **Live state synchronization**: Real-time delta updates of research progress via AG-UI protocol
|
- **Live state synchronization**: Real-time delta updates of research progress via AG-UI protocol
|
||||||
- **Rich reporting**: Generates comprehensive research reports with findings, conclusions, and sources
|
- **Rich reporting**: Generates comprehensive research reports with findings, conclusions, and sources
|
||||||
|
|
||||||
|
|
@ -25,9 +25,7 @@ Research assistant powered by [haiku.rag](https://ggozad.github.io/haiku.rag/),
|
||||||
|
|
||||||
**Option A: Create a new database**
|
**Option A: Create a new database**
|
||||||
```bash
|
```bash
|
||||||
mkdir -p data
|
haiku-rag init --db data/haiku_rag.lancedb
|
||||||
haiku-rag add "Your documents here" --db data/haiku_rag.lancedb
|
|
||||||
# Or add from files
|
|
||||||
haiku-rag add-src document.pdf --db data/haiku_rag.lancedb
|
haiku-rag add-src document.pdf --db data/haiku_rag.lancedb
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -63,27 +61,29 @@ Research assistant powered by [haiku.rag](https://ggozad.github.io/haiku.rag/),
|
||||||
DB_PATH=/path/to/your/existing/haiku_rag.lancedb # If using an existing db.
|
DB_PATH=/path/to/your/existing/haiku_rag.lancedb # If using an existing db.
|
||||||
```
|
```
|
||||||
|
|
||||||
1. **Start the application**
|
4. **Start the application**
|
||||||
```bash
|
```bash
|
||||||
docker compose up --build
|
docker compose up --build
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Access the interface**
|
5. **Access the interface**
|
||||||
- Frontend: http://localhost:3000
|
- Frontend: http://localhost:3000
|
||||||
- Backend health: http://localhost:8000/health
|
- Backend health: http://localhost:8000/health
|
||||||
|
|
||||||
## How It Works
|
## How It Works
|
||||||
|
|
||||||
1. **Ask a question**: Type your research question in the chat
|
1. **Ask a question**: Type your research question in the chat
|
||||||
2. **Plan phase**: The research graph automatically:
|
2. **Plan phase**: The research graph decomposes your question into targeted sub-questions
|
||||||
- Decomposes your question into targeted sub-questions
|
3. **Decision point**: Review the proposed questions in the right panel
|
||||||
- Gathers initial context about the topic
|
- Add new questions using the input field
|
||||||
3. **Research iterations**: The graph autonomously:
|
- Remove questions you don't need
|
||||||
- Searches the knowledge base for each sub-question in parallel
|
- Click **Search** to execute searches for pending questions
|
||||||
- Assesses confidence in gathered information
|
- Click **Generate Report** to skip to synthesis (when you have enough answers)
|
||||||
- Generates new follow-up questions if needed
|
4. **Research iterations**: After each search cycle, you return to a decision point where you can:
|
||||||
- Iterates until confidence threshold is met or max iterations reached
|
- Review collected answers
|
||||||
4. **Synthesis**: Generates a comprehensive research report with:
|
- Add follow-up questions based on findings
|
||||||
|
- Continue searching or generate the final report
|
||||||
|
5. **Synthesis**: Generates a comprehensive research report with:
|
||||||
- Executive summary
|
- Executive summary
|
||||||
- Main findings with supporting evidence
|
- Main findings with supporting evidence
|
||||||
- Conclusions and recommendations
|
- Conclusions and recommendations
|
||||||
|
|
@ -99,32 +99,35 @@ This example demonstrates the **agent+graph** architecture pattern:
|
||||||
- Pydantic AI agent handles user conversations
|
- Pydantic AI agent handles user conversations
|
||||||
- Decides when to invoke the research tool based on user intent
|
- Decides when to invoke the research tool based on user intent
|
||||||
- Responds directly to greetings/casual chat without tools
|
- Responds directly to greetings/casual chat without tools
|
||||||
- Formats research results for the user
|
|
||||||
|
|
||||||
2. **Research Graph** (haiku.rag):
|
2. **Interactive Research Graph** (haiku.rag):
|
||||||
- Multi-step research workflow invoked by the agent's tool
|
- Multi-step research workflow invoked by the agent's tool
|
||||||
- Autonomous execution with plan → search → analyze → decide → synthesize flow
|
- Pauses at decision points waiting for human input via async queue
|
||||||
- Emits AG-UI events for real-time progress tracking
|
- Emits AG-UI events for real-time progress tracking
|
||||||
|
|
||||||
3. **Shared Event Stream**:
|
3. **Decision Endpoint** (`main.py`):
|
||||||
|
- `/v1/research/decide` receives human decisions from frontend
|
||||||
|
- Forwards decisions to the waiting graph via `HumanDecision` queue
|
||||||
|
- Supports actions: `search`, `synthesize`, `modify_questions`
|
||||||
|
|
||||||
|
4. **Shared Event Stream**:
|
||||||
- `AGUIEmitter` is shared between agent and graph
|
- `AGUIEmitter` is shared between agent and graph
|
||||||
- Events from both flow through a single stream to the frontend
|
- Events from both flow through a single stream to the frontend
|
||||||
- Custom streaming endpoint (`main.py`) uses anyio memory streams for proper async handling
|
- `STATE_DELTA` events sync research state to frontend in real-time
|
||||||
|
|
||||||
### Components
|
### Components
|
||||||
|
|
||||||
- **Backend** (Python):
|
- **Backend** (Python):
|
||||||
- Uses published `ghcr.io/ggozad/haiku.rag:latest` Docker image as base
|
- Uses published `ghcr.io/ggozad/haiku.rag:latest` Docker image as base
|
||||||
- `agent.py`: Pydantic AI agent with `run_research` tool
|
- `agent.py`: Pydantic AI agent with `run_research` tool, manages `ActiveResearch` registry
|
||||||
- `main.py`: Custom AG-UI streaming endpoint with anyio memory object streams
|
- `main.py`: Custom AG-UI streaming endpoint, decision endpoint for human input
|
||||||
- Real-time event forwarding from emitter to SSE stream
|
- Real-time event forwarding from emitter to SSE stream
|
||||||
- Filters out `ACTIVITY_SNAPSHOT` events (not yet supported by CopilotKit)
|
|
||||||
|
|
||||||
- **Frontend** (Next.js/React):
|
- **Frontend** (Next.js/React):
|
||||||
- CopilotKit for AG-UI protocol integration
|
- CopilotKit for AG-UI protocol integration
|
||||||
- Split-pane UI: chat on left, live research state on right
|
- Split-pane UI: chat on left, live research state on right
|
||||||
|
- Decision UI: question editor with add/remove, search and generate report buttons
|
||||||
- Real-time state synchronization via Server-Sent Events (SSE)
|
- Real-time state synchronization via Server-Sent Events (SSE)
|
||||||
- `StateDisplay` component with collapsible sections for questions and report
|
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
"""Research assistant agent with graph integration."""
|
"""Research assistant agent with graph integration."""
|
||||||
|
|
||||||
from dataclasses import dataclass
|
import asyncio
|
||||||
|
from dataclasses import dataclass, field
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
|
@ -11,7 +12,7 @@ from haiku.rag.config import load_yaml_config
|
||||||
from haiku.rag.config.models import AppConfig
|
from haiku.rag.config.models import AppConfig
|
||||||
from haiku.rag.graph.research.dependencies import ResearchContext
|
from haiku.rag.graph.research.dependencies import ResearchContext
|
||||||
from haiku.rag.graph.research.graph import build_research_graph
|
from haiku.rag.graph.research.graph import build_research_graph
|
||||||
from haiku.rag.graph.research.state import ResearchDeps, ResearchState
|
from haiku.rag.graph.research.state import HumanDecision, ResearchDeps, ResearchState
|
||||||
from haiku.rag.utils import get_model
|
from haiku.rag.utils import get_model
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
|
|
@ -27,6 +28,20 @@ Config = (
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ActiveResearch:
|
||||||
|
"""Tracks state for active research awaiting human decision."""
|
||||||
|
|
||||||
|
queue: asyncio.Queue[HumanDecision]
|
||||||
|
sub_questions: list[str] = field(default_factory=list)
|
||||||
|
qa_responses: list[dict] = field(default_factory=list)
|
||||||
|
original_question: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
# Global registry of active research by thread_id
|
||||||
|
_active_research: dict[str, ActiveResearch] = {}
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class AgentDeps:
|
class AgentDeps:
|
||||||
"""Dependencies for research agent."""
|
"""Dependencies for research agent."""
|
||||||
|
|
@ -34,6 +49,7 @@ class AgentDeps:
|
||||||
client: HaikuRAG
|
client: HaikuRAG
|
||||||
agui_emitter: "AGUIEmitter[ResearchState, ResearchReport] | None" = None
|
agui_emitter: "AGUIEmitter[ResearchState, ResearchReport] | None" = None
|
||||||
search_filter: str | None = None
|
search_filter: str | None = None
|
||||||
|
thread_id: str | None = None
|
||||||
|
|
||||||
|
|
||||||
model = get_model(Config.research.model, Config)
|
model = get_model(Config.research.model, Config)
|
||||||
|
|
@ -50,10 +66,10 @@ CRITICAL RULES:
|
||||||
4. NEVER answer substantive questions from your own knowledge - always use the tool
|
4. NEVER answer substantive questions from your own knowledge - always use the tool
|
||||||
|
|
||||||
How to decide:
|
How to decide:
|
||||||
- "Hi" / "Hello" / "How are you?" → Respond directly, NO tools
|
- "Hi" / "Hello" / "How are you?" -> Respond directly, NO tools
|
||||||
- "What can you do?" → Respond directly, NO tools
|
- "What can you do?" -> Respond directly, NO tools
|
||||||
- "How does X work in the codebase?" → Use run_research tool
|
- "How does X work in the codebase?" -> Use run_research tool
|
||||||
- "Tell me about Y" → Use run_research tool
|
- "Tell me about Y" -> Use run_research tool
|
||||||
|
|
||||||
When you use run_research, the graph will decompose questions, search the knowledge base,
|
When you use run_research, the graph will decompose questions, search the knowledge base,
|
||||||
and generate a comprehensive report.
|
and generate a comprehensive report.
|
||||||
|
|
@ -70,23 +86,39 @@ async def run_research(ctx: RunContext[AgentDeps], question: str) -> str:
|
||||||
DO NOT use for greetings or casual conversation.
|
DO NOT use for greetings or casual conversation.
|
||||||
"""
|
"""
|
||||||
if ctx.deps.agui_emitter:
|
if ctx.deps.agui_emitter:
|
||||||
ctx.deps.agui_emitter.log(f"🔍 Starting research on: {question}")
|
ctx.deps.agui_emitter.log(f"Starting research on: {question}")
|
||||||
|
|
||||||
graph = build_research_graph(Config)
|
# Create queue for human decisions
|
||||||
|
queue: asyncio.Queue[HumanDecision] = asyncio.Queue()
|
||||||
|
|
||||||
|
# Build interactive graph
|
||||||
|
graph = build_research_graph(Config, interactive=True)
|
||||||
context = ResearchContext(original_question=question)
|
context = ResearchContext(original_question=question)
|
||||||
state = ResearchState.from_config(context=context, config=Config)
|
state = ResearchState.from_config(context=context, config=Config)
|
||||||
state.search_filter = ctx.deps.search_filter
|
state.search_filter = ctx.deps.search_filter
|
||||||
|
|
||||||
|
# Register active research for decision endpoint to find
|
||||||
|
thread_id = ctx.deps.thread_id
|
||||||
|
if thread_id:
|
||||||
|
_active_research[thread_id] = ActiveResearch(
|
||||||
|
queue=queue,
|
||||||
|
sub_questions=[],
|
||||||
|
qa_responses=[],
|
||||||
|
original_question=question,
|
||||||
|
)
|
||||||
|
|
||||||
graph_deps = ResearchDeps(
|
graph_deps = ResearchDeps(
|
||||||
client=ctx.deps.client,
|
client=ctx.deps.client,
|
||||||
agui_emitter=ctx.deps.agui_emitter,
|
agui_emitter=ctx.deps.agui_emitter,
|
||||||
|
human_input_queue=queue,
|
||||||
|
interactive=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
result = await graph.run(state=state, deps=graph_deps)
|
result = await graph.run(state=state, deps=graph_deps)
|
||||||
|
|
||||||
if ctx.deps.agui_emitter:
|
if ctx.deps.agui_emitter:
|
||||||
ctx.deps.agui_emitter.log("✅ Research complete!")
|
ctx.deps.agui_emitter.log("Research complete!")
|
||||||
|
|
||||||
return f"""Research completed successfully!
|
return f"""Research completed successfully!
|
||||||
|
|
||||||
|
|
@ -108,5 +140,9 @@ The full research report with all citations has been provided to the user.
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
if ctx.deps.agui_emitter:
|
if ctx.deps.agui_emitter:
|
||||||
ctx.deps.agui_emitter.log(f"❌ Research error: {str(e)}")
|
ctx.deps.agui_emitter.log(f"Research error: {str(e)}")
|
||||||
return f"I encountered an error while researching: {str(e)}"
|
return f"I encountered an error while researching: {str(e)}"
|
||||||
|
finally:
|
||||||
|
# Cleanup
|
||||||
|
if thread_id and thread_id in _active_research:
|
||||||
|
del _active_research[thread_id]
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,9 @@
|
||||||
|
import json
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from agent import AgentDeps, agent
|
from agent import AgentDeps, _active_research, agent
|
||||||
from anyio import create_memory_object_stream, create_task_group
|
from anyio import create_memory_object_stream, create_task_group
|
||||||
from anyio.streams.memory import MemoryObjectSendStream
|
from anyio.streams.memory import MemoryObjectSendStream
|
||||||
from starlette.applications import Starlette
|
from starlette.applications import Starlette
|
||||||
|
|
@ -19,7 +20,7 @@ from haiku.rag.graph.agui.emitter import AGUIEmitter
|
||||||
from haiku.rag.graph.agui.server import RunAgentInput, format_sse_event
|
from haiku.rag.graph.agui.server import RunAgentInput, format_sse_event
|
||||||
from haiku.rag.graph.research.dependencies import ResearchContext
|
from haiku.rag.graph.research.dependencies import ResearchContext
|
||||||
from haiku.rag.graph.research.models import ResearchReport
|
from haiku.rag.graph.research.models import ResearchReport
|
||||||
from haiku.rag.graph.research.state import ResearchState
|
from haiku.rag.graph.research.state import HumanDecision, ResearchState
|
||||||
|
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||||
|
|
@ -79,11 +80,11 @@ async def stream_research_agent(request: Request) -> StreamingResponse:
|
||||||
"""Execute agent and forward emitter events to memory stream."""
|
"""Execute agent and forward emitter events to memory stream."""
|
||||||
async with send_stream:
|
async with send_stream:
|
||||||
try:
|
try:
|
||||||
# Create shared emitter
|
# Create shared emitter (use_deltas=True for CopilotKit compatibility)
|
||||||
emitter: AGUIEmitter[ResearchState, ResearchReport] = AGUIEmitter(
|
emitter: AGUIEmitter[ResearchState, ResearchReport] = AGUIEmitter(
|
||||||
thread_id=input_data.thread_id,
|
thread_id=input_data.thread_id,
|
||||||
run_id=input_data.run_id,
|
run_id=input_data.run_id,
|
||||||
use_deltas=False,
|
use_deltas=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Get client
|
# Get client
|
||||||
|
|
@ -99,11 +100,14 @@ async def stream_research_agent(request: Request) -> StreamingResponse:
|
||||||
ids_str = ", ".join(f"'{id}'" for id in document_ids)
|
ids_str = ", ".join(f"'{id}'" for id in document_ids)
|
||||||
search_filter = f"id IN ({ids_str})"
|
search_filter = f"id IN ({ids_str})"
|
||||||
|
|
||||||
|
thread_id = input_data.thread_id
|
||||||
|
|
||||||
# Create agent dependencies with shared emitter
|
# Create agent dependencies with shared emitter
|
||||||
agent_deps = AgentDeps(
|
agent_deps = AgentDeps(
|
||||||
client=client,
|
client=client,
|
||||||
agui_emitter=emitter,
|
agui_emitter=emitter,
|
||||||
search_filter=search_filter,
|
search_filter=search_filter,
|
||||||
|
thread_id=thread_id,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Start run with empty initial state
|
# Start run with empty initial state
|
||||||
|
|
@ -148,6 +152,37 @@ async def stream_research_agent(request: Request) -> StreamingResponse:
|
||||||
await send_stream.send(format_sse_event(delta_event))
|
await send_stream.send(format_sse_event(delta_event))
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
# When human_decision tool starts, set awaiting_decision flag
|
||||||
|
if event_type == "TOOL_CALL_START":
|
||||||
|
tool_name = event.get("toolCallName")
|
||||||
|
if tool_name == "human_decision":
|
||||||
|
delta_event = {
|
||||||
|
"type": "STATE_DELTA",
|
||||||
|
"delta": [
|
||||||
|
{
|
||||||
|
"op": "add",
|
||||||
|
"path": "/awaiting_decision",
|
||||||
|
"value": True,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
await send_stream.send(format_sse_event(delta_event))
|
||||||
|
|
||||||
|
# Sync state to ActiveResearch when human_decision tool call
|
||||||
|
if event_type == "TOOL_CALL_ARGS" and thread_id:
|
||||||
|
delta = event.get("delta", "{}")
|
||||||
|
args = (
|
||||||
|
json.loads(delta) if isinstance(delta, str) else delta
|
||||||
|
)
|
||||||
|
active = _active_research.get(thread_id)
|
||||||
|
if active:
|
||||||
|
active.sub_questions = list(
|
||||||
|
args.get("sub_questions", [])
|
||||||
|
)
|
||||||
|
active.qa_responses = list(args.get("qa_responses", []))
|
||||||
|
if "original_question" in args:
|
||||||
|
active.original_question = args["original_question"]
|
||||||
|
|
||||||
await send_stream.send(format_sse_event(event))
|
await send_stream.send(format_sse_event(event))
|
||||||
|
|
||||||
# Run agent and event forwarding concurrently
|
# Run agent and event forwarding concurrently
|
||||||
|
|
@ -248,10 +283,36 @@ async def visualize_chunk(request: Request) -> JSONResponse:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def research_decide(request: Request) -> JSONResponse:
|
||||||
|
"""Endpoint to receive human decisions for active research."""
|
||||||
|
body = await request.json()
|
||||||
|
action = body.get("action", "search")
|
||||||
|
questions = body.get("questions", [])
|
||||||
|
|
||||||
|
# Get first active research (single-user example)
|
||||||
|
active = next(iter(_active_research.values()), None)
|
||||||
|
if not active:
|
||||||
|
return JSONResponse({"error": "No active research found"}, status_code=404)
|
||||||
|
|
||||||
|
# When "search" action is sent with questions, use "modify_questions" to update them
|
||||||
|
effective_action = (
|
||||||
|
"modify_questions" if action == "search" and questions else action
|
||||||
|
)
|
||||||
|
|
||||||
|
decision = HumanDecision(
|
||||||
|
action=effective_action,
|
||||||
|
questions=questions or None,
|
||||||
|
)
|
||||||
|
await active.queue.put(decision)
|
||||||
|
|
||||||
|
return JSONResponse({"status": "ok", "action": effective_action})
|
||||||
|
|
||||||
|
|
||||||
# Create Starlette app
|
# Create Starlette app
|
||||||
app = Starlette(
|
app = Starlette(
|
||||||
routes=[
|
routes=[
|
||||||
Route("/v1/research/stream", stream_research_agent, methods=["POST"]),
|
Route("/v1/research/stream", stream_research_agent, methods=["POST"]),
|
||||||
|
Route("/v1/research/decide", research_decide, methods=["POST"]),
|
||||||
Route("/api/documents", list_documents, methods=["GET"]),
|
Route("/api/documents", list_documents, methods=["GET"]),
|
||||||
Route("/api/visualize/{chunk_id}", visualize_chunk, methods=["GET"]),
|
Route("/api/visualize/{chunk_id}", visualize_chunk, methods=["GET"]),
|
||||||
Route("/health", health_check, methods=["GET"]),
|
Route("/health", health_check, methods=["GET"]),
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@
|
||||||
import { CopilotKit, useCoAgent } from "@copilotkit/react-core";
|
import { CopilotKit, useCoAgent } from "@copilotkit/react-core";
|
||||||
import { CopilotChat } from "@copilotkit/react-ui";
|
import { CopilotChat } from "@copilotkit/react-ui";
|
||||||
import "@copilotkit/react-ui/styles.css";
|
import "@copilotkit/react-ui/styles.css";
|
||||||
|
import { useState, useEffect } from "react";
|
||||||
import DocumentSelector from "./DocumentSelector";
|
import DocumentSelector from "./DocumentSelector";
|
||||||
import StateDisplay from "./StateDisplay";
|
import StateDisplay from "./StateDisplay";
|
||||||
|
|
||||||
|
|
@ -58,6 +59,7 @@ interface ResearchState {
|
||||||
current_activity?: string;
|
current_activity?: string;
|
||||||
current_activity_message?: string;
|
current_activity_message?: string;
|
||||||
documentFilter?: string[];
|
documentFilter?: string[];
|
||||||
|
awaiting_decision?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
function AgentContent() {
|
function AgentContent() {
|
||||||
|
|
@ -75,13 +77,61 @@ function AgentContent() {
|
||||||
max_concurrency: 1,
|
max_concurrency: 1,
|
||||||
last_eval: null,
|
last_eval: null,
|
||||||
documentFilter: [],
|
documentFilter: [],
|
||||||
|
awaiting_decision: false,
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const [editableQuestions, setEditableQuestions] = useState<string[]>([]);
|
||||||
|
const [newQuestion, setNewQuestion] = useState("");
|
||||||
|
const [submitting, setSubmitting] = useState(false);
|
||||||
|
|
||||||
|
// Sync editable questions when state changes
|
||||||
|
useEffect(() => {
|
||||||
|
if (state.awaiting_decision && state.context.sub_questions) {
|
||||||
|
setEditableQuestions([...state.context.sub_questions]);
|
||||||
|
}
|
||||||
|
}, [state.awaiting_decision, state.context.sub_questions]);
|
||||||
|
|
||||||
const handleDocumentFilterChange = (ids: string[]) => {
|
const handleDocumentFilterChange = (ids: string[]) => {
|
||||||
setState({ ...state, documentFilter: ids });
|
setState({ ...state, documentFilter: ids });
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const handleRemoveQuestion = (index: number) => {
|
||||||
|
setEditableQuestions(editableQuestions.filter((_, i) => i !== index));
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleAddQuestion = () => {
|
||||||
|
if (newQuestion.trim()) {
|
||||||
|
setEditableQuestions([...editableQuestions, newQuestion.trim()]);
|
||||||
|
setNewQuestion("");
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleDecision = async (action: "search" | "synthesize") => {
|
||||||
|
setSubmitting(true);
|
||||||
|
try {
|
||||||
|
const response = await fetch(
|
||||||
|
`${process.env.NEXT_PUBLIC_API_URL || "http://localhost:8000"}/v1/research/decide`,
|
||||||
|
{
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({
|
||||||
|
thread_id: state.context.original_question, // Use as identifier
|
||||||
|
action,
|
||||||
|
questions: editableQuestions,
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
);
|
||||||
|
if (response.ok) {
|
||||||
|
setState({ ...state, awaiting_decision: false });
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error("Failed to send decision:", error);
|
||||||
|
} finally {
|
||||||
|
setSubmitting(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<style>{`
|
<style>{`
|
||||||
|
|
@ -141,13 +191,162 @@ function AgentContent() {
|
||||||
</p>
|
</p>
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
<div style={{ marginBottom: "1rem" }}>
|
{/* Document filter - hidden when research is running */}
|
||||||
<DocumentSelector
|
{!running && (
|
||||||
selected={state.documentFilter || []}
|
<div style={{ marginBottom: "1rem" }}>
|
||||||
onChange={handleDocumentFilterChange}
|
<DocumentSelector
|
||||||
disabled={running}
|
selected={state.documentFilter || []}
|
||||||
/>
|
onChange={handleDocumentFilterChange}
|
||||||
</div>
|
/>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Decision UI when awaiting human input - hidden when report exists, submitting, or not running */}
|
||||||
|
{state.awaiting_decision && !submitting && !state.result && running && (
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
marginBottom: "1rem",
|
||||||
|
background: "#f0f9ff",
|
||||||
|
border: "2px solid #0ea5e9",
|
||||||
|
borderRadius: "8px",
|
||||||
|
padding: "1rem",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
fontWeight: "bold",
|
||||||
|
color: "#0369a1",
|
||||||
|
marginBottom: "0.75rem",
|
||||||
|
fontSize: "1rem",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Research Decision Point
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div style={{ fontSize: "0.85rem", color: "#64748b", marginBottom: "0.75rem" }}>
|
||||||
|
{state.context.qa_responses?.length || 0} answers collected | Iteration {state.iterations || 0}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Questions list */}
|
||||||
|
<div style={{ marginBottom: "0.75rem" }}>
|
||||||
|
<div style={{ fontSize: "0.8rem", color: "#475569", marginBottom: "0.5rem" }}>
|
||||||
|
Pending Questions ({editableQuestions.length}):
|
||||||
|
</div>
|
||||||
|
{editableQuestions.map((q, idx) => (
|
||||||
|
<div
|
||||||
|
key={idx}
|
||||||
|
style={{
|
||||||
|
display: "flex",
|
||||||
|
alignItems: "center",
|
||||||
|
gap: "0.5rem",
|
||||||
|
padding: "0.375rem 0.5rem",
|
||||||
|
background: "white",
|
||||||
|
borderRadius: "4px",
|
||||||
|
marginBottom: "0.25rem",
|
||||||
|
fontSize: "0.85rem",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<span style={{ flex: 1 }}>{q}</span>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => handleRemoveQuestion(idx)}
|
||||||
|
disabled={submitting}
|
||||||
|
style={{
|
||||||
|
background: "#ef4444",
|
||||||
|
color: "white",
|
||||||
|
border: "none",
|
||||||
|
borderRadius: "4px",
|
||||||
|
padding: "0.25rem 0.5rem",
|
||||||
|
cursor: submitting ? "not-allowed" : "pointer",
|
||||||
|
fontSize: "0.75rem",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Remove
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Add question input */}
|
||||||
|
<div style={{ display: "flex", gap: "0.5rem", marginBottom: "1rem" }}>
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
value={newQuestion}
|
||||||
|
onChange={(e) => setNewQuestion(e.target.value)}
|
||||||
|
placeholder="Add a new question..."
|
||||||
|
disabled={submitting}
|
||||||
|
style={{
|
||||||
|
flex: 1,
|
||||||
|
padding: "0.5rem",
|
||||||
|
border: "1px solid #cbd5e1",
|
||||||
|
borderRadius: "4px",
|
||||||
|
fontSize: "0.85rem",
|
||||||
|
}}
|
||||||
|
onKeyDown={(e) => {
|
||||||
|
if (e.key === "Enter") {
|
||||||
|
handleAddQuestion();
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={handleAddQuestion}
|
||||||
|
disabled={submitting}
|
||||||
|
style={{
|
||||||
|
background: "#22c55e",
|
||||||
|
color: "white",
|
||||||
|
border: "none",
|
||||||
|
borderRadius: "4px",
|
||||||
|
padding: "0.5rem 1rem",
|
||||||
|
cursor: submitting ? "not-allowed" : "pointer",
|
||||||
|
fontSize: "0.85rem",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Add
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Action buttons */}
|
||||||
|
<div style={{ display: "flex", gap: "0.5rem" }}>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => handleDecision("search")}
|
||||||
|
disabled={editableQuestions.length === 0 || submitting}
|
||||||
|
style={{
|
||||||
|
flex: 1,
|
||||||
|
background: editableQuestions.length === 0 || submitting ? "#94a3b8" : "#0ea5e9",
|
||||||
|
color: "white",
|
||||||
|
border: "none",
|
||||||
|
borderRadius: "4px",
|
||||||
|
padding: "0.75rem",
|
||||||
|
cursor: editableQuestions.length === 0 || submitting ? "not-allowed" : "pointer",
|
||||||
|
fontWeight: "bold",
|
||||||
|
fontSize: "0.9rem",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{submitting ? "Submitting..." : `Search (${editableQuestions.length})`}
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => handleDecision("synthesize")}
|
||||||
|
disabled={submitting || (state.context.qa_responses?.length || 0) === 0}
|
||||||
|
style={{
|
||||||
|
flex: 1,
|
||||||
|
background: submitting || (state.context.qa_responses?.length || 0) === 0 ? "#94a3b8" : "#8b5cf6",
|
||||||
|
color: "white",
|
||||||
|
border: "none",
|
||||||
|
borderRadius: "4px",
|
||||||
|
padding: "0.75rem",
|
||||||
|
cursor: submitting || (state.context.qa_responses?.length || 0) === 0 ? "not-allowed" : "pointer",
|
||||||
|
fontWeight: "bold",
|
||||||
|
fontSize: "0.9rem",
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
Generate Report
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
<StateDisplay state={state} />
|
<StateDisplay state={state} />
|
||||||
</div>
|
</div>
|
||||||
|
|
|
||||||
|
|
@ -192,8 +192,8 @@ export default function StateDisplay({ state }: StateDisplayProps) {
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Research Progress - only show when research has started */}
|
{/* Research Progress - only show when research is in progress (not when complete) */}
|
||||||
{(state.iterations > 0 || (state.current_activity && !state.result)) && (
|
{(state.iterations > 0 || state.current_activity) && !state.result && (
|
||||||
<div
|
<div
|
||||||
style={{
|
style={{
|
||||||
background: "white",
|
background: "white",
|
||||||
|
|
@ -202,8 +202,8 @@ export default function StateDisplay({ state }: StateDisplayProps) {
|
||||||
boxShadow: "0 1px 3px rgba(0,0,0,0.1)",
|
boxShadow: "0 1px 3px rgba(0,0,0,0.1)",
|
||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
{/* Current Activity - hide when complete */}
|
{/* Current Activity */}
|
||||||
{state.current_activity && !state.result && (
|
{state.current_activity && (
|
||||||
<div
|
<div
|
||||||
style={{
|
style={{
|
||||||
padding: "0.75rem",
|
padding: "0.75rem",
|
||||||
|
|
@ -363,9 +363,8 @@ export default function StateDisplay({ state }: StateDisplayProps) {
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Sub-Questions and QA Responses */}
|
{/* Answers */}
|
||||||
{(state.context.sub_questions.length > 0 ||
|
{state.context.qa_responses.length > 0 && (
|
||||||
state.context.qa_responses.length > 0) && (
|
|
||||||
<div
|
<div
|
||||||
style={{
|
style={{
|
||||||
background: "white",
|
background: "white",
|
||||||
|
|
@ -392,10 +391,7 @@ export default function StateDisplay({ state }: StateDisplayProps) {
|
||||||
color: "#2d3748",
|
color: "#2d3748",
|
||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
<span>
|
<span>Answers ({state.context.qa_responses.length})</span>
|
||||||
Sub-Questions ({state.context.sub_questions.length}) • Answers (
|
|
||||||
{state.context.qa_responses.length})
|
|
||||||
</span>
|
|
||||||
<span>{expandedSections.questions ? "▼" : "▶"}</span>
|
<span>{expandedSections.questions ? "▼" : "▶"}</span>
|
||||||
</button>
|
</button>
|
||||||
{expandedSections.questions && (
|
{expandedSections.questions && (
|
||||||
|
|
@ -408,38 +404,6 @@ export default function StateDisplay({ state }: StateDisplayProps) {
|
||||||
borderRadius: "0 0 4px 4px",
|
borderRadius: "0 0 4px 4px",
|
||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
{/* Show pending sub_questions */}
|
|
||||||
{state.context.sub_questions.map((question, idx) => (
|
|
||||||
<div
|
|
||||||
key={`pending-${idx}`}
|
|
||||||
style={{
|
|
||||||
marginBottom: "0.5rem",
|
|
||||||
background: "white",
|
|
||||||
borderRadius: "4px",
|
|
||||||
border: "1px solid #e2e8f0",
|
|
||||||
padding: "0.75rem",
|
|
||||||
display: "flex",
|
|
||||||
gap: "0.75rem",
|
|
||||||
alignItems: "center",
|
|
||||||
}}
|
|
||||||
>
|
|
||||||
<div
|
|
||||||
style={{
|
|
||||||
fontSize: "1.25rem",
|
|
||||||
color: "#a0aec0",
|
|
||||||
flexShrink: 0,
|
|
||||||
}}
|
|
||||||
>
|
|
||||||
⏳
|
|
||||||
</div>
|
|
||||||
<div
|
|
||||||
style={{ flex: 1, fontSize: "0.875rem", color: "#4a5568" }}
|
|
||||||
>
|
|
||||||
<Markdown content={question} />
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
))}
|
|
||||||
|
|
||||||
{/* Show all qa_responses (each has query + answer) */}
|
{/* Show all qa_responses (each has query + answer) */}
|
||||||
{state.context.qa_responses.map((qaResponse, idx) => {
|
{state.context.qa_responses.map((qaResponse, idx) => {
|
||||||
const questionId = `q-${idx}`;
|
const questionId = `q-${idx}`;
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,5 @@
|
||||||
|
import asyncio
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from pydantic_ai.models.test import TestModel
|
from pydantic_ai.models.test import TestModel
|
||||||
|
|
||||||
|
|
@ -5,7 +7,7 @@ from haiku.rag.client import HaikuRAG
|
||||||
from haiku.rag.graph.agui.stream import stream_graph
|
from haiku.rag.graph.agui.stream import stream_graph
|
||||||
from haiku.rag.graph.research.dependencies import ResearchContext
|
from haiku.rag.graph.research.dependencies import ResearchContext
|
||||||
from haiku.rag.graph.research.graph import build_research_graph
|
from haiku.rag.graph.research.graph import build_research_graph
|
||||||
from haiku.rag.graph.research.state import ResearchDeps, ResearchState
|
from haiku.rag.graph.research.state import HumanDecision, ResearchDeps, ResearchState
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
|
|
@ -61,3 +63,83 @@ async def test_graph_end_to_end_with_test_model(monkeypatch, temp_db_path):
|
||||||
assert "STEP_STARTED" in event_types
|
assert "STEP_STARTED" in event_types
|
||||||
|
|
||||||
client.close()
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_interactive_graph_with_human_decision(monkeypatch, temp_db_path):
|
||||||
|
"""Test interactive research graph pauses and resumes with human decisions."""
|
||||||
|
|
||||||
|
# Mock get_model to return TestModel
|
||||||
|
def test_model_factory(_provider, _model, _config=None):
|
||||||
|
return TestModel()
|
||||||
|
|
||||||
|
monkeypatch.setattr("haiku.rag.utils.get_model", test_model_factory)
|
||||||
|
monkeypatch.setattr("haiku.rag.graph.research.graph.get_model", test_model_factory)
|
||||||
|
|
||||||
|
# Build interactive graph
|
||||||
|
graph = build_research_graph(interactive=True)
|
||||||
|
|
||||||
|
state = ResearchState(
|
||||||
|
context=ResearchContext(original_question="What is haiku.rag?"),
|
||||||
|
max_iterations=1,
|
||||||
|
confidence_threshold=0.5,
|
||||||
|
max_concurrency=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create human input queue
|
||||||
|
human_input_queue: asyncio.Queue[HumanDecision] = asyncio.Queue()
|
||||||
|
|
||||||
|
client = HaikuRAG(temp_db_path, create=True)
|
||||||
|
deps = ResearchDeps(
|
||||||
|
client=client,
|
||||||
|
human_input_queue=human_input_queue,
|
||||||
|
interactive=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
events = []
|
||||||
|
tool_call_received = asyncio.Event()
|
||||||
|
result = None
|
||||||
|
|
||||||
|
async def run_graph():
|
||||||
|
nonlocal result
|
||||||
|
async for event in stream_graph(graph, state, deps):
|
||||||
|
events.append(event)
|
||||||
|
if event["type"] == "TOOL_CALL_START":
|
||||||
|
tool_name = event.get("toolCallName")
|
||||||
|
if tool_name == "human_decision":
|
||||||
|
tool_call_received.set()
|
||||||
|
elif event["type"] == "RUN_FINISHED":
|
||||||
|
result = event["result"]
|
||||||
|
elif event["type"] == "RUN_ERROR":
|
||||||
|
pytest.fail(f"Graph execution failed: {event['message']}")
|
||||||
|
|
||||||
|
async def send_decisions():
|
||||||
|
# Wait for first tool call (after planning)
|
||||||
|
await asyncio.wait_for(tool_call_received.wait(), timeout=30)
|
||||||
|
tool_call_received.clear()
|
||||||
|
|
||||||
|
# Send search decision
|
||||||
|
await human_input_queue.put(HumanDecision(action="search"))
|
||||||
|
|
||||||
|
# Wait for second tool call (after search cycle)
|
||||||
|
await asyncio.wait_for(tool_call_received.wait(), timeout=30)
|
||||||
|
|
||||||
|
# Send synthesize decision
|
||||||
|
await human_input_queue.put(HumanDecision(action="synthesize"))
|
||||||
|
|
||||||
|
# Run graph and decision sender concurrently
|
||||||
|
await asyncio.gather(run_graph(), send_decisions())
|
||||||
|
|
||||||
|
# Verify result
|
||||||
|
assert result is not None, (
|
||||||
|
f"No result. Events collected: {[e['type'] for e in events]}"
|
||||||
|
)
|
||||||
|
assert isinstance(result, dict)
|
||||||
|
assert "title" in result
|
||||||
|
|
||||||
|
# Verify human_decision tool calls were emitted
|
||||||
|
event_types = [e["type"] for e in events]
|
||||||
|
assert "TOOL_CALL_START" in event_types
|
||||||
|
assert "TOOL_CALL_END" in event_types
|
||||||
|
|
||||||
|
client.close()
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue