203 lines
7.2 KiB
Markdown
203 lines
7.2 KiB
Markdown
---
|
|
name: debug-ingestion
|
|
description: Debug haiku.rag ingestion in Logfire. Use when asked to look at Logfire for ingestion, find failed or dead ingestion jobs, investigate retries or circuit-breaker events, trace a document through convert/chunk/embed/store, find which docling-serve instance served a request, spot slow conversions, or tell concurrent ingesters apart. Drives the Logfire MCP against the `haiku-ingester` service.
|
|
---
|
|
|
|
# Debug ingestion in Logfire
|
|
|
|
The ingester ships spans to Logfire under `service_name = 'haiku-ingester'` (or a
|
|
custom `OTEL_SERVICE_NAME` if set per process). This skill finds failing jobs,
|
|
traces a document through the pipeline, and pinpoints the docling-serve instance
|
|
that served a request. Read-only.
|
|
|
|
## How to query
|
|
|
|
1. Confirm the current schema with `mcp__logfire__query_schema_reference` (spans
|
|
and logs share the `records` table).
|
|
2. Run SQL with `mcp__logfire__query_run` (`query` + `project: "haiku"` +
|
|
`start_timestamp`/`end_timestamp`, max 14 days). The remote MCP is
|
|
org-scoped, so `project` is required; the ingester ships to project `haiku`.
|
|
The same SQL works pasted into Logfire's Explore UI.
|
|
3. Read span attributes as JSON: `attributes->>'key'`, cast when needed
|
|
(`(attributes->>'attempt')::int`).
|
|
4. Hand back a clickable trace with
|
|
`mcp__logfire__project_logfire_link(trace_id, project="haiku")`.
|
|
5. For recent exceptions tied to a file,
|
|
`mcp__logfire__query_find_exceptions_in_file` accepts
|
|
`client/documents.py`, `ingester/workers/pool.py`, or
|
|
`ingester/pollers/base.py`.
|
|
|
|
Adjust `service_name` if the operator set `OTEL_SERVICE_NAME` (e.g. per tenant).
|
|
Interactive `haiku-rag` ingests emit the same `document.*` spans under the CLI's
|
|
service name (or `unknown_service` for older runs), not `haiku-ingester`.
|
|
|
|
## Vocabulary
|
|
|
|
Span tree (all scope `haiku.rag`), each level nests under the one above and
|
|
shares its `trace_id`:
|
|
|
|
- Poller: `ingester.poller.sweep` | `ingester.poller.dry_run` |
|
|
`ingester.poller.watch_event`.
|
|
- `attributes->>'source_id'`; watch adds `change`, `uri`.
|
|
- sweep sets `skipped`, `skip_reason` (`pending_work` / `circuit_open`),
|
|
`upsert`, `delete`, `unchanged`, `consecutive_failures`. A failed sweep
|
|
records the exception on the span (`exception_type` / `exception_message`).
|
|
- Job: `ingester.job` — `attributes->>'source_id'`, `->>'uri'`, `->>'op'`
|
|
(`UPSERT`/`DELETE`), `(->>'attempt')::int`. `is_exception=true` marks a job
|
|
that raised.
|
|
- Document pipeline: `document.fetch` (`bytes`, `content_hash`),
|
|
`document.convert`, `document.chunk` (`chunks_created`), `document.embed`,
|
|
`document.store` (`op` `create`/`update`, `document_id`).
|
|
- docling-serve: `docling_serve.request` — `attributes->>'name'` (operation),
|
|
`->>'url'` (instance), `(->>'attempt')::int`. A retry emits a new span with a
|
|
different `url`, so failover shows as sibling spans.
|
|
|
|
Failures in Logfire are span-level: sweep and job exceptions sit on the span
|
|
(`is_exception`, `exception_type`, `exception_message`, `level >= 17`). A job that
|
|
raised `PermanentError` was dead-lettered on that attempt; `attempt >= 2` on an
|
|
`ingester.job` span is a retry. The worker circuit breaker opening emits a
|
|
dedicated event `span_name = 'ingester.worker breaker opened'` (attributes
|
|
`source_id`, `threshold`, `cooldown_s`), fired once per closed->open transition.
|
|
The ingester's per-job dead/reschedule narration stays on stderr and the queue
|
|
(`GET /jobs?status=dead` on the control-plane API), not Logfire.
|
|
|
|
## Canned queries
|
|
|
|
Job outcomes by source:
|
|
|
|
```sql
|
|
SELECT attributes->>'source_id' AS source_id,
|
|
attributes->>'op' AS op,
|
|
is_exception,
|
|
count(*) AS n
|
|
FROM records
|
|
WHERE service_name='haiku-ingester' AND span_name='ingester.job'
|
|
GROUP BY 1,2,3
|
|
ORDER BY n DESC;
|
|
```
|
|
|
|
Failed jobs with the exception and trace to drill in:
|
|
|
|
```sql
|
|
SELECT attributes->>'uri' AS uri,
|
|
(attributes->>'attempt')::int AS attempt,
|
|
exception_type, exception_message, trace_id
|
|
FROM records
|
|
WHERE service_name='haiku-ingester' AND span_name='ingester.job'
|
|
AND is_exception=true
|
|
ORDER BY start_timestamp DESC
|
|
LIMIT 50;
|
|
```
|
|
|
|
Retried and dead-lettered jobs (a `PermanentError` was dead-lettered on that
|
|
attempt; `TransientError` at a high `attempt` was retried):
|
|
|
|
```sql
|
|
SELECT attributes->>'uri' AS uri,
|
|
(attributes->>'attempt')::int AS attempt,
|
|
attributes->>'op' AS op,
|
|
exception_type, trace_id
|
|
FROM records
|
|
WHERE service_name='haiku-ingester' AND span_name='ingester.job'
|
|
AND (is_exception=true OR (attributes->>'attempt')::int > 1)
|
|
ORDER BY start_timestamp DESC
|
|
LIMIT 50;
|
|
```
|
|
|
|
Trace one document end-to-end (take `trace_id` from a job above):
|
|
|
|
```sql
|
|
SELECT span_name, duration, is_exception,
|
|
attributes->>'url' AS docling_url,
|
|
attributes->>'op' AS op
|
|
FROM records
|
|
WHERE trace_id='<TRACE_ID>'
|
|
ORDER BY start_timestamp;
|
|
```
|
|
|
|
Worker circuit-breaker trips (a source paused after consecutive transient
|
|
failures):
|
|
|
|
```sql
|
|
SELECT start_timestamp,
|
|
attributes->>'source_id' AS source_id,
|
|
(attributes->>'cooldown_s')::float AS cooldown_s
|
|
FROM records
|
|
WHERE service_name='haiku-ingester'
|
|
AND span_name='ingester.worker breaker opened'
|
|
ORDER BY start_timestamp DESC
|
|
LIMIT 50;
|
|
```
|
|
|
|
docling-serve instance health and failover:
|
|
|
|
```sql
|
|
SELECT attributes->>'url' AS instance,
|
|
(attributes->>'attempt')::int AS attempt,
|
|
is_exception,
|
|
count(*) AS n
|
|
FROM records
|
|
WHERE service_name='haiku-ingester' AND span_name='docling_serve.request'
|
|
GROUP BY 1,2,3
|
|
ORDER BY n DESC;
|
|
```
|
|
|
|
Slowest pipeline stages:
|
|
|
|
```sql
|
|
SELECT span_name,
|
|
attributes->>'uri' AS uri,
|
|
duration, trace_id
|
|
FROM records
|
|
WHERE service_name='haiku-ingester'
|
|
AND span_name IN ('document.convert','docling_serve.request','document.embed','document.chunk')
|
|
ORDER BY duration DESC
|
|
LIMIT 20;
|
|
```
|
|
|
|
Per-source sweep summary:
|
|
|
|
```sql
|
|
SELECT attributes->>'source_id' AS source_id,
|
|
sum((attributes->>'upsert')::int) AS upserts,
|
|
sum((attributes->>'delete')::int) AS deletes,
|
|
sum((attributes->>'unchanged')::int) AS unchanged,
|
|
max(attributes->>'skip_reason') AS last_skip_reason
|
|
FROM records
|
|
WHERE service_name='haiku-ingester' AND span_name='ingester.poller.sweep'
|
|
GROUP BY 1;
|
|
```
|
|
|
|
Tell concurrent ingesters apart (set distinct `OTEL_SERVICE_NAME` per process; a
|
|
single host still separates by `process_pid` / `service_instance_id`):
|
|
|
|
```sql
|
|
SELECT service_name, service_instance_id, process_pid, count(*) AS n
|
|
FROM records
|
|
WHERE span_name='ingester.job'
|
|
GROUP BY 1,2,3
|
|
ORDER BY n DESC;
|
|
```
|
|
|
|
## Workflow
|
|
|
|
1. Job outcomes by source shows where failures cluster.
|
|
2. Failed jobs surfaces the exception and each failing `trace_id`.
|
|
3. Trace one document end-to-end to see which stage failed and, for a docling
|
|
source, which `docling_url` served it (and whether it failed over).
|
|
4. Retried/dead-lettered jobs show what the queue kept struggling with; a
|
|
`ingester.worker breaker opened` event flags a source the pool paused. The
|
|
per-job dead/reschedule narration stays in the ingester console and the queue
|
|
API, not Logfire.
|
|
5. `project_logfire_link(trace_id)` for a document lets the user expand the
|
|
full tree.
|
|
|
|
## When a query returns nothing
|
|
|
|
Span names or attributes may have changed. Probe:
|
|
|
|
```sql
|
|
SELECT DISTINCT span_name
|
|
FROM records WHERE service_name='haiku-ingester'
|
|
ORDER BY 1;
|
|
```
|