Relay Observability

Relay tracks every message from publish to final delivery. The tracing system records timing, budget consumption, and error details for each delivery attempt. Aggregate metrics give you a snapshot of bus health, and the dead letter queue captures messages that could not be delivered.

Message Tracing

Every message published through Relay gets a trace ID. Each delivery to an endpoint creates a span — a complete record of that delivery attempt: when it was sent, when it arrived, when processing completed, and whether it succeeded.

Trace Span Fields

Looking Up a Trace

curl http://localhost:4242/api/relay/messages/01HX.../trace

The response includes every span in the trace chain, ordered by sentAt ascending:

{
  "traceId": "01HXABC123",
  "spans": [
    {
      "id": "01HXDEF456",
      "messageId": "01HXABC123",
      "traceId": "01HXABC123",
      "subject": "relay.agent.backend",
      "status": "delivered",
      "sentAt": "2025-02-26T12:00:00.000Z",
      "deliveredAt": "2025-02-26T12:00:00.050Z",
      "processedAt": "2025-02-26T12:00:00.200Z",
      "errorMessage": null,
      "metadata": null
    }
  ]
}

For messages that fan out to multiple endpoints, the trace contains one span per endpoint.

Trace Statuses

A healthy message moves from sent to delivered. The time between sentAt and deliveredAt is delivery latency. The time between deliveredAt and processedAt is processing latency, which includes the time the adapter (e.g., Claude Code) takes to handle the message.

Using MCP Tools

Agents can inspect traces without HTTP calls using the built-in MCP tools:

relay_get_trace     Get the full delivery trace for a message by ID
relay_get_metrics   Get aggregate delivery metrics for the bus

relay_get_trace accepts a messageId and returns the same trace data as the REST endpoint. Use this when an agent needs to verify a message was delivered before proceeding.

Delivery Metrics

Relay computes aggregate delivery metrics from the trace store using live SQL aggregates. These give you a summary of bus health without inspecting individual traces.

Fetching Metrics

curl http://localhost:4242/api/relay/trace/metrics

{
  "totalMessages": 1542,
  "deliveredCount": 1480,
  "failedCount": 12,
  "deadLetteredCount": 50,
  "avgDeliveryLatencyMs": 45.2,
  "p95DeliveryLatencyMs": null,
  "activeEndpoints": 8,
  "budgetRejections": {
    "hopLimit": 0,
    "ttlExpired": 0,
    "cycleDetected": 0,
    "budgetExhausted": 0
  }
}

Metrics Fields

Interpreting the Numbers

A healthy bus has a high delivered-to-total ratio and low dead letter counts.

• High deadLetteredCount relative to total — Check budget rejections. A spike in hopLimit rejections often indicates a message loop between two agents. TTL expirations may mean agents are too slow to process within the 1-hour window.

• Rising failedCount — Subscriber handlers are throwing errors. Check individual traces for the errorMessage field. The circuit breaker automatically stops delivering to endpoints with 5 consecutive failures.

• High avgDeliveryLatencyMs — A single slow endpoint can pull up the mean. Inspect traces filtered by endpoint to find the bottleneck.

The DorkOS client includes a Delivery Metrics Dashboard in the Relay panel. Access it from the Relay tab in the sidebar when Relay is enabled.

Debugging Failed Deliveries

curl http://localhost:4242/api/relay/dead-letters

curl http://localhost:4242/api/relay/messages/{messageId}/trace

SSE Stream for Real-Time Monitoring

Subscribe to the Relay SSE stream for live visibility:

curl -N http://localhost:4242/api/relay/stream?subject=%3E

The subject query parameter filters which messages appear. Use > (URL-encoded as %3E) to see all messages, or provide a pattern like relay.agent.* to filter by audience.

The stream emits four event types:

Next Steps