Relay Observability

When your agents talk to each other through Relay, most messages just arrive. When one doesn't, you need answers: did it fail, time out, or never reach anywhere at all? Relay keeps a full paper trail of every message, from the moment it's published to the moment it's delivered, so you can find out.

Haven't turned Relay on yet? Start with Relay Messaging to enable it and register your first adapter.

The Fastest Way to Check Delivery Health

Open the Relay tab in the DorkOS sidebar. The Delivery Metrics Dashboard there shows how many messages delivered, failed, or landed in the dead letter queue, no terminal required. For most people, that's as deep as you'll ever need to go.

Everything below this point is deep reference: raw API calls, JSON shapes, and internals for anyone debugging a stuck message by hand.

Reference: Tracing and Metrics

Message Tracing

Every message published through Relay gets a trace ID. Each delivery to an endpoint creates a span, a complete record of that one delivery attempt: when it was sent, when it arrived, when processing finished, and whether it succeeded.

Trace and message IDs are ULIDs (sortable, unique IDs that look like 01HXABC123).

Trace Span Fields

Looking Up a Trace

curl http://localhost:4242/api/relay/messages/01HX.../trace

The response includes every span in the trace chain, ordered by sentAt ascending:

{
  "traceId": "01HXABC123",
  "spans": [
    {
      "id": "01HXDEF456",
      "messageId": "01HXABC123",
      "traceId": "01HXABC123",
      "subject": "relay.agent.backend",
      "status": "delivered",
      "sentAt": "2025-02-26T12:00:00.000Z",
      "deliveredAt": "2025-02-26T12:00:00.050Z",
      "processedAt": "2025-02-26T12:00:00.200Z",
      "errorMessage": null,
      "metadata": null
    }
  ]
}

For messages that fan out to multiple endpoints, the trace contains one span per endpoint.

Trace Statuses

A healthy message moves from sent to delivered. The time between sentAt and deliveredAt is delivery latency. The time between deliveredAt and processedAt is processing latency, which includes the time the adapter (e.g., Claude Code) takes to handle the message.

Using MCP Tools

Agents can inspect traces without HTTP calls using the built-in MCP tools:

relay_get_trace     Get the full delivery trace for a message by ID
relay_get_metrics   Get aggregate delivery metrics for the bus

relay_get_trace accepts a messageId and returns the same trace data as the REST endpoint. Use this when an agent needs to verify a message was delivered before proceeding.

Delivery Metrics

Relay computes aggregate delivery metrics straight from the trace store, covering the last 24 hours by default. These give you a summary of bus health without inspecting individual traces.

Fetching Metrics

curl http://localhost:4242/api/relay/trace/metrics

{
  "totalMessages": 1542,
  "deliveredCount": 1480,
  "failedCount": 12,
  "deadLetteredCount": 50,
  "avgDeliveryLatencyMs": 45.2,
  "p50DeliveryLatencyMs": 28.6,
  "p95DeliveryLatencyMs": 190.5,
  "p99DeliveryLatencyMs": 420.1,
  "activeEndpoints": 8,
  "budgetRejections": {
    "hopLimit": 0,
    "ttlExpired": 0,
    "cycleDetected": 0,
    "budgetExhausted": 0
  }
}

Metrics Fields

Interpreting the Numbers

A healthy bus has a high delivered-to-total ratio and low dead letter counts.

• High deadLetteredCount relative to total: Check budget rejections. A spike in hopLimit rejections usually means two agents stuck replying to each other in a loop (it happens to the best of us). TTL expirations may mean agents are too slow to process within the 1-hour window.
• Rising failedCount: Subscriber handlers are throwing errors. Check individual traces for the errorMessage field. The circuit breaker (a safety switch that stops sending to an endpoint that keeps failing) automatically stops delivering to endpoints with 5 consecutive failures.
• High avgDeliveryLatencyMs: A single slow endpoint can pull up the mean. Compare it against p50DeliveryLatencyMs: an average far above the median means a few slow deliveries are skewing it. Inspect traces filtered by endpoint to find the bottleneck.

When a Message Goes Missing

Work through these steps in order. Each one rules out a different cause.

curl http://localhost:4242/api/relay/dead-letters

curl http://localhost:4242/api/relay/messages/{messageId}/trace

SSE Stream for Real-Time Monitoring

Subscribe to the Relay SSE stream for live visibility:

curl -N http://localhost:4242/api/relay/stream?subject=%3E

The subject query parameter filters which messages appear. Use > (URL-encoded as %3E) to see all messages, or provide a pattern like relay.agent.* to filter by audience.

The stream emits four event types:

Next Steps