# Relay Observability
Source: https://dorkos.ai/docs/guides/relay-observability

Find out what happened to a message between your agents, and what to do when delivery fails





# Relay Observability [#relay-observability]

When your agents talk to each other through Relay, most messages just arrive. When one doesn't, you need answers: did it fail, time out, or never reach anywhere at all? Relay keeps a full paper trail of every message, from the moment it's published to the moment it's delivered, so you can find out.

<Callout type="info">
  Multi-agent coordination through Relay is a newer part of DorkOS. It works, but we're still
  hardening it end to end, so treat delivery guarantees as "strong, not bulletproof" while it
  matures.
</Callout>

Haven't turned Relay on yet? Start with [Relay Messaging](/docs/guides/relay-messaging) to enable it and register your first adapter.

## The Fastest Way to Check Delivery Health [#the-fastest-way-to-check-delivery-health]

Open the **Relay** tab in the DorkOS sidebar. The Delivery Metrics Dashboard there shows how many messages delivered, failed, or landed in the dead letter queue, no terminal required. For most people, that's as deep as you'll ever need to go.

Everything below this point is deep reference: raw API calls, JSON shapes, and internals for anyone debugging a stuck message by hand.

## Reference: Tracing and Metrics [#reference-tracing-and-metrics]

### Message Tracing [#message-tracing]

Every message published through Relay gets a trace ID. Each delivery to an endpoint creates a **span**, a complete record of that one delivery attempt: when it was sent, when it arrived, when processing finished, and whether it succeeded.

Trace and message IDs are [ULIDs](/docs/glossary) (sortable, unique IDs that look like `01HXABC123`).

#### Trace Span Fields [#trace-span-fields]

<TypeTable
  type="{
  id: { type: 'string', description: 'ULID uniquely identifying this trace span' },
  messageId: { type: 'string', description: 'ULID of the message being delivered' },
  traceId: { type: 'string', description: 'ULID grouping all spans for a publish operation' },
  subject: { type: 'string', description: 'The target subject the message was published to' },
  status: { type: 'string', description: 'Delivery status: sent, delivered, failed, or timeout' },
  sentAt: { type: 'string', description: 'ISO 8601 timestamp when the message was published' },
  deliveredAt: {
    type: 'string | null',
    description: 'ISO 8601 timestamp when the message reached the endpoint',
  },
  processedAt: {
    type: 'string | null',
    description: 'ISO 8601 timestamp when the subscriber handler completed',
  },
  errorMessage: { type: 'string | null', description: 'Error message if delivery failed' },
  metadata: {
    type: 'string | null',
    description: 'JSON-encoded additional metadata about the delivery',
  },
}"
/>

#### Looking Up a Trace [#looking-up-a-trace]

```bash
curl http://localhost:4242/api/relay/messages/01HX.../trace
```

The response includes every span in the trace chain, ordered by `sentAt` ascending:

```json
{
  "traceId": "01HXABC123",
  "spans": [
    {
      "id": "01HXDEF456",
      "messageId": "01HXABC123",
      "traceId": "01HXABC123",
      "subject": "relay.agent.backend",
      "status": "delivered",
      "sentAt": "2025-02-26T12:00:00.000Z",
      "deliveredAt": "2025-02-26T12:00:00.050Z",
      "processedAt": "2025-02-26T12:00:00.200Z",
      "errorMessage": null,
      "metadata": null
    }
  ]
}
```

For messages that fan out to multiple endpoints, the trace contains one span per endpoint.

#### Trace Statuses [#trace-statuses]

| Status      | Meaning                                                                                                        |
| ----------- | -------------------------------------------------------------------------------------------------------------- |
| `sent`      | Message published, delivery in progress                                                                        |
| `delivered` | Message delivered and processed successfully                                                                   |
| `failed`    | Subscription handler threw an error                                                                            |
| `timeout`   | Message rejected (budget exceeded, access denied, TTL expired, no matching endpoints, or circuit breaker open) |

A healthy message moves from `sent` to `delivered`. The time between `sentAt` and `deliveredAt` is delivery latency. The time between `deliveredAt` and `processedAt` is processing latency, which includes the time the adapter (e.g., Claude Code) takes to handle the message.

#### Using MCP Tools [#using-mcp-tools]

Agents can inspect traces without HTTP calls using the built-in MCP tools:

```
relay_get_trace     Get the full delivery trace for a message by ID
relay_get_metrics   Get aggregate delivery metrics for the bus
```

`relay_get_trace` accepts a `messageId` and returns the same trace data as the REST endpoint. Use this when an agent needs to verify a message was delivered before proceeding.

### Delivery Metrics [#delivery-metrics]

Relay computes aggregate delivery metrics straight from the trace store, covering the last 24 hours by default. These give you a summary of bus health without inspecting individual traces.

#### Fetching Metrics [#fetching-metrics]

```bash
curl http://localhost:4242/api/relay/trace/metrics
```

```json
{
  "totalMessages": 1542,
  "deliveredCount": 1480,
  "failedCount": 12,
  "deadLetteredCount": 50,
  "avgDeliveryLatencyMs": 45.2,
  "p50DeliveryLatencyMs": 28.6,
  "p95DeliveryLatencyMs": 190.5,
  "p99DeliveryLatencyMs": 420.1,
  "activeEndpoints": 8,
  "budgetRejections": {
    "hopLimit": 0,
    "ttlExpired": 0,
    "cycleDetected": 0,
    "budgetExhausted": 0
  }
}
```

#### Metrics Fields [#metrics-fields]

<TypeTable
  type="{
  totalMessages: { type: 'number', description: 'Total messages tracked in the trace store' },
  deliveredCount: { type: 'number', description: 'Messages successfully delivered or processed' },
  failedCount: {
    type: 'number',
    description: 'Messages where the subscriber handler threw an error',
  },
  deadLetteredCount: {
    type: 'number',
    description: 'Messages rejected to the dead letter queue',
  },
  avgDeliveryLatencyMs: {
    type: 'number | null',
    description:
      'Average time from publish to endpoint delivery (milliseconds). Null when no deliveries are in the window.',
  },
  p50DeliveryLatencyMs: {
    type: 'number | null',
    description:
      'Median (50th percentile) delivery latency in milliseconds. Null when no deliveries are in the window.',
  },
  p95DeliveryLatencyMs: {
    type: 'number | null',
    description:
      '95th percentile delivery latency in milliseconds. Null when no deliveries are in the window.',
  },
  p99DeliveryLatencyMs: {
    type: 'number | null',
    description:
      '99th percentile delivery latency in milliseconds. Null when no deliveries are in the window.',
  },
  activeEndpoints: {
    type: 'number',
    description: 'Number of distinct endpoints that have received messages',
  },
  'budgetRejections.hopLimit': {
    type: 'number',
    description:
      'Messages rejected for exceeding the maximum hop count (how many agent-to-agent hops a message is allowed to make)',
  },
  'budgetRejections.ttlExpired': {
    type: 'number',
    description:
      'Messages rejected because their TTL (time to live: how long a message is allowed to wait before it expires) expired before delivery',
  },
  'budgetRejections.cycleDetected': {
    type: 'number',
    description: 'Messages rejected because the ancestor chain contained a cycle',
  },
  'budgetRejections.budgetExhausted': {
    type: 'number',
    description: 'Messages rejected because the call budget was depleted',
  },
}"
/>

#### Interpreting the Numbers [#interpreting-the-numbers]

A healthy bus has a high delivered-to-total ratio and low dead letter counts.

* **High `deadLetteredCount` relative to total**: Check budget rejections. A spike in `hopLimit` rejections usually means two agents stuck replying to each other in a loop (it happens to the best of us). TTL expirations may mean agents are too slow to process within the 1-hour window.
* **Rising `failedCount`**: Subscriber handlers are throwing errors. Check individual traces for the `errorMessage` field. The circuit breaker (a safety switch that stops sending to an endpoint that keeps failing) automatically stops delivering to endpoints with 5 consecutive failures.
* **High `avgDeliveryLatencyMs`**: A single slow endpoint can pull up the mean. Compare it against `p50DeliveryLatencyMs`: an average far above the median means a few slow deliveries are skewing it. Inspect traces filtered by endpoint to find the bottleneck.

<Callout type="info">
  Relay metrics cover the last 24 hours by default, not the full history of the database.
  `budgetRejections` counters return `0` in this endpoint. They're tracked at the RelayCore level
  but not yet aggregated into trace store metrics.
</Callout>

## When a Message Goes Missing [#when-a-message-goes-missing]

Work through these steps in order. Each one rules out a different cause.

<Steps>
  <Step>
    ### Check the dead letter queue [#check-the-dead-letter-queue]

    Dead letters are messages that could not be delivered. Fetch them:

    ```bash
    curl http://localhost:4242/api/relay/dead-letters
    ```

    Each dead letter includes the original envelope with subject, payload, and budget. Common reasons:

    * **No matching endpoints**: The subject has no registered endpoints. Verify with `GET /api/relay/endpoints`.
    * **Budget exceeded**: The message's hop count, TTL, or call budget was exhausted.
    * **Access denied**: The sender lacks permission to publish to the target subject. Check `access-rules.json`.
  </Step>

  <Step>
    ### Inspect the message trace [#inspect-the-message-trace]

    If the message was delivered but the handler failed, look up the trace:

    ```bash
    curl http://localhost:4242/api/relay/messages/{messageId}/trace
    ```

    Check `status` and `errorMessage` on each span. A `failed` status with an error message tells you exactly what went wrong. For Claude Code adapter failures, the error typically includes the Agent SDK error message.
  </Step>

  <Step>
    ### Check endpoint health [#check-endpoint-health]

    If deliveries are being rejected, the endpoint's circuit breaker may be open. The circuit breaker opens after 5 consecutive failures (configurable in `~/.dork/relay/config.json`). After a 30-second cooldown, it allows a single probe through. If the probe succeeds, normal delivery resumes.
  </Step>

  <Step>
    ### Check rate limits and backpressure [#check-rate-limits-and-backpressure]

    If a sender is publishing too quickly, messages are rejected by the rate limiter. The default allows 100 messages per 60-second window per sender. If an endpoint's mailbox (its inbox of unprocessed messages) has too many unprocessed messages, backpressure triggers at 80% capacity (warning) and 100% capacity (rejection).

    Both settings are tunable in `~/.dork/relay/config.json` and hot-reloaded without a restart.
  </Step>
</Steps>

## SSE Stream for Real-Time Monitoring [#sse-stream-for-real-time-monitoring]

Subscribe to the Relay SSE stream for live visibility:

```bash
curl -N http://localhost:4242/api/relay/stream?subject=%3E
```

The `subject` query parameter filters which messages appear. Use `>` (URL-encoded as `%3E`) to see all messages, or provide a pattern like `relay.agent.*` to filter by audience.

The stream emits four event types:

| Event                | Description                                                                      |
| -------------------- | -------------------------------------------------------------------------------- |
| `relay_connected`    | Initial connection confirmation with the filter pattern                          |
| `relay_message`      | A message envelope matching the subscription pattern                             |
| `relay_backpressure` | A backpressure signal from an endpoint approaching or exceeding mailbox capacity |
| `relay_signal`       | Other signals (dead letters, typing indicators, delivery receipts)               |

<Callout type="warn">
  The SSE stream is for debugging and monitoring. Long-lived SSE connections consume server
  resources, so close them when no longer needed. For production monitoring, use the REST API or the
  UI dashboard.
</Callout>

## Next Steps [#next-steps]

<Cards>
  <Card title="Relay Messaging" href="/docs/guides/relay-messaging">
    Enable Relay, send messages, and configure adapters.
  </Card>

  <Card title="Relay Concepts" href="/docs/concepts/relay">
    Relay's architecture, envelope format, and delivery pipeline in depth.
  </Card>

  <Card title="Agent Discovery" href="/docs/guides/agent-discovery">
    Discover and register agents that communicate through Relay using Mesh.
  </Card>
</Cards>