Operator Events

Operator events let you subscribe to lifecycle events from your Outpost deployment, including delivery failures, destination disabling, retry exhaustion, and subscription changes.

Configuration

Enable operator events by specifying topics and configuring a sink.

Topics

OPERATOR_EVENTS_TOPICS=alert.destination.disabled,alert.attempt.exhausted_retries

Available topics:

Sinks

OPERATOR_EVENTS_HTTP_URL=https://example.com/outpost-events
OPERATOR_EVENTS_HTTP_SIGNING_SECRET=your-secret

import hashlib
import hmac

expected = hmac.new(signing_secret.encode(), body, hashlib.sha256).hexdigest()
assert signature_header == f"v0={expected}"

OPERATOR_EVENTS_AWS_SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789/outpost-events
OPERATOR_EVENTS_AWS_SQS_ACCESS_KEY_ID=AKIA...
OPERATOR_EVENTS_AWS_SQS_SECRET_ACCESS_KEY=...
OPERATOR_EVENTS_AWS_SQS_REGION=us-east-1
OPERATOR_EVENTS_AWS_SQS_ENDPOINT=          # optional, for local dev

OPERATOR_EVENTS_GCP_PUBSUB_PROJECT_ID=my-project
OPERATOR_EVENTS_GCP_PUBSUB_TOPIC_ID=outpost-events
OPERATOR_EVENTS_GCP_PUBSUB_CREDENTIALS={"type":"service_account",...}

OPERATOR_EVENTS_RABBITMQ_SERVER_URL=amqp://guest:guest@localhost:5672
OPERATOR_EVENTS_RABBITMQ_EXCHANGE=outpost-events

Event Envelope

All operator events share this envelope:

{
  "id": "unique-event-id",
  "topic": "alert.destination.consecutive_failure",
  "time": "2025-06-01T12:00:00Z",
  "deployment_id": "my-deployment",
  "tenant_id": "tenant_123",
  "data": {}
}

Event Payloads

alert.destination.consecutive_failure

Emitted when a destination reaches 50%, 70%, 90%, or 100% of ALERT_CONSECUTIVE_FAILURE_COUNT.

{
  "tenant_id": "tenant_123",
  "event": {},
  "attempt": {},
  "destination": {
    "id": "des_456",
    "tenant_id": "tenant_123",
    "type": "webhook",
    "topics": ["order.created"],
    "disabled_at": null
  },
  "consecutive_failures": {
    "current": 50,
    "max": 100,
    "threshold": 50
  }
}

alert.destination.disabled

Emitted when a destination is auto-disabled after reaching the 100% threshold (only when ALERT_AUTO_DISABLE_DESTINATION=true).

{
  "tenant_id": "tenant_123",
  "destination": {
    "id": "des_456",
    "tenant_id": "tenant_123",
    "type": "webhook",
    "topics": ["order.created"],
    "disabled_at": "2025-06-01T12:00:00Z"
  },
  "disabled_at": "2025-06-01T12:00:00Z",
  "reason": "consecutive_failure",
  "event": {},
  "attempt": {}
}

alert.attempt.exhausted_retries

Emitted when a delivery exhausts all retry attempts. At most one alert per destination within ALERT_EXHAUSTED_RETRIES_WINDOW_SECONDS; the alert payload carries the first exhausted event in the window. Set the window to 0 to alert on every exhaustion.

{
  "tenant_id": "tenant_123",
  "event": {},
  "attempt": {},
  "destination": {
    "id": "des_456",
    "tenant_id": "tenant_123",
    "type": "webhook",
    "topics": ["order.created"],
    "disabled_at": null
  }
}

attempt.success / attempt.failed

Emitted once per delivery attempt, keyed by outcome. The two topics share one payload shape; attempt.status carries the outcome. attempt.failed fires on every failed attempt including retries — use attempt.attempt_number and event.eligible_for_retry to distinguish retryable failures from final ones.

{
  "tenant_id": "tenant_123",
  "event": {
    "id": "evt_789",
    "tenant_id": "tenant_123",
    "destination_id": "des_456",
    "matched_destination_ids": ["des_456"],
    "topic": "order.created",
    "eligible_for_retry": true,
    "time": "2025-06-01T12:00:00Z",
    "metadata": {},
    "data": {
      "order_id": "ord_123"
    }
  },
  "attempt": {
    "id": "att_abc",
    "tenant_id": "tenant_123",
    "event_id": "evt_789",
    "destination_id": "des_456",
    "destination_type": "webhook",
    "attempt_number": 2,
    "manual": false,
    "status": "failed",
    "time": "2025-06-01T12:00:05Z",
    "code": "500",
    "latency_ms": 1502,
    "response_data": {
      "status": 500,
      "body": "internal server error"
    }
  },
  "destination": {
    "id": "des_456",
    "tenant_id": "tenant_123",
    "type": "webhook",
    "topics": ["order.created"],
    "disabled_at": null
  }
}

An attempt.success payload is identical except attempt.status is "success" and code/response_data carry the destination's response (for a webhook, the HTTP status and body). For network-level failures, code is an error class such as connection_refused or dns_error instead of an HTTP status.

The alert payloads above carry the same event and attempt objects, elided for brevity.

tenant.subscription.updated

Emitted when destination changes affect tenant-level subscribed topics or destination count.

{
  "tenant_id": "tenant_123",
  "topics": ["order.created", "order.updated"],
  "previous_topics": ["order.created"],
  "destinations_count": 3,
  "previous_destinations_count": 2
}

Delivery Guarantees

alert.* and attempt.* topics are delivered with an at-least-once guarantee. For other topics (e.g. tenant.subscription.updated), delivery is on a best-effort basis with up to 3 attempts. Consumers should deduplicate using the event id.

Related Configuration