Publish from GCP Pub/Sub
This guide provides information on using GCP Pub/Sub to publish events to Outpost.
Message Structure
GCP Pub/Sub messages should have the same payload structure as the Publish API endpoint.
{
"tenant_id": "<TENANT_ID>",
"destination_id": "<DESTINATION_ID>", // Optional. Provide a way of routing events to a specific destination
"topic": "topic.name", // Topic defined in TOPICS environment variable
"eligible_for_retry": true | false, // Should event delivery be retried? Default is true.
"metadata": Payload, // can be any JSON payload,
"data": Payload // can be any JSON payload
}
GCP Pub/Sub Setup
Create a topic and create a pull subscription within GCP Pub/Sub.
Once you have these in place, you can configure Outpost to subscribe to receive events and then publish them to subscribed Outpost Destinations.
Failed messages
When Outpost fails to process a message, it nacks the message and Pub/Sub redelivers it without limit. This includes messages that can never succeed, such as invalid JSON, data that is not a JSON object, or a missing topic when TOPICS is set.
Dead-lettering, redelivery limits and backoff are configured on your subscription, not in Outpost. Set a dead-letter topic with max_delivery_attempts, and a retry policy with exponential backoff to space out redeliveries. Without a retry policy, Pub/Sub redelivers a nacked message immediately.
PUBLISH_MAX_REDELIVERIES sets a limit on the Outpost side instead: once a message has been redelivered that many times, Outpost acks it and it is lost. 0 acks a message on its first failure. To keep those messages, set max_delivery_attempts no higher than PUBLISH_MAX_REDELIVERIES so your dead-letter topic receives them first.
Configuration
Provide Outpost with credentials, topic, and subscription for your GCP Pub/Sub instance used for publishing events.
Environment Variables
PUBLISH_GCP_PUBSUB_PROJECT=""
PUBLISH_GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS=""
PUBLISH_GCP_PUBSUB_TOPIC=""
PUBLISH_GCP_PUBSUB_SUBSCRIPTION=""
Example
PUBLISH_GCP_PUBSUB_PROJECT="my-gcp-project"
PUBLISH_GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS="single_line_json_string"
PUBLISH_GCP_PUBSUB_TOPIC="outpost-publish"
PUBLISH_GCP_PUBSUB_SUBSCRIPTION="outpost-publish-sub"
YAML
publishmq:
gcp_pubsub:
project: <GCP_PROJECT>
service_account_credentials: <GCP_SERVICE_ACCOUNT_CREDENTIALS>
topic: <GCP_PUBSUB_TOPIC>
subscription: <GCP_PUBSUB_SUBSCRIPTION>
Example
publishmq:
gcp_pubsub:
project: "GCP_PUBSUB_PROJECT" # GCP project ID
service_account_credentials: "GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS" # Contents of service account credentials JSON file
topic: "outpost-publish" # Pub/Sub to read for published events
subscription: "outpost-publish-sub" # Pub/Sub subscription for published events
Health
In a self-hosted deployment, if the publish consumer fails (it can't subscribe, or 10 receives fail in a row, about 70 seconds), /healthz reports it failed (HTTP 503) until the process restarts. You can turn on restarts instead by adding publishmq to SUPERVISOR_: Outpost then restarts the consumer with a backoff and reports it degraded while it retries. See Supervisor for the limits after which it reports failed.