Publish from SQS
This guide provides information on using SQS to publish events to Outpost.
Message Structure
SQS messages should have the same payload structure as the Publish API endpoint.
{
"tenant_id": "<TENANT_ID>",
"destination_id": "<DESTINATION_ID>", // Optional. Provide a way of routing events to a specific destination
"topic": "topic.name", // Topic defined in TOPICS environment variable
"eligible_for_retry": true | false, // Should event delivery be retried? Default is true.
"metadata": Payload, // can be any JSON payload,
"data": Payload // can be any JSON payload
}
Failed messages
When Outpost fails to process a message, it makes the message visible again and SQS redelivers it immediately, with no backoff and no limit. This includes messages that can never succeed, such as invalid JSON, data that is not a JSON object, or a missing topic when TOPICS is set.
Dead-lettering and redelivery limits are configured on your queue, not in Outpost. Set a redrive policy with a dead-letter queue and maxReceiveCount. Outpost resets the visibility timeout on failure, so the queue's visibility timeout does not delay redeliveries.
PUBLISH_MAX_REDELIVERIES sets a limit on the Outpost side instead: once a message has been redelivered that many times, Outpost deletes it. 0 deletes a message on its first failure. To keep those messages, set maxReceiveCount no higher than PUBLISH_MAX_REDELIVERIES so your dead-letter queue receives them first.
Configuration
Provide Outpost with connection and routing information for your SQS instance used for publishing events.
Environment Variables
PUBLISH_AWS_SQS_REGION="<REGION>"
PUBLISH_AWS_SQS_QUEUE="<QUEUE_NAME>"
PUBLISH_AWS_SQS_ACCESS_KEY_ID="<KEY_ID>"
PUBLISH_AWS_SQS_SECRET_ACCESS_KEY="<SECRET>"
Optionally, you can provide the PUBLISH_AWS_SQS_ENDPOINT environment variable to specify the endpoint URL for the SQS service. This can be useful for local development or when using a non-standard SQS endpoint.
Example
PUBLISH_AWS_SQS_REGION="eu-north-1"
PUBLISH_AWS_SQS_QUEUE="outpost-pub-queue"
PUBLISH_AWS_SQS_ACCESS_KEY_ID="REDACTED"
PUBLISH_AWS_SQS_SECRET_ACCESS_KEY="REDACTED"
YAML
publishmq:
aws_sqs:
region: <AWS_REGION>
queue: <QUEUE_NAME>
access_key_id: <KEY_ID>
secret_access_key: <SECRET>
# Optional.
# Useful for local development or when using a non-standard SQS endpoint.
endpoint: <URL>
Example
publishmq:
aws_sqs:
region: eu-north-1
queue: outpost-pub-queue
access_key_id: REDACTED
secret_access_key: REDACTED
Required Permissions
The following permissions are required for the provided access key:
sqs:ChangeMessageVisibilitysqs:DeleteMessagesqs:GetQueueUrlsqs:ReceiveMessage
Health
In a self-hosted deployment, if the publish consumer fails (it can't subscribe, or 10 receives fail in a row, about 70 seconds), /healthz reports it failed (HTTP 503) until the process restarts. You can turn on restarts instead by adding publishmq to SUPERVISOR_: Outpost then restarts the consumer with a backoff and reports it degraded while it retries. See Supervisor for the limits after which it reports failed.