Publish and receive through SQS¶
By the end of this you will have an SNS topic, an SQS queue behind it with a dead-letter queue and a redrive policy, and a Go program that publishes an event to the topic and receives it from the queue through the adapter.
It needs Docker and the AWS CLI. Nothing here touches a real AWS account: LocalStack stands in for it, at the version the adapter's own suite runs against.
1. Start LocalStack¶
Point the CLI, and later the Go program, at it. LocalStack accepts any credentials:
$ export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1
$ export AWS_ENDPOINT_URL=http://127.0.0.1:4566
2. Create the topology¶
The adapter creates none of this for a durable subscription, so you do. The subject will be
orders.created. The adapter derives the topic name by replacing each dot with an underscore, so
the topic is orders_created. The queue is named by the subscription's DurableName, here
billing. See Topology and naming.
$ TOPIC=$(aws sns create-topic --name orders_created --query TopicArn --output text)
$ DLQ=$(aws sqs create-queue --queue-name billing-dlq --query QueueUrl --output text)
$ DLQARN=$(aws sqs get-queue-attributes --queue-url $DLQ \
--attribute-names QueueArn --query Attributes.QueueArn --output text)
$ Q=$(aws sqs create-queue --queue-name billing --attributes \
"{\"VisibilityTimeout\":\"30\",\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQARN\\\",\\\"maxReceiveCount\\\":\\\"5\\\"}\"}" \
--query QueueUrl --output text)
$ QARN=$(aws sqs get-queue-attributes --queue-url $Q \
--attribute-names QueueArn --query Attributes.QueueArn --output text)
$ aws sns subscribe --topic-arn $TOPIC --protocol sqs --notification-endpoint $QARN \
--attributes RawMessageDelivery=true
Three things there matter to the adapter. The queue has a redrive policy whose maxReceiveCount is
5, which the subscription's MaxDeliver must equal. The queue is subscribed to the topic, which
the adapter does not check. And the subscription is raw: the adapter reads the SQS message's body
and attributes as SNS published them, so a subscription that wraps each message in SNS's JSON
envelope would make every message unreadable.
3. Write the program¶
package main
import (
"context"
"log"
"log/slog"
"os"
"time"
"gitlab.com/phpboyscout/go/awsclient"
"gitlab.com/phpboyscout/go/envelope"
"gitlab.com/phpboyscout/go/messaging"
sqs "gitlab.com/phpboyscout/go/messaging-sqs"
)
func main() {
ctx := context.Background()
backend := sqs.Standard(sqs.Settings{
AWS: awsclient.Ambient(awsclient.WithRegion("us-east-1")),
Logger: slog.New(slog.NewTextHandler(os.Stderr, nil)),
})
got := make(chan messaging.Delivery, 1)
bus, err := messaging.New(backend, messaging.Settings{
Source: "urn:example:tutorial",
Subscriptions: []messaging.SubscriptionSpec{{
Name: "billing",
Pattern: "orders.created",
Shape: messaging.Fanout,
Mode: messaging.AtLeastOnce,
MaxDeliver: 5,
DurableName: "billing",
Handler: func(ctx context.Context, d messaging.Delivery) error {
got <- d
return d.Ack()
},
}},
})
if err != nil {
log.Fatal(err)
}
if err := bus.Start(ctx); err != nil {
log.Fatal(err)
}
defer bus.Stop(context.Background())
select {
case <-bus.Ready():
case <-time.After(30 * time.Second):
log.Fatal("the bus did not become ready")
}
err = bus.Publish(ctx, "orders.created", envelope.Event{
ID: "order-4172",
Source: "urn:example:tutorial",
Type: "com.example.order.created.v1",
DataContentType: "application/json",
Data: []byte(`{"order":"4172"}`),
})
if err != nil {
log.Fatal(err)
}
select {
case d := <-got:
log.Printf("received %s on %s, attempt %d: %s", d.Event.ID, d.Subject, d.Attempt, d.Event.Data)
case <-time.After(30 * time.Second):
log.Fatal("nothing arrived")
}
}
The pattern is orders.created, an exact subject. A pattern with * or > is refused, because
SQS has no pattern language. The subscription asks for AtLeastOnce, which both varieties offer,
and names a DurableName, which the adapter uses as the queue name. The Region is passed
because the adapter never guesses one.
4. Run it¶
Publish returned when SNS accepted the message. The queue got it through the raw subscription, the
adapter received it with a one-second long poll, and Ack deleted it.
5. Make it fail¶
Change MaxDeliver to 3 and run again. Start fails with the adapter's refusal inside it:
messaging: subscription "billing" could not start: messaging: subscribing "billing": messaging-sqs: MaxDeliver 3 but queue "billing"'s maxReceiveCount is 5; the two must be equal, because the queue's redrive is where the budget is enforced
The queue's redrive policy is where the retry budget is enforced, so the subscription has to agree
with it. Put MaxDeliver back to 5.
Then publish something no format can read, straight to the topic, past the adapter:
Run the program. The handler never sees it. The adapter logs that a message could not be read, each of
its five receives releases it at once, and SQS then moves it to billing-dlq. See
What happens to a message nothing can read.
Clean up¶
Where next¶
Build the topology for a real account, including FIFO and the queue policy AWS enforces and LocalStack does not. Connect to AWS for credentials and region.