Capturing SNS Events with Lambda

resource "aws_sns_topic" "my_sns_topic" {
  name = var.my_sns_topic_name
  tags = var.tags
}

SQS Configuration

Queue types: Amazon SQS has 2 queue types: standard(default) and FIFO, the latter guarantees no duplicates in the queue and strictly preserved messages order and thus more expensive. When using standard queues, it is important to make sure that consumer services handle messages idempotently because of possible duplicate messages and send the resource version number when the order of messages is important.

Encryption: All Amazon SQS queues are encrypted via SSE(server-side encryption) by default.

Visibility timeout (0-43200 sec, default: 30 sec): In Amazon SQS the message stays in the queue until it is explicitly deleted. Visibility timeout sets the length of time that a message received from a queue(by 1 consumer) will not be visible to the other consumers. The visibility timeout begins when Amazon SQS returns a message. If the consumer fails to process and delete the message before the visibility timeout expires, the message becomes visible to other consumers. This setting typically should be set to the maximum time that it takes for the application to process it. For Lambdas it's recommended to set it 6x the Lambda's timeout + MaximumBatchingWindowInSeconds so that it has enough time to process the message, ie if Lambda's timeout is 30s, set SQS Visibility Timeout to 180s + MaximumBatchingWindowInSeconds.

Retention period (60-1209600 sec, default: 345600 sec or 4 days): Amazon SQS automatically deletes the messages that have been in the queue for more than the maximum message retention period.

Receive message wait time (0-20 sec, default: 0 sec): When consumer polls for the message this parameter indicates the maximum wait time for a message to become available in the queue before returning an empty response. If there are messages available in the queue at the time of the request, the response will be immediate, regardless of the specified wait time. Long polling is recommended.

Redrive policy: Allows to specify the source queue, the dead-letter queue and conditions under which Amazon SQS moves messages from the former to the latter if the consumer of the source queue fails to process a message a specified number of times. The maxReceiveCount is the number of times a consumer tries receiving a message from a queue without deleting it before being moved to the dead-letter queue. Setting maxReceiveCount to 1 results in any failure to receive a messsage to cause the message to be moved to the dead-letter queue. IMPORTANT: Don't use dead-letter queues when you want to be able to keep retrying the transmission of a message indefinitely.

locals {
  visibility_timeout_seconds = 180
  message_retention_seconds  = 1209600 # 14 days, max
  receive_wait_time_seconds  = 20 # long polling
}

resource "aws_sqs_queue" "my_sqs_queue" {
  name                       = var.my_sqs_queue_name
  fifo_queue                 = false
  visibility_timeout_seconds = local.visibility_timeout_seconds
  message_retention_seconds  = local.message_retention_seconds
  receive_wait_time_seconds  = local.receive_wait_time_seconds
  tags                       = var.tags

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.my_sqs_queue_dlq.arn,
    maxReceiveCount     = 5
  })
}

resource "aws_sqs_queue" "my_sqs_queue_dlq" {
  name                       = var.my_sqs_queue_name_dlq
  message_retention_seconds  = local.message_retention_seconds
  tags                       = var.tags
}

Lambda Configuration

TODO

Capturing SNS Events with Lambda

SNS Configuration

SQS Configuration

Lambda Configuration