Deploy AWS SQS Queue with Serverless framework

Deploy FIFO queue - Edition #32

Hey, I'm Marco and welcome to my newsletter!

As a software engineer, I created this newsletter to share my first-hand knowledge of the development world. Each topic we will explore will provide valuable insights, with the goal of inspiring and helping all of you on your journey.

In this episode, I show you how to deploy a FIFO queue with a producer and a consumer with the Serverless framework.

You can download all the code shown directly from my Github repository: https://github.com/marcomoauro/sqs-queue-lambda

---

🔄 What are Queues?

A queue is an architecture component that manages tasks or messages in a specific order.

Queues are helpful because they allow systems to handle tasks efficiently and avoid being overwhelmed. They make sure that tasks are completed in the right order and can be processed when resources are available.

This leads to better performance, easier management of tasks, and more reliable systems.

Advantages of using queues:

• Load Balancing: By managing the flow of tasks, queues prevent systems from being overloaded. When too many tasks arrive, the queue holds them until the system is ready to handle them, thus maintaining a balanced workload and preventing crashes or slowdowns.

• Reliability: Queues ensure that every task gets processed, even if the system is busy or experiencing issues. This reduces the risk of losing tasks and ensures that all operations are completed successfully.

• Scalability: Queues can handle a growing number of tasks without a drop in performance. As the demand on the system increases, queues allow it to scale up smoothly, ensuring that additional tasks are managed efficiently.

• Decoupling producer and consumer rates: Queues allow the producer and consumer to operate at different speeds. If the producer generates tasks faster than the consumer can process them, the queue temporarily stores the excess tasks until the consumer is ready. This decoupling helps manage traffic spikes allowing the system to handle peak loads without overwhelming the consumer. Only the component producing messages needs to be scaled, while the consumer can remain unchanged.

AWS Simple Queue Service (SQS)

SQS is a cloud service that helps manage and store messages between software components. It ensures that messages are sent and received reliably.

SQS offers two types of queues: Standard and FIFO.

• Standard queue:

Standard queues provide high throughput, allowing a large number of messages to be processed quickly.

They are designed for applications where occasional duplicate messages and out-of-order processing are acceptable. This type of queue is ideal for scenarios where speed and scalability are important, such as processing tasks in parallel or distributing work across multiple consumers.

Messages are delivered at least once and there is no guarantee of the order of delivery.

• FIFO queue:

FIFO (First In First Out) queues ensure that messages are processed exactly once and in the exact order they are sent.

This makes them suitable for tasks that require strict order and exactly once processing, such as financial transactions or inventory updates.

FIFO queues have a limited throughput compared to Standard queues, but they provide stronger guarantees about message delivery and processing.

Each message is delivered once and remains available until it is processed and deleted.

---

👨‍💻 Let's implement together

You can download all the code shown directly from my Github repository: https://github.com/marcomoauro/sqs-queue-lambda

We implement a Lambda function that exposes an API via AWS API Gateway to process HTTP calls and queue messages.

The presence of a queued message triggers a second Lambda function, which acts as the consumer of the message.

serverless.yml

We can create resources in this way:

org: implementing
service: sqs-queue-lambda

provider:
  name: aws
  runtime: nodejs20.x
  region: eu-west-2
  tags:
    Name: sqs-queue-lambda
  environment:
    QUEUE_URL: ${self:custom.queue.url}
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - sqs:SendMessage
      Resource: ${self:custom.queue.arn}

plugins:
  - serverless-offline

resources:
  Resources:
    queue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: queue.fifo
        FifoQueue: true

functions:
  app:
    handler: lambda.handler
    events:
      - httpApi:
          path: /{proxy+}
          method: ANY
  consumer:
    handler: consumer.handler
    events:
      - sqs:
          arn: ${self:custom.queue.arn}
          batchSize: 1

custom:
  serverless-offline:
    reloadHandler: true
  queue:
    arn: !GetAtt queue.Arn
    url: !Ref queue

• Queue definition: