A comparison of popular message queues

Message queues provide a means of fast asynchronous communication between software components. They come handy in solving some common problems that arise while building software, e.g.

• loosely couple components by delegating message delivery and retry logic to the message queues.
• split long running tasks into multiple sub tasks (without having to use threading or concurrency libraries).
• "hold" messages when destination is unavailable or overwhelmed.
• provide asynchronous updates of long running transactions.
• scheduling messages for future delivery.

At Better.com, we tried Beanstalk and Amazon SWF and realized that they weren't a fit for our stack. For example, Beanstalk doesn't have message persistence or authentication. SWF doesn't scale well at high frequencies and it is built for complex asynchronous multi-step workflows. So it isn't the right product for high frequency simple async communications.

Message queue vs Message broker

A Message queue simply enqueues and dequeues messages

A Message broker can perform additional operations on messages before dequeueing them. You can find more information regarding these transforms here

Comparison

Our criteria for a message broker were thus:

Must have: Delivery retry, CI/CD friendly, Fast, High Availability Nice to have: Admin console, PubSub

Here are the contenders

• ActiveMQ: Open source Apache project written in Java. 15+ years old and very mature.
• Amazon SQS: AWS product for message queueing.
• RabbitMQ: Open source message broker written in Erlang.
• IronMQ: Closed source message broker.
• Redis: In memory cache that can be used as a queue using redis operations.

Core features

Management features

SQS

Pros:

• Easy setup and API.
• Outstanding client libraries for popular language.
• Managed
• "Infinite" scalability

Cons:

• Consumers pull messages, but long polling is available on the clients.
• Max batch size of 10 messages per consumer
• No pub/sub. You'd have to use SNS for pub/sub.

RabbitMQ

Pros:

• Self hosted
• Cheap and fast
• Outstanding client libraries for popular language.
• Supports multiple protocols (amqp, mqtt, stomp)

Cons:

• AMQP is complex
• RabbitMQ is written in erlang. The configurations are in erlang.

ActiveMQ

Pros:

• Extensive broker capabilities link
• Free
• Fast
• Supports multiple protocols (amqp, openwire, mqtt, jmx, stomp)
• Java client support and feature set is impeccable.

Cons:

• Client library support for non java languages is mediocre
• High availability is achieved via a failover mechanism, which is not supported out of the box by non-jms libraries.
• Requires reboots for configuration changes.

Redis

Pros:

• Free
• Fast
• Outstanding client libraries for popular languages

Cons:

• Advanced message queue features like atleast-once delivery need to be implemented in the clients.
• 3rd party Admin UIs need to be used.

Winner

None of the queue implementations check all the boxes. We went with ActiveMQ since it was the best fit for our stack and traffic volume. in production, ActiveMQ on AWS was very easy to setup. The ability to spin up ActiveMQ instances in Docker ties in neatly into our CI/CD pipelines and lets us run proper integration tests without the need for mocking. It was significant work to implement and test client libraries in Javascript and Python, but this was a good tradeoff for the ease of use and speed provided by ActiveMQ.

Final Impressions

We've been running ActiveMQ for over a year in production. It's proven to be lightweight, fast and requires low maintenence. On the other hand, writing client libraries in Javascript and Python has not been as easy as we thought. It required a lot of effort to write these libraries and get them production ready. If this is not something you are up for, RabbitMQ might be a very good alternative.