Comparing NATS, NATS Streaming and NATS JetStream

I was recently involved in a project migrating our messaging system from AWS’s SNS/SQS to NATS.

This was my first experience with NATS and I found the concepts slightly confusing at first — particularly the differences between Core NATS, NATS Streaming (aka STAN) and NATS JetStream.

I hope this post helps shed some light on the three and gets people up to speed faster with this great messaging system. As NATS adoption grows, I’m sure the documentation will improve, and, what you’re reading, will become obsolete. Until then, enjoy!

I’ll start with a quick explanation of NATS, NATS Streaming and NATS JetStream. I’ll then dive into a quick demo of setting up a secure, highly available, message-persistent NATS cluster (using JetStream), and publishing/subscribing messages, using the nats.go client library.

What is NATS?

NATS is an open-source, cloud-native, high-performance messaging system. At its core, it’s a Publish/Subscribe (PubSub) system whereby clients can communicate with one another without knowledge of where services are located or what their precise endpoints are. Clients simply publish/subscribe to a subject and NATS takes full responsibility for routing the messages.

NATS is a “send-and-pray” type system: if a message is sent but the intended client recipient(s) is not connected to the server, the message is lost.

A short demo of plain NATS Server.

What is NATS Streaming (aka STAN)?

Where NATS provides at most once quality of service, streaming adds at least once. Streaming is implemented as a request-reply service on top of NATS.

In other words, Nats Streaming introduces message persistence & message delivery guarantees.

A short demo of NATS Streaming, showing message persistence.

Under the hood of a NATS Streaming Server

Under the hood, a NATS Streaming Server embeds a normal NATS Server and a Streaming Module. The Streaming Module is a client to a NATS Server: this means that Streaming clients are not directly connected to the streaming server, but instead communicate with the streaming server through the NATS Server.

Also, something important to note is that streaming messages are NATS messages made of a protobuf. From the documentation:

NATS clients and NATS Streaming clients cannot exchange data between each other. That is, if a streaming client publishes on foo, a NATS client subscribing on that same subject will not receive the messages. Streaming messages are NATS messages made of a protobuf. The streaming server is expected to send ACKs back to producers and receive ACKs from consumers. If messages were freely exchanged with the NATS clients, this would cause problems.

A NATS Streaming Server cluster can be made to be secure and highly available. However, due to the way it’s architected, it does have the following limitations:

• Cannot remove messages from the system that have been acknowledged (#1114)
• Cannot “NAck” (not acknowledge) messages (#1108, #1049)
• Clients cannot “pull” messages, messages are only pushed to them.
• Relatively poor integration with the NATS 2.0/accounts/security concepts (#1043)
• Cannot restrict clients to subscribe on specific channels (#1122)
• Not horizontally scalable (#999)

Despite these, there are multiple NATS Streaming projects in production and it has 40+ million docker downloads!

What is NATS JetStream?

To overcome these limitations of NATS Streaming (STAN), JetStream was created. It’s a new persistence offering, separate from NATS streaming. Like the latter, it supports at least once delivery with durable subscribers and message replay. However, it also:

• Supports Horizontal Scaling (without client indexes)
• Is optimized for extremely large data sets (securely handling billions of messages)
• Is tightly coupled with core NATS through API, deployment, and operations (unlike the NATS Streaming server, JetStream is not a client to a NATS server — it is embedded into it).

At the time of this writing, JetStream is close to entering General availability (GA). If you’re starting a new project with NATS, this the recommended technology to use.

From our experience, we did encounter a few issues with it, but we found the community very responsive in helping resolve them. Be sure to check the open issues on nats-server and client in order not to be hit with any nasty surprises during your implementation.

Finally, a quick note on the documentation. A quick google search on JetStream will direct you to github.com/nats-io/jetstream: there, you’ll find a lot of documentation. However, we found quite a few parts of that repo to be outdated. Also, the purpose of the code in that repository is not very clear: most of it exists in nats-server and nats.go.

Our approach was to read the README in github.com/nats-io/jetstream but not trust it 100%. We would often dive into the code of nats-server and nats.go to understand what was going on. Also, in our implementation, we did not use /jetstream but nats-server and nats.go.

With that out of the way, let’s do a small demo of how you can create a NATS cluster locally!

Setting up a Secure, Highly Available NATS Cluster with message persistence (using JetStream)

Our cluster will consist of 3 servers. We need an odd number because the cluster elects a leader through the RAFT algorithm. Our servers will authenticate with one another using X509 certificates. And our message log will be persisted on disk, on a file.

Here’s a demo of what we’ll be building:

A short demo of NATS JetStream Cluster and a Go client.

You can check out the entire repo on github.

Let’s get started with our docker-compose:

Note the following:

• We are creating three identical servers: n1, n2 and n3
• All three servers are joining the network nats
• We are exposing internal ports 4222 to external (host) ports 4222, 4223 & 4224 so that clients can talk to the cluster
• We are persisting our state on our host at ./persistent-data/server-n…
• The servers are configured using jetstream.conf— let’s look at that next.

Check the comments in the below snippet to understand how our cluster is configured. Note that this configuration is applied to every one of the three servers.

That’s it for our cluster. How about our client?

Setting up a NATS JetStream client

Creating a client is relatively straight-forward. We need to:

• Connect to the cluster, authenticating with an X509 certificate
• Create an ORDERS.received Subject. This entity lives on the server.
• Create an ORDERS Stream which will persist messages for the above Subject. This entity lives on the server.
• Create an ORDERS Consumer which will consume messages from the above Stream. This entity lives on the server.
• Publish a message to the Subject.
• Wait to receive the message (the consumer will push it to us)
• Acknowledge the reception of the message.

For a detailed explanation on how Subjects, Consumers and Streams, check the JetStream documentation. Here’s our client implementation:

Running this should produce the following output:

connecting securely to cluster
getting JetStream context
stream not found
creating stream "ORDERS" and subject "ORDERS.received"
publishing an order
attempting to receive order
got order: "one big burger"

Note that you can also use nats.io/nats-tools/nats to issue manual commands to the cluster. If you do so, you may need to change the client publish & subscribe permissions in config/jetstream.conf.

You can also check out the entire repo on github.

Useful Links:

• NATS creator explaining NATS motivation & vision
• Book explaining core NATS concepts
• Guide on deciding between At-Least-Once & At-Most-Once delivery
• NATS slack group (very responsive — got quite a few questions answered)