Redpanda 22.1 is here!

We’re thrilled to announce the release of version 22.1 of our Redpanda streaming data platform. Our engineering team has been hard at work on this for many months, and we want to thank those who collaborated with us so that we could build these vital features for your applications. This release brings new capabilities to further simplify your operational experience with Redpanda, and significantly improves reliability and Apache Kafka^® compatibility. It also promotes to general availability Redpanda’s tiered storage capabilities.

Redpanda 22.1 comes loaded with the following new goodies for you:

• Transparent tiered storage
• Centralized configuration
• Maintenance mode
• Rack awareness
• Consumer offsets support
• Idempotent producers

But before we dive deep into each of those, we want to quickly revisit the recent change to our product version semantics.

Starting in 2022, we transitioned to a schedule with four major releases per year to ensure an optimal balance between feature velocity and product stability. The release naming format is now YY.F.P, where YY is the year, F is the feature release version (starting with 1), and P is the patch release version (starting with 1).

For example, the first feature release was numbered 22.1.1. The next patch for this release was 22.1.2, and the next feature release will be 22.2.1. We will always support the current feature release and the two immediately preceding feature releases.

Now, on to the good stuff!

Transparent Tiered Storage: Access to real-time & historical data

In our previous release, we introduced a multi-tiered storage solution that archives log segments in real-time to object storage in the cloud (AWS S3 or Google GCS). It also allows message consumers to transparently read local or historical (archived) data via the standard Kafka API in a way that optimizes both read latency and TCO for Redpanda clusters.

We’re pleased to announce that Transparent Tiered Storage is now in general availability as a commercial feature in Redpanda. Customers like LiveRamp use this capability to manage millions of user profiles as LiveRamp moves from batch to real-time processing.

This unique feature is powered by Redpanda’s Shadow Indexing technology, which enables you to access both historical and real-time data from a single API with minimal impact on read latencies, all while reducing storage costs and leveraging the durability of cloud object storage. This “hands-free” approach does not force a Redpanda customer to choose between performance and cost. Instead, it delivers a low-cost, low-latency solution that also unlocks new and powerful capabilities for analytics, such as combining current and historical data for fraud detection in payments systems.

Centralized Configuration: Simplified use & administration

In previous versions of Redpanda, when an administrator wanted to set a cluster-wide configuration property, they had to edit each node’s configuration file and then restart the node. That’s no longer the case.

Redpanda 22.1 splits the configuration into cluster-level and node-level properties that can both be set with the rpk CLI.

Cluster-level properties are now stored and replicated across each node in the cluster using Raft, the same piece of the architecture that maintains Redpanda’s reliability and prevents single points of failure. Administrators will still make node-level configurations directly on each node.

Centralized configuration for cluster-level properties aligns with our mission to make everything simpler for users and administrators. This feature not only ensures that cluster-level changes are less prone to human error and applied cluster-wide from a single change, but it also enables administrators to use version control and develop templates to use with IaaC automation workflows for Redpanda.

This is a big step toward reducing the impact of normal activities on the cluster’s operation, and we know it’s a feature that all of our users will appreciate.

Maintenance Mode: Downtime-free operation

All nodes need maintenance throughout their life, and the challenges of performing work on always-on systems are well-known. Applications that don’t provide a means of graceful shutdown can lead operators to neglect this crucial task, which in turn creates an unhealthy environment for security and reliability.

When the time comes to install security patches for the operating system or perform a rolling update of Redpanda itself, you can use the new “maintenance mode” feature of Redpanda 22.1 to keep your cluster operating smoothly while you do the work.

When you put a node into this state, Redpanda migrates its partition leadership to other nodes in the cluster and removes it as a partition leader. You’re then free to perform maintenance tasks on the node with no disruption to the applications that depend on Redpanda. This new feature ensures uninterrupted availability of all partition replicas and reduces the spikes in network traffic that occur when partition replication adjusts after a node suddenly goes offline.

Rack Awareness: Prevent data loss from hardware failures

If you’re operating a cluster in production, you want it to be resilient and protect against data loss, right? One of the ways that Redpanda 22.1 does that is with the new rack awareness feature. This allows an administrator to set a value for rack_id on the node, which specifies a failure zone that it belongs to. This value might represent a rack in a datacenter or a public cloud availability zone – anything that will create a unique failure domain to which the node belongs.

Once this value is set, then when the administrator enables the Redpanda configuration property enable_rack_awareness, Redpanda’s scheduler will attempt to distribute replicas of the same partition across nodes in different failure zones. The result is a Redpanda cluster that preserves the availability of all data after the loss of an entire failure zone, which in turn limits the chance of data loss in even the most catastrophic failures of a datacenter or public cloud.

Consumer Offsets: Greater ecosystem interoperability

Consumer offsets are a private topic on a Redpanda node. This topic stores committed offsets from each Kafka consumer that’s attached to Redpanda.

Redpanda 22.1 exposes the __consumer_offset key for customers to use. Many tools in the Kafka ecosystem rely on this value for their operation, and with it, Redpanda fits into and works seamlessly with even more environments and applications.

Idempotent Producers: More stable & reliable communication

Idempotency is a crucial attribute for any distributed system. It ensures that an action can be repeated with the same result, even if the action has already taken place. With Kafka, external factors like outages or high latency on a network connection can prevent a producer from receiving an acknowledgment in time, and when that happens, the producer might resend the message again and again until the acknowledgement makes it back.

The new Idempotent Producers feature in Redpanda 22.1 removes the impact of duplicate messages that were received as a result of failed acknowledgements, and this dramatically improves the stability and reliability of communication between Redpanda and remote producers.

Idempotent Producers are enabled by default in Redpanda 22.1. Administrators can disable this feature by setting redpanda.enable_idempotence: false.

How to get started with Redpanda 22.1

Redpanda 22.1 is available right now for you to use with your applications. You can download it from our GitHub repo or contact us to sign up for our cloud services. Information on how to upgrade is available in our documentation:

• Transparent tiered storage
• Centralized configuration
• Maintenance mode
• Rack awareness
• Consumer offsets support
• Idempotent producers

If you have any technical questions, please join our Slack community, where you can engage directly with our product team, and also follow us on Twitter and on YouTube to get regular updates on all of the new and exciting things happening at Redpanda Data.

We look forward to seeing what you build next with Redpanda.

‍