Announcing new managed connectors in Redpanda Cloud and general availability of Redpanda platform 23.1

Last year Redpanda grew over 5x in revenue, and adoption soared for our source-available Community Edition. Global banks, hedge funds, technology unicorns, manufacturing companies, and gaming companies are just some of the enterprises choosing Redpanda for its unique combination of low latency, high throughput performance, ease of use, and generous cost savings in cloud spend for both small and large Apache Kafka® workloads.

Although with more adoption comes more responsibility, as Uncle Ben says.

So, Redpanda’s winter update introduces an exciting bundle of new features and product enhancements that make it even easier to adopt Redpanda, secure your data, and improve the resiliency of your data. Here’s a quick look at what’s new in Redpanda.

Managed connectors for Snowflake, BigQuery, Postgres, and MySQL in Redpanda Cloud

Redpanda Cloud now features built-in connectors for the most popular data sources and sinks to simplify your data pipelines. With a few clicks, you can push data to analytics services like Snowflake and BigQuery, or capture changes from Postgres and MySQL. This adds to the existing list of connectors (Amazon S3, Google Cloud Storage, and MirrorMaker) available in Redpanda Cloud to cover many analytical use cases.

New updates also introduce configuration pre-checks to reduce errors. Your connector configurations can now be tested before deployment, saving you hours of troubleshooting after the fact.

Plus, sensitive connector configuration information—like credentials, mTLS certificates, and passwords—is now managed by the underlying cloud secret store, giving you more robust privacy and auditability.

Kerberos support and enterprise manageability improvements in the Redpanda platform

With Redpanda platform 23.1 now generally available, the new release adds the following capabilities to your Redpanda clusters:

• Kerberos authentication
• Support for Azure Blob Store
• Cluster-wide diagnostics bundle
• Ingress/Egress rate limits
• Apache Kafka® compatibility enhancements
• Improved scalability and simpler administration

Kerberos authentication

Enterprise Kafka users leveraging Kerberos can now migrate their clusters to Redpanda without making changes to their authentication code. And, Redpanda supports Kerberos for self-hosted deployments running on bare-metal environments, allowing enterprises to maintain their existing identity management system (say, Active Directory), and comply with their current requirements or mandatory standards.

Redpanda also makes it a breeze to integrate with Kerberos. Redpanda’s zero dependency binary contains all the required libraries to support the Kerberos protocol. What’s more, administrators can leverage rpk to enable and configure Kerberos. No configuration files to manage, and no restarts necessary.

Support for Azure Blob Storage (ABS)

For enterprises that rely on Microsoft Azure, Redpanda now supports Azure Blob Storage with its cloud-first storage engine!

Redpanda capabilities, such as Tiered Storage and Remote Read Replicas, can now be backed by data in the Azure Blob Storage service. Azure Blob Storage can be used as the primary storage tier for topics. This reduces the cost of storage by 10x or more. Under the hood, tiered storage topics are offloaded to ABS “containers,” using Azure “Shared Keys” for authentication. This new capability includes support for compacted tiered storage topics as well as topic recovery.

Cluster-wide diagnostics bundle

We made it easier to collect diagnostic data from Redpanda clusters to speed up recovery times and cut down on manual effort. Now, enterprise administrators can gather diagnostic data from all nodes within a Redpanda cluster using a single rpk command and optionally upload it directly to Redpanda Support.

rpk debug bundle

This handy new command collects logs, controller metadata, metrics, system information, and Kubernetes information from an entire cluster. It then either saves a copy to a local disk or uploads the data directly to a Redpanda Support endpoint from the cluster, eliminating errors and resolving support cases faster.

Ingress/Egress rate limits

Redpanda 23.1 introduces configurable rate limits for Redpanda cluster nodes for more reliable operations and better Quality of Service (QoS) for all Kafka clients.

Measured in bytes per second, these rate limits can be applied for data ingress and egress. Administrators can leverage these limits to manage I/O backoff and backpressure for Kafka clients, protecting the cluster from rogue clients or unexpected spikes in upstream traffic that might compromise cluster stability. With a single setting, it’s simple to enforce an ingress or egress rate limit for all nodes in a cluster.

Kafka compatibility enhancements

One of Redpanda’s best attributes is its high degree of compatibility with Kafka so you can operate existing applications without changing your code. This new release supports additional Kafka APIs and controls for better interoperability with Kafka ecosystem tools, along with fine-grained control over data placement using standard Kafka approaches and tooling.

• Support for AlterPartitionAssignments, ListPartitionReassignments APIs to allow interoperability with 3rd party data balancing tools, such as CruiseControl® and support explicit partition placement for advanced use cases with highly skewed data/workload
• Support for time-based segment expiration (segment.ms) for tighter control over local disk usage and RPO with tiered storage.

Improved scalability and simpler administration

The 23.1 release also adds numerous improvements for better scalability and streamlined administration in enterprise environments, including:

• Automatic merging of small adjacent segments in Amazon S3 to support managing larger volumes of data in tiered storage efficiently.
• Smoother rolling upgrades when running on Kubernetes, with configurable tolerances for replication lag as upgrades safely roll through the pods in a cluster.
• A new node “self-test” tool to benchmark broker throughput, tail latency, and IOPS.
• A new explicit decommission status API to allow administrators to safely stop brokers when manually replacing or removing cluster nodes.

Additionally, Redpanda Console now sports a new view for cluster health checks, and as well as a new guided experience for topic configuration, simplifying your day-to-day cluster management activities. We also upgraded the look and feel with a brand new theme!

Together, these capabilities help enterprises save on administrative costs, while running more reliable and scalable streaming data pipelines that can process petabytes of data.

Reduce cloud spend and improve streaming data performance with Redpanda

Redpanda is proven to help you reduce your streaming data costs by up to 6x and improve performance by up to 10x. All of these winter updates are available today, so request a 15-day trial of Redpanda Enterprise or Cloud, or grab the free Community Edition from our Redpanda GitHub repo.

Have questions about how much we can reduce your cloud bill? Join the Redpanda Community on Slack, or get in touch with one of our experts.