Redpanda 24.3 creates a query-ready table|stream duality with Apache Iceberg™, launches Native Postgres CDC

While the beta of Iceberg Topics is an important milestone, that's not all we've been up to. We have a range of updates to Redpanda Cloud (now open for business on Azure Marketplace), plus a bumper crop of new connectors for Redpanda Connect.

Iceberg Topics beta

Extend your data lake or cloud data warehouse with streaming data, delivered automatically and with zero ETL. Iceberg Topics in Redpanda unify streaming and analytics workloads, giving teams more choice for querying data and greater simplicity in how it flows into your data warehouse or data lake. Today, we are officially opening the beta of Iceberg Topics for self-managed Redpanda Enterprise and Redpanda Cloud BYOC customers.

We detailed the vision for Redpanda's Iceberg integration recently in the announcement of our multi-modal streaming engine, Redpanda One (R1). R1 allows data teams to have a single copy of the data made accessible in multiple ways, making it easier to trade off latency, durability, and availability per topic/table. Map business requirements to topics across your cluster using a single line of configuration.

Note that while in beta, Iceberg Topics functionality is supported for non-production use only. The integration works on a per-topic basis, allowing you to mix and match Iceberg Topics alongside other regular topics in the same cluster. Talk to your Redpanda customer success manager to enable access in a BYOC or self-managed environment.

Mountable Topics for Tiered Storage

"Hibernate" unused topics to conserve cluster resources and maximize cost savings and flexibility. You can now mount and unmount unused Tiered-Storage topics from a Redpanda cluster safely with zero data loss. This allows you to use 100% of your cluster capacity to support your primary workloads while preserving the full history of other workloads. It also makes them easy and cost-effective to recreate — even in a new cluster and with a new name.

The rpk CLI and Redpanda Cloud API  feature new mount and unmount commands for topics using Tiered Storage, allowing you to connect and disconnect them to and from your cluster. Mountable Topics are now available for self-managed Redpanda Enterprise clusters and for Redpanda Cloud BYOC and Dedicated clusters.

‍

Leader pinning

Save costs on cross-zone networking fees from your cloud provider and improve performance with geographically-distributed clusters. For a Redpanda cluster deployed across multiple availability zones (AZs) or multiple regions, leader pinning ensures that topic-partition leaders are geographically closer to their producers, optimizing write latency and cost efficiency.

Leader pinning complements follower fetching, which lowers costs for consumers with geographically-optimized reads. Now available for self-managed Redpanda Enterprise clusters and for Redpanda Cloud BYOC and Dedicated clusters.

For more updates from the 24.3 release, check out the docs for self-managed Redpanda.

Redpanda Connect for Postgres CDC and beyond!

We're introducing a range of new connectors and capabilities for Redpanda Connect, speeding integration with databases, AI models, and security.

Postgres CDC beta

Simplify data integration, enforce quality controls, and seamlessly stream changes from relational databases using Redpanda Connect. We're introducing new CDC input connectors that help users more easily "shift left," picking up transformations and schema validation earlier in pipelines, before data propagates to downstream systems, and while reducing load on upstream legacy databases.

The new PostgreSQL connector supports multiple replication modes and marks the beginning of a larger CDC effort that is optimized for Redpanda Connect's native Go (vs. Debezium's Java). The pg_stream input is an Enterprise-tier connector, available in beta for self-managed Redpanda Connect and as a managed connector for Redpanda Cloud. Stay tuned for our upcoming CDC connector for MySQL, which uses binary log file replication!

Secrets management

Look up secrets on remote systems at runtime, for improved compliance and pipeline lifecycle management. Redpanda Connect now allows the rpk connect CLI flag --secrets to interpolate secrets values from external systems like AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, and Redis. No more fumbling with different sets of environment variables or embedding credentials in your YAML! 😱

Secrets management is available now for self-managed Redpanda Connect deployments. Look for a fully managed version of secrets management (coming soon) built into the console for all Redpanda Cloud deployments. Here's a preview:

Offset mapping for Redpanda Migrator

Experience zero downtime migrations and faster disaster recovery! Redpanda Migrator now supports offset translation for consumer applications that need to switch between reading data from the source cluster and the target cluster without losing their place in the stream.

Enable seamless failover and ensure that consumers automatically read from the correct offsets in your migration's target cluster. Redpanda Migrator with offset mapping is available now as an Enterprise connector for self-managed Redpanda Connect, and in Redpanda Cloud via Redpanda Connect.

New AI connectors

AI is evolving fast, and so is the roster of AI processors for Redpanda Connect:

• Cohere: We've added Cohere as a new AI processor, with support for chat and embedding models. Cohere is the creator of the Command R LLM and joins a growing number of connectors that can run fully self-contained on your own infrastructure as part of our reference architecture for Sovereign AI.
  
• Embedding models: Embedding model processors for Amazon Bedrock and Google Cloud Vertex AI make it easier to build efficient RAG applications when combined with existing Redpanda Connect outputs to vector databases.

The new processors for Cohere, Bedrock, and Vertex AI are available now as Enterprise-tier connectors for self-managed versions of Redpanda Connect, and as managed connectors for Redpanda Cloud — in addition to a growing number of processors for OpenAI embeddings, chat, and other models.

Blobl playground

Bloblang, or blobl for short, is the high-level mapping language at the heart of transformation logic in Redpanda Connect. To help you explore its convenient built-in functions, we've simplified the learning experience with a new interactive Blobl coding playground in our docs. Check it out and bookmark blobl.redpanda.com for your future Bloblang reference!

Deep partner integrations

1. Ockam: The first zero-trust Kafka platform, with mutual authentication and end-to-end encryption. New input and output connectors are available now as open-source Community-tier connectors for self-managed Redpanda Connect.
2. Timeplus: The only SQL stream processing and analytics engine with columnar and row stores in a single binary. New input and output connectors are available as open-source Community-tier connectors for self-managed Redpanda Connect and as managed connectors in Redpanda Cloud.

Looking for more on the latest Redpanda Connect updates? Check out our docs.

More Redpanda Cloud in more places

In addition​ to the Iceberg Topics beta for BYOC clusters, Redpanda Cloud has its own highlights to share, making our service more available and better meeting customers where they are with an expanded presence.

Azure Marketplace launch

Simplify procurement on Azure! We're live in the Azure Marketplace with Redpanda Cloud Dedicated clusters, available through an annual commitment with metered billing. Available in Azure's East US, North Europe, and UK South regions. Access to Serverless clusters and monthly pay-as-you-go subscriptions is coming soon.

Customer-managed VNets on Azure

With a standard BYOC cluster, Redpanda manages the networking lifecycle. For additional security, you can deploy the Redpanda data plane into your existing virtual network (VNet) and manage the lifecycle yourself. Take even more control over your BYOC cluster and check out Customer-Managed VNets in our docs, in addition to existing support for Customer-Managed VPCs on GCP and Customer-Managed VPCs on AWS.

Scheduled maintenance windows

Redpanda Cloud runs maintenance and upgrade operations on clusters in a rolling fashion to minimize disruption, and now  — unlike other managed Kafka services — we're letting you set your own schedule. Align system upkeep to your business calendar by defining your own maintenance window via the console's Cluster settings menu. Available now for BYOC and Dedicated clusters.

Redpanda Terraform provider for Redpanda Cloud

Automate deployments, scale operations, and reduce human error. The Redpanda Terraform provider has landed in public beta! Integrate with the Redpanda Cloud API to create and manage resources like clusters, topics, users, ACLs, networks, and resource groups.

99.99% uptime SLA

When mission-critical apps go down, you lose money. That's why we're increasing the uptime SLA for Redpanda Cloud BYOC & Dedicated to 99.99% when configured as multi-zone clusters. This rating exceeds the SLAs of many cloud-provider-native Kafka services and brings Redpanda Cloud up to parity with other third-party streaming data providers.

See the docs for more on what's new for Redpanda Cloud.

Let's table it, for now…

Thus concludes our tour of Redpanda's 24.3 release. We're laying some exciting groundwork that draws Redpanda closer to relational databases as sources of truth at either end of the pipeline. In between it all sits Iceberg Topics, and the promise to unite streaming and SQL engines with maximum flexibility, keeping everything inside your own sovereign cloud infrastructure thanks to Redpanda’s BYOC deployment model.

In the meantime, sign up for Redpanda Serverless to get started quickly, or give Redpanda Cloud in Azure Marketplace a look for more power with the bulk savings of an annual commitment. You can also grab the free Community Edition from GitHub and run it yourself, which includes a built-in trial of Redpanda Enterprise features. Until next time, keep on streaming, and always be tabulating!