High-performance data connectors for CDC data integration with Redpanda

Agent identity, full audit trails, task-scoped data access, per-agent and per-provider spend limits, and a kill switch, plus AI gateways, MCP hosting, and observability. Critically, all of it should run out of band, separate from the agents themselves, so agents can't govern or bypass their own controls.

Put a governance layer between your agents and everything they access, and run it out of band, separate from the agents, so they can't govern themselves. Then enforce four things through it: a verified identity for every agent, a tamper-proof record of every action, least-privilege scope so each agent reaches only what its job needs, and hard limits with a kill switch.

It was built for humans and deterministic software. It has no answer for agent-specific failure modes like hallucination and prompt injection, and no way to scope or shut down an autonomous process mid-task.

Centralized governance for AI agents is a single control layer between your agents and everything they access, enforcing identity, permissions, auditing, and spend in one place. Instead of each team or framework governing its agents differently, every agent follows the same rules.

Because any policy enforced through the agent is only as strong as the agent’s ability to retain and obey it perfectly under all conditions. Prompt injection can override instructions in the system prompt. Hallucination can cause an agent to invent permissions it doesn’t have. Routine context management can silently drop rules the agent was told to follow. Real governance has to run in channels the agent cannot access, modify, or even see. It has to be enforced by infrastructure, not by the agent’s good intentions.

Agent accountability means establishing a clear chain from every agent action back to the specific human who authorized the task, and having the infrastructure to act on that chain when something goes wrong. Instance-bound identity enables surgical revocation: you can halt a specific agent instance without affecting others, quarantine work in flight, and use the full transcript to understand and remediate whatever the agent already did. Accountability also means building in escalation paths and approval workflows for high-stakes actions, so agents check in with humans before doing something that can’t be undone.

Agent observability means capturing a complete record of everything an agent does: every input, every output, every tool call with every argument and response. Explainability goes further. It means that record is structured, versioned, and queryable enough to reconstruct the agent’s reasoning chain for a regulator, auditor, or affected customer. Both must be captured out-of-band by infrastructure the agent cannot influence. The EU AI Act already mandates transparency and explainability for high-risk AI systems, and similar requirements are spreading globally.

Agent authorization requires a fundamentally different model from human role-based access control. An agent should receive only the permissions it needs for the specific task at hand: narrowly scoped, short-lived, and deny-capable. When it acts on behalf of a human, its effective permissions are the intersection of its own scope and the delegating human’s. No link in the chain can escalate beyond what every other link permits. Most deployed systems today have no concept of this granularity.

Agent identity matters because most deployments today use shared API keys or service account tokens, making it impossible to distinguish one agent’s actions from another’s or trace decisions back to the human who authorized them. Instance-bound identity gives each agent its own cryptographic credential tied to a specific task and delegation chain. When an agent acts on behalf of a human, that chain needs to be propagated through infrastructure the agent cannot modify. Not in a prompt or a header, but in a channel that the infrastructure controls.

The four pillars of AI agent governance are identity, authorization, observability and explainability, and accountability and control. Each must be enforced through infrastructure that the agent cannot access or circumvent. Redpanda calls this out-of-band governance.

Durable execution enhances debugging by allowing developers to replay workflows with the exact same inputs and conditions that previously led to issues. This deterministic replay helps in identifying and fixing bugs more effectively, especially in non-deterministic environments like those involving LLMs.

State management is crucial in durable execution as it ensures that the current status of a workflow is accurately recorded and can be retrieved when needed. This allows for seamless recovery and continuation of processes, which is vital for maintaining reliable and consistent AI operations.

Durable execution is important for enterprise AI agents as it allows for consistent and reliable task execution, even in the face of unexpected failures. This ensures that workflows can continue from the last known good state, which is essential for maintaining data integrity and operational continuity in enterprise environments.

No. Since Redpanda SQL is only Postrgres wire-compatible, but does not otherwise share the same Postgres code base, you cannot employ extensions with it, such as PostGIS or pgvector.

Redpanda SQL is generally available today on Redpanda BYOC on AWS. GCP support is coming soon. Self-managed / on-premises deployment is targeted for 2H FY2027. Reach out to your Redpanda account team for current availability or to join the waitlist for upcoming deployment options, including Azure.

Redpanda SQL is not built on PostgreSQL. It's built on the MPP (Massively Parallel Processing) engine technology that Redpanda acquired from Oxla, written from the ground up in C++ including many novel query engine innovations. The engine speaks the Postgres wire protocol natively, which is why standard clients like psql and DBeaver connect without modification.

It's designed for large-scale analytical workloads with strong data compression and cost-efficient scaling, and is managed entirely by Redpanda — no separate Oxla relationship or additional configuration required. The Oxla engine's novel ring-buffer streaming shuffle allows it to maximize CPU usage, especially at large core counts. Its design is described in a peer-reviewed paper, conditionally accepted for publication at VLDB 2026, one of the premier academic conferences for database systems research.

Iceberg Topics is a Redpanda Streaming feature that writes streaming data to both a hot tier (the live Redpanda topics on local storage) and a cold tier (Parquet/Iceberg in S3 or GCS). At this time, Redpanda SQL queries Iceberg tables written by Redpanda Iceberg Topics specifically — not external Iceberg tables populated by other tools. Standard Redpanda topics work fine without Iceberg for SQL use cases like topic debugging and real-time analytics.

No, Redpanda SQL is additive. Snowflake and Databricks handle the workloads they're designed for. Redpanda SQL covers what they can't: streaming data still in your topics before lakehouse ingestion, queried inside your own VPC without egress charges or compliance issues.

Redpanda SQL uses the Postgres wire protocol, so almost any standard Postgres-compatible SQL client connects without modification: psql, DBeaver, DataGrip, and most analytics tooling that supports the Postgres interface. No proprietary drivers, no new SDKs, no obscure query syntax or semantics (like with FlinkSQL), and no special configuration required.

No. Redpanda SQL is fully ad-hoc: connect your Postgres client and query any topic, any time, with no pipelines or pre-defined views. This separates Redpanda SQL from tools like ksqlDB, which creates separate materialized views and requires additional infrastructure.

Warehouses can only query data that's already been explicitly ingested, which requires a data engineer and typically adds 1–10 minutes of pipeline lag (or longer, for some infrequent batching jobs). Redpanda SQL automatically and transparently queries live streaming data and historical Iceberg data in a single, simple statement. No data engineering, complicated joins or union logic. It covers what your warehouse can't reach.

Redpanda SQL is a Postgres-compatible analytical query engine that runs inside your Redpanda BYOC cluster. It gives you direct SQL access to live data in your Apache Kafka-compatible Redpanda Streaming topics and historical data in Iceberg tables, combined in a single table/query, without pipelines or new infrastructure to provision. It performs a single federated query across both and combines the results seamlessly and accurately. Connect using almost any standard Postgres client and start querying immediately.

Hardening requires running OpenClaw in isolated containers with strict firewall rules, limiting outbound egress, auditing community skills for injection patterns, and using agent application firewalls. Even with these measures, the fundamental issue — the agent holding credentials inside the sandbox — remains unresolved without an external gateway and token vault architecture.

Enterprise teams replacing OpenClaw's sandbox-only model adopt platforms with built-in governance, such as Redpanda's Agentic Data Plane, which provide the gateway, audit trail, and credential management stack required to run agents at scale without sacrificing security for functionality.

Production-ready agent infrastructure combines four components: a gateway for centralized access control and observability, audit logs and transcripts for compliance and performance tracking, a token vault for out-of-band credential management, and sandboxed compute with governed network access. Redpanda Connect provides the data pipeline layer that integrates these components into a streaming-first agentic architecture.

A token vault manages credentials out-of-band so agents never hold sensitive tokens directly. When an agent requests access to a service like Salesforce or ServiceNow, the vault issues a short-lived, scoped token for that specific operation, preventing credential leakage even if the agent is compromised.

OpenClaw stores credentials in config files that the agent can read, copy, and transmit. Threat actors exploit prompt injection to extract these plaintext tokens, and unsecured instances — over 18,000 of which are currently exposed to the public internet — amplify the risk further.

A gateway creates a single choke point for all agent interactions with external systems, enabling full observability, rate limiting, and guardrails enforcement in one place. This architecture lets security teams monitor every tool call and LLM invocation, and provides a kill switch to shut down a rogue agent without hunting across dozens of services — a capability the Redpanda Agentic Data Plane delivers out of the box.

OpenClaw operates as a "personal assistant" with a single-operator trust model, and it lacks the centralized governance, audit logging, and credential isolation that enterprise environments require. Organizations scaling OpenClaw beyond a single developer machine face a fundamentally different threat model that the sandbox alone cannot address.

OpenClaw exposes organizations to indirect prompt injection, credential theft, and malicious community skills — over 15% of which have been flagged as potentially containing harmful code. Because OpenClaw holds API keys and tokens directly in config files, a compromised agent can leak plaintext credentials or exfiltrate data to external services.

Yes. Redpanda Connect includes built-in AI connectors for major LLMs including AWS Bedrock, Google Vertex, OpenAI, Llama, and Cohere. It can convert unstructured data into vector embeddings and stream them to vector databases, making it well suited for GenAI pipelines and RAG (retrieval-augmented generation) workflows. Kafka Connect has no equivalent AI support.

Kafka Connect is a Java-based distributed integration framework that requires complex setup, manual JAR management, and separate logging configuration. Redpanda Connect is a cloud-native alternative built on a single binary, configured in YAML, with built-in observability and no distributed node coordination. It is fully Kafka API-compatible and uses 3x less compute than Kafka Connect.

Key optimizations from Redpanda's benchmark include: using a binary format like AVRO (~20% throughput improvement over JSON), batching by message count rather than byte size, increasing build_parallelism close to available instance cores, and maximizing Snowpipe Streaming channels via channel_prefix and max_in_flight settings. Scaling inputs and outputs within a single node unlocked significant additional throughput.

Common use cases include market surveillance and financial analytics (where sub-second decisions matter), fraud detection, operational dashboards, IoT data ingestion, gaming analytics, and autonomous driving telemetry. Any workload requiring up-to-the-minute data in Snowflake for SQL-based analytics benefits from a streaming ingestion pipeline.

The simplest approach is Redpanda Connect's snowflake_streaming connector, which is built on the Snowpipe Streaming API and configured in a few lines of YAML (no custom code required). It supports schema evolution, exactly-once delivery, and parallelism tuning. It ingests data nearly 2x faster than Kafka Connect and scales horizontally by adding more Redpanda Connect nodes.

In benchmark testing using Redpanda Connect and Snowpipe Streaming, Redpanda achieved a P50 latency of 2.18 seconds and P99 latency of 7.49 seconds while streaming 14.5 GB/s — well above Snowflake's own recommended limit of 10 GB/s on a single table (at the time of this benchmark). Using AWS PrivateLink instead of the public internet would reduce latency further.

Snowpipe Streaming is a Snowflake API that enables continuous, low-latency ingestion of rows directly into Snowflake tables, without staging files first. Regular Snowpipe loads data in micro-batches from staged files, introducing higher latency. Snowpipe Streaming is designed for real-time use cases where data needs to land in Snowflake in seconds rather than minutes.

Serverless is the fastest and easiest way to start streaming data. With Serverless clusters, you host your data in Redpanda’s VPC, and Redpanda handles automatic scaling, provisioning, operations, and maintenance. It includes Redpanda Connect for hundreds of pre-built data connectors and is available across 7 AWS regions. A free trial on AWS is the fastest way to get started with Serverless.

Common use cases include building APIs, processing real-time event streams, running trigger-based workflows (such as order confirmations or inventory updates), media processing, data pipelines, webhook handlers, microservices, chatbots, and scheduled jobs. It's well-suited for any workload that is event-driven or unpredictable in volume.

The four key benefits are: cost optimization (you only pay for actual usage, not idle capacity), automatic scalability (resources scale up or down without manual intervention), improved developer experience (no infrastructure to manage), and faster time to market (less setup means quicker deployment and iteration).

Traditional cloud hosting requires you to provision servers or virtual machines in advance and pay for them whether they're in use or not. With serverless, resources are provisioned on demand and freed up after each task, so you only pay for actual compute time. The tradeoff is less control over the runtime environment, which is why teams often use traditional cloud or containers for stable, long-running services and serverless for event-driven or variable workloads

Iceberg Topics lets you enable Apache Iceberg on any Redpanda topic with a single property. Once enabled, Redpanda automatically writes streaming data to Iceberg tables in cloud object storage, preserving offsets and timestamps. If a schema is registered, Redpanda derives the Iceberg schema automatically. If not, it uses a default schema of key, value, and timestamp. The table is then immediately queryable by any Iceberg-compatible analytics tool. Pretty neat.

Any Iceberg-compatible tool works, including Snowflake, Databricks, Google BigQuery, ClickHouse, AWS Athena, Apache Spark, Apache Flink, and Starburst. Data is stored as Parquet files in your cloud object store and registered with your existing Iceberg catalog such as AWS Glue or Snowflake Open Catalog.

Zero-ETL eliminates the custom pipelines traditionally needed to move data from a source system into a data lake. With Redpanda's Iceberg Topics, streaming data is written directly to Iceberg tables as it arrives. No separate ETL job, no extra infrastructure, and analysts can query it immediately.

A data lake stores raw data in cloud object storage. A data lakehouse adds a table format like Iceberg on top, enabling SQL queries and governance. A streaming platform like Redpanda handles real-time event ingestion and transport. The three work together: the streaming platform feeds data into the lakehouse, where it becomes queryable.

Apache Iceberg™ is an open table format for storing large-scale analytical data in cloud object stores. It adds database-like capabilities — efficient queries, schema evolution, and catalog registration — without vendor lock-in. For streaming data, it means you can query real-time events using standard SQL tools without moving or copying data.

Redpanda is a streaming data platform with the speed and reliability needed for real-time analytics. Written in C++ with a thread-per-core architecture, it runs up to 10x faster than Apache Kafka while consuming 3x less compute. It deploys as a single binary without external dependencies (bye-bye JVM) and is fully Kafka-compatible, so it works with your existing ecosystem of stream processors, OLAP databases, and BI tools.

A typical stack has four layers: a streaming data platform (like Redpanda or Apache Kafka) for ingestion, a stream processor (like Apache Flink or Spark) for on-the-fly computation, a real-time OLAP database (like ClickHouse, Pinot, or Druid) for querying, and a dashboard or BI tool for surfacing insights to end users.

Real-time analytics is used across finance (fraud detection, algorithmic trading), e-commerce (personalized recommendations), gaming (live player behavior tracking), AdTech (real-time bidding), cybersecurity (instant breach alerts), supply chain (live inventory updates), and IoT (sensor-driven automation).

Real-time analytics lets organizations act on data before it loses relevance. It enables faster decision-making, better customer experiences, and earlier detection of fraud or system failures. More importantly, it shifts teams from reactive to proactive: instead of responding to problems after the fact, they can predict and prevent them.

Real-time analytics is the process of analyzing data as soon as it's generated, enabling instant insights and immediate action. Unlike batch processing, it handles continuous data streams with millisecond-to-second latency, making it essential for use cases like fraud detection, personalized recommendations, and IoT monitoring.

Redpanda acquired Benthos (benthos.dev) in May 2024 and rebranded it as Redpanda Connect. The core engine remains MIT licensed and is maintained at github.com/redpanda-data/benthos, so teams that built on Benthos can continue to do so. The broader connector ecosystem ships under Apache 2.0.

Yes. Redpanda Connect ships certified CDC inputs for popular databases including PostgreSQL, MySQL, Oracle, MongoDB, Microsoft SQL Server, Amazon DynamoDB, and Google Spanner, among others. Each CDC input deploys as part of a standard pipeline and runs in containers with no additional infrastructure required.

Redpanda Connect has three tiers of components for self-managed deployments. Community and Certified connectors are completely free under the Apache 2.0 license:

• Community connectors are open source with best-effort support via the Redpanda Community Slack.
• Certified connectors are tested for high performance and observability with Redpanda Enterprise, and include expert support as part of a paid Redpanda Connect plan.
• Enterprise connectors are a small portion of components that require either a paid self-managed Redpanda Connect plan, or are available as managed pipelines on Redpanda Cloud with pay-as-you-go pricing.

Note that Redpanda Cloud supports a subset of Community and Certified connectors — components that require local resources (filesystem, shell access, local model inference, raw sockets) or are self-managed-specific are not available as managed pipelines. See the full list of components available on Redpanda Cloud.

Yes. Redpanda Connect is a complete alternative to Kafka Connect. It uses 3x fewer compute resources, ships as a single 128 MiB binary with all connectors included, and requires no separate worker cluster or connector REST API. Pipelines are defined in declarative YAML and can be deployed instantly.

Logical replication is the PostgreSQL mechanism that decodes WAL entries into a stream of row-level changes. CDC (Change Data Capture) is the broader pattern of capturing and propagating those changes to downstream systems. In practice, most Postgres CDC tools (including Debezium and Redpanda Connect) use logical replication as their underlying transport.

Redpanda Connect currently supports CDC inputs for PostgreSQL, MySQL, MongoDB, Microsoft SQL Server, Oracle, Amazon DynamoDB, and TigerBeetle.

Yes. Redpanda Connect has native CDC inputs for Postgres that capture changes using logical replication without requiring Kafka Connect or a JVM runtime. It runs as a single binary optimized for containers.

CDC captures incremental data changes as they happen, streaming only the rows that were inserted, updated, or deleted. ETL (Extract, Transform, Load) typically processes data in scheduled batches, extracting full datasets or large snapshots. CDC is better suited for real-time use cases where low latency matters.

Yes. PostgreSQL supports CDC through its logical replication feature, which uses the Write-Ahead Log (WAL) to capture row-level changes. Tools like Debezium and Redpanda Connect use logical replication slots to stream committed transactions from Postgres in real time.

Yes. The tigerbeetle_cdc input captures change events from TigerBeetle, an open-source financial accounting database built for high-throughput double-entry bookkeeping. It's a natural fit for real-time audit trails, financial event streaming, and fraud detection pipelines. Unlike the other CDC inputs listed here, TigerBeetle is community tier . No Enterprise license required. (Debezium has no TigerBeetle connector, just sayin'.)

Yes. The aws_dynamodb_cdc input uses DynamoDB Streams to capture item-level changes and route them downstream. This is a meaningful difference from Debezium: there's no stable Debezium connector for DynamoDB, which means teams have historically had to wire together Streams, Kinesis, and Lambda to get change events out. Certified, Enterprise-licensed. See the Amazon DynamoDB CDC docs.

Yes. The oracledb_cdc input reads directly from Oracle's redo logs via LogMiner. It handles snapshot-then-stream automatically: a consistent snapshot of your tables on first run, then streaming from that point forward. Restarts resume from a checkpoint stored in Oracle itself — no external state required. Certified, Enterprise-licensed. See the Oracle CDC docs.

Redpanda Connect includes certified CDC inputs for: MySQL, PostgreSQL, MongoDB, Oracle, Microsoft SQL Server, Amazon DynamoDB, Google Cloud Spanner, and TigerBeetle. All are production-ready and backed by Redpanda support, with the exception of TigerBeetle (community tier). Debezium's stable connector set covers MySQL, PostgreSQL, Oracle, MongoDB, SQL Server, Db2, and Cassandra. (DynamoDB and TigerBeetle aren't there, and Google Spanner is still in preview.)

Yes. Redpanda Connect ships certified CDC inputs that capture change events natively — no Kafka Connect cluster, no connector REST API, no separate workers to manage. Each input deploys as part of a standard Redpanda Connect pipeline and runs in containers. See the full list in the Redpanda Connect Components Catalog.

Yes. The mysql_cdc input captures inserts, updates, and deletes directly from MySQL's binary log — no Kafka Connect, no Debezium, no JVM. The entire pipeline is a YAML config that runs as part of standard Redpanda Connect. It's a certified, Enterprise-licensed component. See the MySQL CDC docs.

Nope. Kafka 4.0 shipped KRaft as the sole consensus mode in March 2025, which is a meaningful simplification for Kafka operators. But KRaft addresses metadata management, not the core broker execution model. Kafka controllers can be colocated with brokers, but the JVM-based architecture, page-cache-centric I/O model, and general tuning profile still remain. Redpanda's performance and operational advantages still come from its C++ thread-per-core execution, DMA-based I/O, and compiler-level optimizations like profile guided optimization (PGO).

Production deployments should follow current Redpanda sizing guidance, which emphasizes fast local SSD/NVMe storage, XFS, modern Linux kernels, and enough network bandwidth for the workload.

While rpk redpanda tune all configures operating system and kernel settings, rpk iotune specifically benchmarks your hardware at startup to generate an optimized io-config.yaml file.

Redpanda is built for Kafka API compatibility, allowing your existing clients, producers, and consumers to work without application rewrites in many cases.

Yes. Because durable execution decouples an agent's identity from its physical process, the agent doesn't need to stay pinned to a specific machine or TCP connection. It can run on serverless functions, suspend when waiting on long-running tasks, and resume on a completely different server once the result is ready.

There are two distinct challenges. The agentic challenge is balancing autonomy with alignment: giving agents enough freedom to be useful while ensuring they stay within governance, security, and compliance boundaries. The distributed systems challenge is making multi-agent workflows reliable, observable, and recoverable across long-running, expensive operations. Durable execution addresses the second challenge but not the first.

In traditional microservices, each function call is cheap and fast, so retrying from the top after a failure is no big deal. With agents, LLM calls burn through expensive tokens, and tools may trigger data pipelines that cost tens of thousands of dollars. Blindly retrying everything below the failure point wastes significant time and money. Durable execution enables fine-grained recovery so only the failed step needs to be addressed.

‍

‍

‍

‍

‍

‍

Both address state management in distributed systems, but they present very different programming models. Actors work well when a single message can be fully processed within one activation. But when an actor triggers something external and needs to wait, engineers end up building manual state machines and managing control flow themselves. Durable execution preserves a sequential programming model where control flow is handled for you, making complex workflows (nested try/catch blocks, loops, conditional branching) much simpler to write and reason about.

A durable promise is a coordination and recovery mechanism with a stable, externally addressable identity (essentially a URL). It represents a point in a distributed call graph. If a process restarts, it can rejoin an existing durable promise instead of re-triggering the underlying computation. It also enables hard suspension: an agent can go away entirely while waiting, then resume when the promise resolves.

A failure is involuntary, like a server crash or a dropped TCP connection. An interruption is a broader concept that includes both involuntary failures and voluntary pauses, such as an agent deliberately sleeping for a week or waiting for an external event. Designing for interruption tolerance covers both cases, making the system more resilient overall.

When AI agents move into enterprise environments and run autonomously for hours or days, they become distributed systems problems. Agents call tools, kick off data pipelines, and invoke other agents, creating complex call graphs. If a connection drops or a process crashes mid-workflow, traditional retry logic means restarting everything from scratch. With LLM calls and data pipelines that can cost thousands of dollars per run, that's not viable. Durable execution lets these workflows survive interruptions and resume where they left off.

Durable execution is a programming approach where code is written in an interruption-agnostic way but executed in an interruption-tolerant manner. In practice, this means engineers can write standard sequential code without worrying about failures or pauses, and the underlying framework ensures the execution survives crashes, restarts, or long wait times without losing progress.

Redpanda Migrator handles offset translation automatically. If using MM2, the MirrorCheckpointConnector manages this, though it often requires manual tuning.

‍

Use MM2 only if you require a legacy Kafka Connect architecture; for most active take-outs, Redpanda Migrator is the faster, single-binary standard.

Yes. You can achieve near-zero downtime by syncing clusters ahead of cutover with Redpanda Migrator, then performing a rolling client cutover. Teams with stricter disaster recovery requirements can also evaluate Redpanda Shadowing as part of that plan.

In most cases, no. Redpanda is fully Kafka API-compatible, so you typically only need to update your broker list and authentication credentials.

Timelines vary by data volume, but Redpanda Migrator can reduce engineering time from weeks to days compared to legacy MM2 setups.

Redpanda uses the Raft consensus algorithm for replication. With typical production settings (for example, acknowledgements requiring replication to multiple nodes), Redpanda commits data durably across a quorum before acknowledging writes, so node failures don't result in data loss.

Redpanda runs on bare metal, in containers (Kubernetes), or as a fully managed cloud service. We also support Bring Your Own Cloud (BYOC) for data sovereignty and Serverless options.

Redpanda implements the Kafka protocol in C++ using a thread-per-core architecture. This allows existing Kafka clients to connect seamlessly while removing JVM garbage collection from the broker's critical path.

A streaming data platform captures and processes data as a continuous flow of real-time events. It serves as the backbone for event-driven systems, allowing consumers to react immediately without waiting for batch jobs.

Remote Read Replicas are a great fit when your main goal is to place data close to consumers without pushing read load onto your primary cluster. For strict business continuity during regional outages, pair this with a DR strategy such as Whole Cluster Restore or Shadowing.

Whole Cluster Restore is a cost-effective method that rebuilds clusters from archived data, best for relaxed RPO/RTO. Shadowing maintains a live replica for mission-critical workloads that need low RPO and low RTO.

Iceberg Topics reduce the need for separate ETL pipelines to format streaming data for analytics. This lowers infrastructure costs and operational complexity by letting engines like Snowflake and Databricks query the stream data directly.

Use Tiered Storage for low-latency local writes and cost-effective long retention (with asynchronous offload to object storage). Use Cloud Topics to persist data directly to object storage and reduce cross-AZ network fees for workloads that can tolerate higher write latency.

No. Tiered Storage is enabled and managed by default for all Redpanda Cloud clusters, so no manual configuration is required there. However, to use Cloud Topics, you must explicitly select the "Cloud" storage mode by setting the redpanda.storage.mode=cloud property when creating a new topic. While standard topics use Tiered Storage as a background process to archive older data, Cloud Topics require this opt-in at creation because they use a specialized "storage-first" path that writes message payload data directly to object storage from the very first byte. You don't need to manage any new infrastructure, but you do need to choose the Cloud Topic mode to trigger the 90% networking cost savings.

Cloud Topics is an enterprise feature available on Redpanda Self-Managed, Redpanda BYOC, and Redpanda Dedicated starting with version 26.1. It is specifically designed and recommended for multi-AZ architectures, where it provides the maximum ROI by eliminating cross-AZ networking fees on high-throughput data. Cloud Topics is not currently available on Redpanda Serverless, as that environment is currently single-AZ and does not yet support the multi-zone networking optimizations that Cloud Topics was built to solve.

One of the defining features of the Redpanda One (R1) engine is its multi-modal nature, which allows different storage classes to coexist seamlessly within a single cluster. This eliminates the need to manage a "zoo" of niche clusters for different workload requirements. You can run low-latency operational apps on standard topics, alongside high-volume telemetry or AI context on Cloud Topics—all managed under a single security model, the same Kafka API, and on one Redpanda cluster.

Yes. Because Cloud Topics maintain the Raft-based metadata layer on local disk, they inherit the same enterprise-grade consistency logic as standard Redpanda topics. This allows Cloud Topics to fully support exactly-once semantics, transactions, and idempotent producers.

Redpanda intentionally avoids fully diskless designs because moving critical metadata and consensus logic (Raft) to an external service creates dependencies and limitations that could break core Kafka features. Cloud Topics keep the "brains" of the partition on local NVMe disks. This hybrid design delivers the 90% cost savings of a diskless system, while maintaining the battle-hardened reliability, strict message ordering, and sub-10-millisecond metadata performance of a stateful broker.

The primary difference lies in the write path and the timing of data movement to the cloud. Tiered Storage is an offload mechanism where data is first written to local disks for maximum speed & safety, and then moved to object storage automatically as it ages. Cloud Topics are "storage-first" from the first byte, using a write pass-through model to persist message payloads directly to the cloud. This allows users to bypass the initial "replication tax" of a high availability architecture—the expensive cross-AZ networking fees—for high-volume, latency-tolerant data.

While both operate in a cloud environment, "Cloud Topics" refers specifically to a specialized storage class within the Redpanda One (R1) architecture rather than a simple service location. Standard topics replicate the entire message across local NVMe disks in multiple availability zones to prioritize performance and high availability. In contrast, Cloud Topics use a write pass-through model where heavy message payloads are streamed directly to object storage (S3/GCS) while only lightweight metadata remains on local disk to ensure consistency.

Real-time analytics visualizes what is happening right now, whereas real-time AI automates the decision of what to do about it. In a nutshell: analytics informs humans, AI takes action.

There is no single "best" model; success depends on the end-to-end system architecture. You need a model that fits your latency budget paired with a streaming data layer that ensures continuous, fresh context.