Autonomy is the future of infrastructure

As we enter a new world of Autonomy [Enterprise Agents], developer experience reigns supreme. Either engineers are empowered to build frontier apps, or data will simply be shared outside of the organization. Fact: 38% of workers share sensitive work information with AI without their employer’s knowledge.

True to our roots as an infrastructure company, our take on autonomy is focused on scalable tooling that gets out of the engineer’s way. Reducing complexity to what we think are the essential pieces to leverage AI advances in simple words: connectivity to data, internal systems and models using open protocols. Durability of execution for agent-to-agent communication via Redpanda. And, a simple SDK to glue it all together in a Ruby-on-Rails style experience. 

Autonomy is where code is in control of the end-to-end flow vs explicit code paths you’ve written, but first, a decomposition from a systems engineering view of how we got here.

Background

Twenty years ago, we were struggling with data. The volume, the speed, the size, and how to index it. Frankly, most companies were on single-node databases that sometimes had backups. In 2006, we called it the new oil. Like crude oil, it was valuable, but distilled insight was what we were after. This required processing. We managed to stitch together great open-source projects to scale growing needs for insights, we modified multi-arm bandit algorithms to work online, and revenue grew —but so did the operational complexity of maintaining these systems. 

What we wanted — even if not apparent to leaders at the time — was a fully managed SaaS that could onboard the operational complexity while businesses scaled at a good price performance. The industry delivered the separation of compute and storage. We did get infinitely scalable systems that supported fine grained security controls, but with that, we also gave up ownership and access of the data. We no longer owned the data or the ability to access it in the most convenient way for the engineer, but instead with a specific vendor batch API, where dollar spend was a new explicit variable to the builder scaling these systems. 

Batch and streaming

Batch was the mental extension of the Unix pipe. Simple stages that one could reason about separately and debug. A simple mental model that scaled…slowly. In no small part was that we didn’t have the mental tooling of incremental computation engines like differential dataflow and DBSP, etc.

The systems community implemented a re-playable log of events with microbatching. Think of it as micro services, consuming and producing to stable APIs like RabbitMQ, Apache Kafka^®, Redpanda. When people were done with it, it felt like turning a database inside out, where most businesses looked like control plane databases or simply views of the log. 

“The truth is the log. The database is a cache of a subset of the log.”

I started Redpanda with the premise that batch was a historical artefact due to a lack of mental tools and, in part, a lack of intuitive industrial streaming implementations that offer a new way of reasoning about the world. Batch was a simple way to manage the vagaries of the real world of full buffer caches, slow and limited disk space, unreliable networks, etc. In general, batch offered a model to manage the resources available to the computers we were working with. 

However, the world is not nicely delineated hourly event-horizons where you simply accumulate the costs for your accounting systems and et-voilà. The world is filled with complexities of time, from late arriving events to the difference between system times and event times, to byzantine systems or attackers that will try to temper the authenticity of messages with manually curated timestamps to reorder sequencing.

Apache Iceberg™ and continuous computation

No one cares about batch and streaming differences today. For many years, we debated the mental model that streaming was a superset of batch once you added a cron-like timer callback. Last year, from customer pressure, the largest data warehouse products settled on Iceberg as an open-source data format. With that handshake, the world began thinking about data as a continuous computation. Use the lakehouse for backfill and the low latency stream for tailing iterators after you are caught up, but fundamentally push the complexity to the compute engines. 

Operational systems like Redpanda eliminated the need for engineers to care about batch or real time by providing zero-shot integration to Iceberg. More importantly, we elevated the mental model of what used to be a bag of bytes to one with schemas — and evolutions — that could be projected and consumed by warehouse technologies like BigQuery, Databricks, and Snowflake.

The engineers' new expectation should be to demand the convergence of complexity of batch and real time into the query engine itself, lifting once again the mental model to that of a continuous computation. Single team, single set of access management rules, onboarding and offboarding. A unified, open-source format to manage your data across all your vendors. Think of infrastructure vendors as materializations of the log or views from your data at rest. In addition to eliminating the complexities of multiple systems, you regain ownership of your data in an open source format to access data in the best way you see fit. Win win.

To empower code to be in control of its end-to-end process, you have to be able to glue all of this historical data access via a tool as well as to carry the low latency, operational context of the present. It isn’t enough to train a model on fraud detection, you must pass it your last five credit card transactions at the time of prompt and potential spending habits and credit score so it can make an informed decision and determine if fraud or non fraud. The rubber meets the road during integration.

Autonomy

Autonomy is the new skill the world has developed with artificial intelligence (AI), where code is in control and can direct its end-to-end process. Instead of sending a prompt to an LLM and getting a one-shot answer, you empower the model to make function calls, choosing which database to read data from, choosing the number of passes and how to divide and conquer tasks, etc. 

For the enterprise, autonomy comes with a very hefty price tag – sharing private data. Most large enterprises do not want to share data because that’s the moat. Our intuition has been that the future will be about sending model to the data, not data to the model for the bulk of the data that an enterprise processes. Smaller, open models like Llama3, Gemma3,  DeepSeekV3, Phi-4 provide state of the art performance for most tasks, and do it often at good enough quality in a single GPU. Beyond the price/performance improvement, it gives enterprises control of exactly what data is shared outside its firewall.

Autonomy is the codifying of business processes with private data along model advances. Think of it as public models already having consumed all of the public data and being suspended in their training because they didn’t have access to your specific data. You can unlock this frontier either by fine-tuning or enriching context via RAG and other similar techniques that give models access to the specific context for the task at hand. Private data is the thing that gives you an unfair advantage in the market.

We all want to innovate, but not at the expense of privacy, security, or regulation. Sensitive workloads demand end-to-end transparency of how data is transformed, used, and protected. Beyond the (often 10x less) cost advantage of purpose-specific, fine-tuned models on your data, there's no better custodian for your data than you.

Our thesis is that frontier models will also forever be valuable. The richness and flexibility they offer are simply unparalleled with anything that can be run by anyone outside of the major half-dozen AI Labs. However, you don’t have to choose one or the other. You can orchestrate single GPU models executing in your local network by a frontier model in a minion-task style delegation, allowing the local GPU to munch through the private information. In contrast, the frontier model orchestrates model routing and multi-turn agents with often much larger context windows and GPU power.  

Redpanda agents

Redpanda agents is a set of three new and composable open-source projects aimed at gluing together an unreasonably good developer experience for enterprise agents. 

1. rpk connect mcp-server exposes Redpanda Connect pipelines, resources, and processors as MCP (model context protocol, more on this later) tools for agents to query. In effect, it allows you to expose any redpanda connect source and destination as a tool with a simple configuration. Simply upgrade your Redpanda Connect and try it out.
2. A Python SDK for writing enterprise agents aimed at providing a seamless experience with Redpanda Connect tools, record tracing, durability, etc., via Redpanda brokers.
3. rpk connect agent is a new project that allows you to automatically wire MCP tools, exposing your Python agents via a gRPC local host command, dynamically injecting your MCP Server address, tool discovery, launching all the tools and Redpanda Connect pipelines and exposing them as simple HTTP endpoints. The Ruby-on-Rails experience for agents.

rpk connect agent run

The focus of the Python SDK project will be on durable execution, automatic logs and metrics collections, with opt-in full persistence. And, the ability to leverage Redpanda Connect (our ~300 connectors) exposed as an MCP server, with fully programmatic (code or declarative config) content filtering for ultra-fine grain access controls.

Our core pillars are:

1. Distributed log - Redpanda storage for durable execution, human-in-the-loop workflows, agent-to-agent communication, trace capture, evaluation replay, logs, metrics, collaborative threads, message sampling, analytics, explainability of actions, time travel debugging, etc. 
2. A new MCP Server project with ~300 pre-built connectors exposes all of your internal tools as a simple HTTP endpoint with a YAML config that manages all of the connection pooling, retries, exponential backoffs, TLS, certificates, authentication, etc.
3. A Python SDK [in preview] -  If you have used Pydantic agents or OpenAI agents, the hope is that you will feel right at home (please give us feedback). Our SDK brings about a consistent developer experience, focused on being standards based (with OpenTelemetry exposed for all the MCP pipelines today with the python runtime instrumentation soon), plug and play with Redpanda MCP declarative Pipelines (see rpk connect mcp-server), as well as seamlessly integrating with Redpanda's BYOC (Bring Your Own Cloud) deployment for sensitive data workloads that cannot leave your firewall. Seamless integration with identity providers, ACLs, Authentication, etc., when you connect it w/ the broker for multi-agent workflows and more. Check out the GitHub project for progress, roadmap, status updates, public docs, and more.

Although a picture is worth 1000 words:

For those looking to keep a consistent Python experience, you can write all of the existing Redpanda Connect YAML files in Python using the Starlark subset. It is effectively Python without imports, but more importantly, it is all Python so no need to learn a new configuration language. If you are a long-time Redpanda Connect user, you will have a consistent experience exposing MCP tools as good ‘ol pipelines that you can describe in YAML, including extensions, filtering, etc, using Bloblang.

‍

MCP - Model Context Protocol

An intent-based, simple HTTP endpoint wrapper to front the complexities of connectivity of internal systems, binary protocols, connection pooling, authentication, caching, etc.

rpk connect mcp-server

MCP is the integration layer for AI agents. An emerging standard that enables agent collaboration for accessing data across a large and diverse set of sources, like database users, LinkedIn mail, IT systems for onboarding, etc. And actioning results like sending emails, resetting passwords, unlocking your car remotely, etc. 

Integration and connectivity are the linchpins in the new world of agents because it makes all of your internal systems immediately queryable by AI agents — from wikis, code repos, documentation, sales, and other systems of record. Auditing, tracing, cost accounting, authentication, authorization, tracing, end-to-end logs from prompt to result, etc., can only be provided if a proxy layer mediates all access to all of your data. MCP inside your firewall allows you to leverage the breakthroughs safely. 

MCP is about intent, “create a Redpanda Cluster in us-east-1”, while the MCP server worries about implementing the five API calls. While MCP is often nothing more than the wrapping of custom protocols for databases, queues, caches, docs, GitHub, Salesforce; the additional layer of abstraction pushes these complexities away from the application that is focused on the linear distribution of the data, sampling, chunking, etc., — not on validating credentials or the correct SASL handshake. 

Most critically, centralization offers composability, understandability, and debugging that would have to be replicated by every agent otherwise. Many products’ APIs are brittle and evolution is riddled with conditional logic. An intent-based proxy pushes that complexity to a central location.

MCP future is full pipelines with dynamic content transformation and filtering. On the surface, wrapping a single Google Docs collection as a tool for agents to call may seem overly onerous. However, the world does not want to hand over full access to their kingdom to non-deterministic machines like these LLMs. While from a technical perspective MCP does offer centralized auditing, logging, tracing, security, API tokens, etc for the systems builder, the business wants ultra-fine grain control — HBase had cell access control list at the cell level for a reason. Control for MCP HTTP calls comes in the form of dynamic pipeline filtering, like so:

For Redpanda Connect specifically, it is the ability to leverage full programming languages via custom code extensions to give engineers the speed of iteration while letting the security team sleep at night, knowing they can enforce overriding global policies for ultra-fine-grain access to any of the ~300 connectors in a declarative fashion.

BYOC - Bring Your Own Cloud

Simply put, enterprises do not want to share their private data with anyone, especially so in the Fortune 2K. There will be exceptional cases that permit enterprises to share their data outside their firewall, but it is not the norm. 

Six years ago, we spent a year writing the first version of our cloud, then threw it away and wrote it again, which led to the birth of what we call BYOC. At the time, this decision was controversial because we were not charging incremental dollars to resell Amazon infrastructure, simply the value of the software we delivered. In a small way, we helped change how infrastructure is delivered and how we wish to consume it.

The goal was to have a single deployment model that we could build on top for years to come that would be compatible with the strictest security and compliance requirements. The main concept we came up with was Data Plane Atomicity. The premise was that no deployment should be able to bring down any other deployment, including a control plane outage. 

All failures on all data planes are independent for the data path while still providing a fully managed service. No externalized consensus algorithm, secret managers, no external databases, no external offset recording service, or metadata look up as you are trying to write your data durably to disk. At worst, you wouldn’t be able to click through an upgrade process, but your data path would always be up — hooray! (And without getting paged at 3 AM.)

Our insight was that since the dataplane is already deployed within the customer’s network, it is used to connect the internal applications already with Redpanda Connect’s portfolio.

We are beyond excited for the future. Join us

We are releasing the framework in preview for all our BYOC customers across three clouds. Stay tuned for an update on Redpanda Serverless. Give it a try on your local laptop. 

# brew upgrade redpanda
# update the connect plugin

rpk connect upgrade



# create a demo repo

mkdir demo && rpk connect agent init . 



# profit! 

rpk connect agent run 

Today, we are excited to announce that we’ve raised 100 million dollars in Series D to help us keep fueling this vision. We are incredibly proud and thankful to partner every day with everyone at Redpanda, our community of hackers, builders, and doers that continue to build with us. Special thanks to Muni at GV and Arif at Lightspeed for their continued support from inception; it takes a village. 

To learn more, sign up for our live Streamcast about the new Redpanda Agentic platform on April 29th at 9 AM PST.