Bring Your Own Cloud (BYOC) is an upcoming deployment model that gives you a managed Redpanda streaming data cluster in the comfort of your own VPC. This means managed provisioning, streamlined cluster management, and automated backups. It also means user data never leaves your environment. Ever.
When shifting from batch to real-time data, the challenge is often operationalizing the tech stack. With our existing streaming data solutions, customers had to choose between the simplicity of having a managed service or the flexibility to run in their own environments for data privacy, security, or governance requirements. Time and time again, we were asked whether it would be possible to solve both needs at once. So we started working on a BYOC option in order to offer the best of both worlds.
BYOC separates the control and data plane, such that the data plane—Redpanda—lives inside your VPC, and the control plane lives in our cloud. This provides the advantages of a fully managed streaming data cloud experience—streamlined resource provisioning, managed availability and scaling, automated upgrades and backups, observability and alerts, along with optimized and self-healing infrastructure. But since it runs in a VPC that you own, you get the benefits of a self-hosted cloud deployment—owning the data plane from a security or privacy perspective, meeting specialized governance or compliance requirements, and taking advantage of any existing public cloud architecture and billing agreements you might have in place.
For a Vectorized customer, it all begins with creating a cluster through the cloud UI and selecting the BYOC option in the cloud provider (today AWS and GCP) and region of your choice. After selecting options including node types, storage, and cluster connectivity, we provide a Terraform script and YAML manifest that will create a specially configured managed Kubernetes cluster (e.g., Amazon EKS or Google GKE) within your own VPC. Next, it will deploy and configure a local agent that receives commands from our Vectorized Cloud control plane, finishing up the configuration needed for the Redpanda cluster. Voilá, you now have a managed streaming data platform running within your own VPC!
From a user perspective it looks straightforward, but building BYOC required significant engineering design and effort. One of the clear requirements was to maintain customer privacy and security by ensuring Vectorized has only the least possible access to the VPC needed to orchestrate the Redpanda cluster. To facilitate this, we started by maintaining a strong separation between our management control plane and your data plane in your VPC. We run a local agent microservice in the customer Kubernetes deployment. You send commands via UI, API, or CLI through Vectorized Cloud; these commands are routed to our management control plane (Ring0 Redpanda). The local agent running in your BYOC cluster polls Ring0 and fetches the commands, sending any feedback from the cluster. The cluster also exports metrics and utilization data to the control plane.
Vectorized has no access to your underlying cloud infrastructure, and no user data leaves yours network. We want to highlight that only system metrics—CPU, Mem, Disk, etc.—are shared with the control plane. That way we can detect, alert, and automatically heal clusters (in the case of failures) transparently to the end-user, which in our case tends to be the application developer. All of this ensures no intrusive access (like custom firewall rules, etc.) to customer deployments, meeting strict security and compliance regulations.
Bring Your Own Cloud has been an instant hit within our community. One of our first customers for BYOC was data connectivity platform LiveRamp, which needed a Kafka API-compatible streaming data platform with high performance and operational simplicity, but strongly desired the dual benefits of a managed service with the compliance requirements of running in their own cloud.
“Redpanda gives us the simplicity and scale we need for our real-time systems without compromising performance and Kafka compatibility. The BYOC approach in Vectorized Cloud gives us a managed service running on our own cloud servers, balancing our internal compliance requirements with ease of use. I am excited to partner with this expert team to modernize our systems for our real-time use cases.” – Kannan DR, Enterprise Data Architect, Liveramp
The Bring Your Own Cloud model provides customers with the best of both worlds. It’s a managed Redpanda service that handles provisioning, orchestration, and management of their streaming data needs, while running on a Kubernetes cluster in their VPC to meet requirements for security, control, and compliance. We are beginning to open up BYOC to the wider market. If this deployment model matches your needs, sign up below. We’d love to talk with you.