Anyone who's ever attempted vegetable gardening knows that it’s far easier to succeed with already-growing plants than ones you start from seed. Literally growing something from the ground up takes a lot of time, work, and careful monitoring. The same can be said of building software.
I’ve worked at both large and small companies in the past but, in both types of settings, my work as an engineer was already determined: the software already had roots; it was already in the ground, so to speak. As an early member of the Redpanda team, however, I’ve witnessed and contributed to the passion and care required to build software from scratch. Specifically, I was the first person at Redpanda to work on engineering our cloud platform.
Today, I’m a Site Reliability Engineer at Redpanda. I joined the team in the early days, back in September of 2020. I live and work from my home country of Greece, collaborating daily with the rest of our fully distributed remote team via Zoom and Slack.
In this post, I’ll tell you about what it’s been like engineering our cloud from the ground up, as well as explain what our cloud product offers to our users.
The origin of Redpanda’s cloud
My first few months at Redpanda were spent ramping up our internal CI/CD process for the Core product, as that was where we needed to focus our attention at the time. However, as time passed, we knew more customers would want to have the option of infinite scalability offered by a cloud platform, and so our cloud initiative was where I began devoting all of my time. Building our cloud into what it is today has been both fun and challenging. When you’re building something completely from the ground up, there are a lot of decisions to be made, and you want to make them all quickly so you can keep moving the project forward and engineering the next phase of the platform.
The main challenge we faced was creating a cloud platform product from scratch. In an early-stage startup, objectives and requirements can change quickly, and you have to be agile in your execution to keep up with and meet demands. Because we were engineering our cloud from nothing, we also essentially needed to be the first consumers of our product while we were building it out. This is challenging because you need to use the product as you’re creating it. In the end, though, we built a cloud product that we’re proud of and that we feel provides a good experience for our cloud customers.
FMC vs BYOC
As we built out our cloud, we realized that many of our customers wanted the ability to choose between having us manage their cloud for them, or bringing their own virtual private cloud (VPC) to Redpanda. To give customers these options, we created our Fully Managed Cloud (FMC) and Bring Your Own Cloud (BYOC) offerings. To put it simply, FMC allows you to host your data in Redpanda and have Redpanda handle the provisioning, operations, and maintenance for you. BYOC, on the other hand, allows customers to deploy Redpanda in their own VPC, keeping all data in their own environment, while still having Redpanda provision, monitor, and handle upgrades for them. Having a cloud offering makes the scaling possibilities with Redpanda infinite and, with these options, we can ensure customers can scale in a way that suits their preferences.
Cutting-edge technologies in the cloud
As a core product, Redpanda is a cutting-edge platform to work on because it is able to make the most of modern hardware resources, and so it can perform faster than other data streaming platforms because it was specifically designed to take advantage of modern computing capabilities. For C++ engineers and developers working on the Core team, that’s an interesting kind of technology to be able to work on.
From the cloud perspective, if you’re a member of our Cloud or SRE teams, you get to work on the latest technologies being used in cloud platforms today, such as developing with Kubernetes or tiered storage in mind. You also get to operate, tweak, and work with Redpanda, which gives you the chance to be curious and experiment in your engineering work on a daily basis. The majority of developers don’t get to work with the software they’re building this much at their day jobs, so it’s a very rewarding experience. In terms of the tech stack we use, I spend most days working with Terraform, Kubernetes, Grafana, Prometheus, and Golang, and some Bash magic is usually in there, too.
Building site reliability engineering into the Cloud team
In the first iterations of the cloud team, I was essentially the person working on all things infrastructure and platform, contributing to product decisions, and also developing small features. As the team scaled, we eventually formed a separate site reliability engineering (SRE) team within the Cloud team, which made divying up responsibilities a bit more manageable with the creation of specific focus areas. More recently, we’ve been working toward defining how reliability impacts our cloud product, levelling up our monitoring game, and defining a robust incident response process for Redpanda as a whole.
Building a reliable and fast cloud platform to host, stream, and transform data is, in my opinion, as fascinating as it gets engineering-wise. My advice for anyone interested in working on Redpanda’s Cloud team would be to come to the opportunity with, obviously, a background in engineering and the relevant technologies required there, but to also come with a sense of curiosity for what you can build.
Working for Redpanda has enabled me to be involved in many different aspects of what it means to develop a solid cloud product, from performance engineering to platform reliability to data management. In a world where real-time data streaming is becoming the norm, I’m excited to be working with a team at the forefront of that and to see what the future of streaming holds. If this sounds interesting to you too, take a look at the Cloud team positions we’re hiring for right now!
Join Our Newsletter
We’ll send you a helpful update every so often, but no spam.