Lacework scales data streaming 10x and saves 30% on cloud costs

Challenge

Expanding to new clouds without increasing costs

Lacework, a leading cloud security platform, provides continuous monitoring and protection across the entire application lifecycle. To secure its customers’ infrastructure, Lacework collects and processes massive volumes of data from multiple sources, including logs, monitoring data, and security events.

“Every aspect of the Lacework platform depends on real-time data flow to protect our customers,” says Chip Turner, engineering director at Lacework.

However, as the company acquired more customers and expanded to new clouds (like Google Cloud), its proprietary streaming solution struggled to handle the data-intensive throughputs. Not to mention the complexity of managing the system and incurring higher costs without delivering the necessary performance.

Lacework needed a more scalable, reliable, and efficient solution to deliver high performance for its rapidly growing 1 GBps+ streaming data workloads — without the high costs.

Why Redpanda

Scalability, reliability, and efficiency

As the Lacework team considered its options for a new, cloud-agnostic streaming data solution, the most critical factor was finding a platform that could handle high throughput with predictable latency while meeting strict service-level objectives (SLOs) and availability requirements. Ultimately, they chose Redpanda due to its:

• Performant architecture written in C++ with intelligent memory handling
• Apache Kafka^® API compatibility to migrate without rewriting code
• Cloud-agnostic to flexibly run across different cloud environments 
• Freedom from JVM for easy deployment and low operational overhead
• Enhanced fault tolerance, high availability, and disaster recovery

“We conducted a thorough evaluation of streaming data options and Redpanda was the clear winner,” Chip explains. “In fact, during our benchmarking we hit the limitations of our benchmarking tool, but Redpanda was barely breaking a sweat.”

Plus, Redpanda provided flexible deployment options that wouldn’t lock Lacework into a single vendor or cloud provider, supporting Lacework’s future growth as a multi-cloud organization.

Result

Efficient scaling and reduced complexity

As its first use case, the Lacework team built its streaming data stack around Redpanda — an architecture the team affectionately dubbed “Pipe Dream,” since it “looked too good to be true.”

“With Redpanda and our new architecture, we have new levels of operational agility that are game-changing,” Chip said. “We can meet our stringent SLOs while keeping our developers focused on high-value work versus managing data infrastructure.”

With Redpanda at its core, Lacework has achieved a 10x increase in scalability, with 14.5 GB/sec of data seamlessly flowing through Redpanda at peak loads. To put this speed into perspective, consider a typical 4K movie, which is about 20–30 GB. At 14.5 GBps, you can download it in less than two seconds.

Since starting with Redpanda in 2021, Lacework has grown its CPU cores footprint by more than 1,200% and now relies on Redpanda as a hub of data exchange between its different analytical systems and microservices, trusting it to help deliver security insights back to Lacework’s customers.

However, this is only the beginning of Lacework’s journey with Redpanda. Now that the organization’s data pipelines are running hyper-efficiently, Lacework’s next priority is migrating to a managed Kubernetes environment with Amazon EKS, leveraging Redpanda’s Tiered Storage to help reduce storage consumption on expensive EC2 nodes.

“With Redpanda’s Tiered Storage, we save up to 30% or more on storage costs. As a partner for streaming data, Redpanda helps us control costs as we grow usage.”

As Lacework embarks on an infrastructure modernization, it will certainly take Redpanda along for the ride.