Seventh Sense achieves the best “price to performance” for predictive analytics at scale

Challenge

Migrating to a streaming-first architecture

Seventh Sense is a predictive analytics platform that makes it easy for email marketers to segment and optimize marketing audiences through the use of AI. Seventh Sense gathers customer engagement data from marketing automation systems like Hubspot, analyzing the data with AI and ML to generate data-backed recommendations and insights. 

The company needed to migrate off its event capture and storage systems to a streaming-first model that would support processing hundreds of millions of record updates.

As a fast-moving startup, Seventh Sense needed to ensure the productivity of their developers and Operations team. They sought a streaming data platform that was not only fast and cost-effective but also easy to deploy, use, and manage.

Why Redpanda

Simplicity, performance, and cost savings

As part of its evaluation, Seventh Sense looked at Confluent Cloud but found the system’s pricing unsustainable. Between retention costs, throughput costs, and partition costs, the workload on Confluent Cloud was “going to get very expensive without great performance.”

Ultimately, they chose Redpanda due to:

• Operational simplicity
• Cost-efficient even at scale
• Better performance than Apache Kafka^®
• Unlocks powerful new use cases for Seventh Sense
• Seamless integration with Kafka-supported libraries
• Superior developer experience

“Redpanda’s performance caught our attention,” explains Erik LabIanca, CTO and co-founder of Seventh Sense. In a benchmark connecting Redpanda to their Databricks Spark cluster, Erik says the difference in performance of Redpanda versus Kafka was orders of magnitude. “We could completely saturate a hundred-to-one nodes on the Spark side versus the size of the Redpanda cluster.”

Results

Scaling to hundreds of millions of record updates

Seventh Sense needed a more scalable data pipeline, and the custom engine it originally built to store and manage ordered files on disk couldn’t keep up as the company grew from processing hundreds of thousands to hundreds of millions of records. Not only did Redpanda handle the data volume while maintaining near real-time performance, but it also significantly lowered costs for Seventh Sense.

“Redpanda ensures our API clients are operating at maximum performance,” Erik says. “Our query system wouldn’t be as cost-effective without Redpanda.”

Redpanda also supercharged Seventh Sense’s business by being an effective developer tool. Seventh Sense developers find Redpanda extremely easy to use and often run it right on their laptops. 

“Our devs can write integration tests and efficiently learn how to work with streaming data,” Erik says. “This is significant because event streaming is a new technology for our team.”

Plus, with Redpanda Cloud — Redpanda’s fully managed cloud service — the Seventh Sense operations team saves a generous amount of time and operational headaches. With more time and resources available, Seventh Sense can explore new use cases that were simply impossible before.