Chess.com trusts Redpanda as its “data backbone” to power +10M games daily

Challenge

Escalating Confluent costs and poor dataplatform performance

Chess.com is the world’s leading online chess platform, offering a wide range of features including live and correspondence games, instructional content, puzzles, tournaments, and community forums. Launched in 2007, it has grown to over 200 million users worldwide, becoming a central hub for both casual players and professionals.

The platform saw a surge in popularity during the COVID-19 pandemic and again following the release of The Queen’s Gambit on Netflix, further amplified by collaborations with top streamers and grandmasters. With regular events, engaging content, and a strong social component, Chess.com hosts over 10 million games daily and remains the go-to destination for playing and learning chess online.

Apache Kafka® is a critical component of Chess.com’s infrastructure, powering the platform’s architecture at a massive scale. From asynchronous communication to infinite storage, Chess.com requires an event-driven architecture that is easily scalable, highly available, and capable of handling terabytes of streaming data per day to serve its players and community. One example of its scale: Chess.com offers an exploration service that allows anyone to view any game played on the platform over the past 20 years — totaling more than 30 billion records.

Chess.com has been using Confluent Cloud for several years, initially operating on standard clusters. However, due to escalating networking costs associated with the standard configuration, the team migrated to dedicated clusters to improve performance management.

To reduce its total cost of ownership, Chess.com prefers to host most of its data infrastructure in-house. However, Confluent’s service offerings weren’t conducive to running on-prem in Chess.com’s infrastructure and Google Cloud Platform (GCP). Furthermore, the growing scale of the business would result in increasing costs year after year.

Chess.com encountered additional challenges, including an inability to establish a private peering connection with Confluent Cloud. To enable this, Chess.com needed to create an entirely new cluster and migrate all workloads, only to have Confluent charge for both the original and new clusters, significantly increasing operational costs.

James Kelty, Head of Infrastructure, recalls, “If we were going to do a migration, we needed to decrease costs. Then we decided we wanted to bring things on-prem, and Confluent simply didn’t have an offering.”

Chess.com sought a cost-effective solution that could reside within its GCP environment, eliminating overages for network charges and bandwidth between peers. At Kelty’s previous companies, running Kafka or Kafka-like buses on-prem required a large team of people to manage its complexity. Chess.com lacked the necessary expertise and personnel to support it internally, making Redpanda’s Bring Your Own Cloud (BYOC) solution particularly appealing.

Kelty sums up why Chess.com made the switch to Redpanda: “Confluent was exorbitantly expensive, with costs climbing year over year. It wasn’t sustainable. Redpanda offered a solution that was performant and cost-effective — we couldn’t pass it up.”

Why Redpanda

BYOC delivers low costs, predictability, infinite scale, and fasterdevelopment

Chess.com selected Redpanda and its BYOC solution to efficiently and seamlessly migrate from Confluent, while maintaining business continuity and rapidly developing new features for its devoted community.

“Because of the growing cost of Confluent, we were being forced to migrate, which takes significant time away from development. We would have had to completely stop building new features for about six months and 100% focus on bug fixes during the migration. We just couldn’t afford that. We were caught in this catch-22 scenario and Redpanda got us out,” said Kelty.

Redpanda BYOC provided a performant, low-latency solution at a significantly lower price point, allowing Chess.com to execute its migration thoughtfully and efficiently.

“Redpanda offered us flexibility, we could take our time for more complex topic migrations versus being forcedto speedrun topics with Confluent. Increasing our utilization of Redpanda up to scraping 90% of the cluster itselfdoesn’t increase cost, whereas with Confluent, it would,” Kelty explains.

“Let’s say somebody created a topic and it wasn’t well thought out, creating a huge amount of data traffic between our VPCs and GCP, we would see an unexpected cost spike. The idea of building a new feature that could inadvertently trigger expensive overages was incredibly frustrating. Redpanda is predictable and reliable; we don’t see surprise overages anymore.”

Chess.com also reports that Redpanda’s customer service has been “amazing.” As a remote-first company, Chess.com relies on Slack for internal communication. Redpanda has been integrated as part of the team at Chess.com and responds to inquiries almost immediately.

Additionally, when Chess.com conducts infrastructure maintenance or migrates topics, they have a predictable understanding of any potential impact on latency or performance. Redpanda has consistently delivered performance equal to or better than Confluent, regardless of the topic and schema creation.

‍

Results

Outstanding user experience, accelerated new feature development, andcommunity growth

Data is Chess.com’s greatest asset, and according to Kelty, “Redpanda is our backbone, holding 20+ years of data. We trust Redpanda with our family jewels. It’s incredibly important that we don’t lose that data, we can easily access it any time, it doesn’t cost us an arm and a leg, and it’s manageable and able to infinitely scale.”

Redpanda’s BYOC data platform enables a myriad of business benefits for Chess.com, including:

• A/B testing of new features, driving accelerated development for the community
• Posting all chess moves and scores immediately after every gam
• Near real-time player ratings and leaderboard updates
• Allowing players to challenge others to match and accept instantly
• Powering “vote chess” where, for example, grandmaster Magnus Carlsen plays the entire world, with 125,000+ players casting votes on what move to make to try to beat the several-time world chess champion
• Detecting fraudulent use of chess engines to identify cheaters and ban accounts

In terms of cost and performance compared to Confluent, Chess.com reports that it has more than two times the capacity with Redpanda at a significantly lower price point, with predictability and no network overages to worry about.

In fact, Chess.com has cut costs by at least 40%, which translates to more than $400,000 in annual savings. Redpanda also contributes to Chess.com’s growth strategy as the organization is able to increase the speed of new feature development without cost concerns.

“Redpanda is at the core of our development cycle. Based on user data stored in Redpanda, we use analytics toanalyze trends and hone in on what our community wants. Then, we can quickly develop new features withoutworrying about making minor configuration mistakes that previously could have cost thousands of dollars,” saysKelty

Redpanda also provides Kelty and his team with peace of mind knowing that its data platform can handle massive traffic spikes from world events – like the surge that followed the release of chess series Queen’s Gambit on Netflix, a global moment when chess interest spiked overnight.

“We have a passionate and fast-growing fan base, so we can’t afford for our infrastructure to crumble. Redpanda not only supports our steady growth, but also delivers an amazing user experience when thousands of new players flood the site,” says Kelty.

“With Confluent, unexpected traffic spikes came with massive bills. Now, we are prepared for whatever comes our way without being penalized for our success.”