Goldsky simplifies streaming data for Web3 and blockchain developers

Challenge 

Empowering developers to build the future of Web3

Goldsky is a real-time data platform for the Web3 ecosystem that provides blockchain indexing, subgraphs, and data streaming pipelines that help developers build powerful decentralized applications. The trouble is that most app developers struggle with the complex technologies that power these streaming data pipelines.

For those unfamiliar with the Web3 world, Goldsky’s customers want specific data from different blockchains, which is a slow and tedious process. Goldsky strives to make streaming data accessible to developers by providing simple API-driven tools so they can bring real-time data into their applications — without having to manage complex underlying technologies like Apache Kafka^®.

Of course, a simplified streaming data engine is fundamental to this mission. So, Goldsky’s engineers needed a modern streaming data platform that had minimal complexity and was cost-effective but that could still handle high throughput reliably at scale.

Why Redpanda

Simple, cost-efficient, and solid data durability

Goldsky’s team had used Confluent Kafka in the past, but the pricing model didn’t suit their needs. Their next stop was Redpanda Cloud, which stood out for several key reasons:

• Unified platform with everything developers need built-in 
• S3-compatible Tiered Storage for near-infinite data retention 
• Minimal overhead and no external dependencies to manage
• Lower cloud infrastructure costs compared to Kafka
• Jepsen-tested, Raft-native architecture for guaranteed data safety

“Redpanda achieves the same throughput with less hardware and has a much simpler architecture,” says a principal software engineer at Goldsky.

Result

A reliable, highly durable “streaming-first” infrastructure 

Redpanda is now Goldsky’s primary streaming data solution, working seamlessly alongside Flink for stream processing. With Redpanda’s fully managed service, Redpanda Cloud, Goldsky ensures minimal overhead and up to 3-4x cloud infrastructure savings versus Kafka.

In Goldsky’s architecture, Redpanda isn’t just a broker; it’s where they store the blockchain data. It’s their source of truth — the primary database and data lake holding tens of terabytes of data. This is possible with Redpanda’s S3-compatible Tiered Storage, which provides near-infinite data retention without incurring high cloud infrastructure costs. It also eliminates the usual need for additional resources to maintain archival systems.

“This is an easy win for us: if our streaming data solution can also be our storage solution, why not?” Says a Goldsky engineer.

With cost-efficient Tiered Storage combined with a Jepsen-tested, Raft-native architecture, Redpanda’s rock-solid data safety and durability have earned it an essential role in Goldsky’s technology stack. 

Looking ahead, Goldsky is optimistic about Redpanda’s developer-focused roadmap, including Data Transforms and its Apache Iceberg integration, which will streamline Goldsky’s architecture further. In the meantime, Redpanda continues to power Goldsky’s mission of helping developers build real-time data pipelines for their exciting Web3 apps.