How Jump Trading drives faster insights at scale with Redpanda

Jump Trading is a proprietary trading firm with a focus on algorithmic and high-frequency trading strategies. To be successful we tap into real-time insights—so streaming data infrastructure is critical to our business.

Over the past 20 years, Jump Trading iterated through several messaging and streaming data platforms. Our primary use cases are telemetry and log files. We have evaluated and deployed most of the best-of-breed on-premises and cloud messaging systems. In doing so, we found all of them lacking due to some combination of:

• Consistency of performance (tail latency, tolerance of slow consumers, etc.)
• Access to low-level metrics to debug performance issues (especially in cloud products)
• Reliability of message delivery
• Cost-effectiveness
• Compatibility with other products

As Jump Trading grew, our workloads required us to process millions of messages per second, and it became clear that we needed a new streaming data platform that could support this mission-critical aspect of our infrastructure.

The search for a simplified streaming data platform

When looking for a new solution, our starting point was Apache Kafka®. We like the Kafka protocol because it's the industry standard for data streaming. Lots of data tools speak Kafka, which makes it much easier to engineer pipelines without reinventing integrations.

However, we did not like having a Java application in such a critical role. We didn't want memory behavior and allocation governed by a Java Virtual Machine (JVM); we wanted memory tailored to the application itself.

We found that Redpanda satisfied that requirement through its implementation in C++. Since Jump Trading is built around C++ software architecture, Redpanda's codebase aligned well with our core software competencies.

With Redpanda, we get Kafka compatibility along with stability, simpler operations, and a straightforward platform footprint. Plus, there's no ZooKeeper to manage. And while Kafka has evolved beyond ZooKeeper, we still preferred Redpanda's lean, self-contained design for a performance-critical deployment.

By eliminating the Java dependencies common in Kafka deployments, Redpanda delivered less complexity and superior performance on like-for-like hardware.

Furthermore, the simple architecture of Redpanda as a single binary and no JVM dependencies created a better day-to-day experience for our engineers. And all this, plus strong durability guarantees from the Raft-native design. The choice was straightforward.

‍

Today, we run Redpanda on bare metal, in containerized environments like Podman and Kubernetes, and via the Redpanda fully-managed cloud service. Communication and collaboration with Redpanda's support and engineering teams were a key factor in keeping us on board, especially early in adoption. We pushed the platform hard, surfaced edge cases, and saw fixes land quickly as the product matured.

The keys to Redpanda’s success

It’s worth spending a little more time delving into the three key benefits that have made Redpanda so popular within Jump Trading: simplicity, performance and open source compatibility.

Simplicity

We appreciate Redpanda's single binary installation, rolling upgrades designed to avoid planned downtime, and streamlined maintenance cycle. The architecture is leaner than Kafka to deploy and scale for several reasons:

• No external dependencies: We don't need to install, monitor, or maintain ZooKeeper or KRaft nodes.
• No JVM tuning: We don't need to deal with JVM compatibility issues, tuning, or upgrade planning.
• Built-in observability: The built-in Prometheus exporter gives us visibility into performance internals to steer system load.

From a getting started perspective, we also appreciate the documentation and blog posts available on the website.

It's essential for our engineers to build on open standards versus proprietary ones, because it means less time learning "yet another" system and more time shipping. That's why we were drawn to Redpanda's Kafka-compatible API. It gives Jump Trading access to the entire existing open-source Kafka ecosystem.

When companies build on open standards, they benefit from the body of knowledge and tooling already out in the world, which shortens time to productivity. And because we're not locked into proprietary tech, we can work across environments and platforms without absorbing unnecessary vendor risk.

In theory, we could swap Redpanda for any other compatible Kafka platform if our needs changed. In the same manner, we can use Redpanda with Kafka-compatible tools, and integrate alongside other ecosystems when it's the right fit, like the Amazon Kinesis Client Library.

In addition to Kafka compliance, it's great to see Redpanda's object-storage integration for cost-effective retention. Redpanda's S3-compatible storage feature is a huge win because there's already a large ecosystem of tooling built around S3.

Performance

One of Jump Trading's largest workloads is our mission-critical telemetry pipeline. It's complex, data-intensive, and heavily producer-oriented. Our many-to-few pipeline pattern is fully supported and highly efficient with Redpanda.

The workload consists of:

• Massive scale: Tens of thousands of nodes publishing data.
• Varied data types: System measurements, market information, and other telemetry.
• Real-time consumption: High-performance nodes consuming and processing data instantly.

Because of Redpanda's performance-engineered architecture, we see little jitter in our p95 and p99 latencies. Tail latency stays tight, without the long, unpredictable spikes that make systems hard to operate. Redpanda's C++ codebase means we can drive it to line rate and still expect packets and messages to land within a well-behaved latency distribution.

That matters, especially for a latency-sensitive financial firm. In an industry with bursty traffic, it's invaluable to have a message bus that holds predictable tails under load. With Redpanda, we're seeing low latencies: 50 milliseconds for P95 and 150 milliseconds for P99.

Open-source compatibility

Redpanda is a source-available platform, which aligns with our values. We believe in open-source software development and actively support many open-source projects. Having the codebase available for inspection helps us understand the system in depth, assist in troubleshooting when it makes sense, and participate as a partner in the product's evolution.

Going above and beyond with Redpanda

We started with a self-hosted deployment of Redpanda on bare metal, then added containerized deployments, and are now expanding into fully-managed, dedicated cloud instances. As of now, we have been running critical workloads on Redpanda for two years, reliably shipping billions of messages daily.

Jump Trading is a leader in global financial markets and our streaming data infrastructure can make or break our business. With Redpanda, it’s not just meeting our requirements, but taking us to the next level. — Alex Davies, CTO, Jump Trading. Originally posted on The New Stack.

For more stories about how Redpanda is helping companies tap into simple, powerful, and cost-efficient data streaming, check out the customers page and browse the Redpanda Blog for examples and tutorials. You can also try Redpanda to see it in action for yourself.