Equal Experts improves data infrastructure to support demand for real-time applications

Challenge 

Architecting data infrastructure with strict latency requirements

Equal Experts is a global IT services company with over 3,000 consultants focused on bringing simple solutions to big business problems via digital products and services. One of Equal Expert’s FinTech clients was building a flexible, low-latency market data alerting system, and needed help architecting a streaming data system that could meet their stringent latency requirement: an average end-to-end latency of tens of milliseconds.

Equal Experts initially proposed a real-time technology stack including Apache Kafka^®, Kafka Connect, Apache Flink^®, KsqlDB, and Eclipse Vert.x. The team began reviewing a major managed Kafka service, but while it couldn’t hit the latency target, it did hit the cost limit very quickly. 

As a result, Equal Experts began looking for other Kafka alternatives that could integrate easily with the client’s existing technology stack, keep costs down, and achieve the extremely low latencies required for effective real-time alerting. 

Why Redpanda

Performance, efficiency, and a minimal learning curve

When the previous managed Kafka solution failed, the client’s architect suggested Redpanda as a Kafka-compatible alternative with a smaller deployment footprint, no dependencies, and a C++ architecture designed to maximize the performance of existing hardware.

Naturally, the Equal Experts team benchmarked Redpanda against their baselines with Kafka. “The conclusion was obvious,” said Eric Laguer, Principal Cloud Architect at Equal Experts. Ultimately, the team chose Redpanda for the following reasons:

• Consistently low end-to-end latencies 
• Lean, unified design for simpler maintenance 
• Full Kafka API compliance for easier integration
• Minimal learning curve thanks to Redpanda’s support
  

Eric also notes that migrating from Kafka to Redpanda was fairly simple. “We were actually able to use the same versions of Java client libraries that we were using with Kafka,” he says. Combined with the quick support from Redpanda’s engineers, switching to Redpanda was a “quick win” for Equal Experts.

Results

Real-time alerting powered by a best-in-class streaming data platform

With Redpanda, Equal Experts successfully architected a solution that met the client’s rigorous latency requirements for their real-time alerting system use case. Where previous solutions ran up to 30ms latency, Redpanda came in at just 5ms. 

“Redpanda achieved the low latencies that are usually only possible when running Kafka with no replication (a risky design choice for production workloads),” Eric explains. 

Using Redpanda’s Self-Managed deployment option, Equal Experts has granular control over their infrastructure, allowing the team to optimize their system wherever possible. For example, they wanted to run on EC2 versus Kubernetes to leverage the ARM-based Graviton instances in AWS. Doing so delivered a 20% performance boost—plus a notably lower cost versus standard EC2 instances.

As the industry continually shifts to real-time data streaming, Redpanda enables Equal Experts to help its clients improve their data infrastructure to support more demanding real-time and event-driven applications.