New York Stock Exchange powers smarter trading decisions with real-time market data

Challenge 

Real-time data offerings drove up cost andcomplexity

The New York Stock Exchange (NYSE), which dates back to 1792, is the largest stock exchange in the world based on the total market capitalization of its listed securities, trading over 2.5 billion shares and processing over 500 billion messages across all of its markets. It was the first formalized stock exchange established in the United States.

A significant part of NYSE’s business involves selling market data to provide hedge funds, high-frequency traders, and retail and individual investors with accurate, real-time information to inform strategic trades. NYSE’s exchange data offerings span five exchanges—including five equities trading platforms, two options trading platforms, and one bonds trading platform—to provide investors with the necessary transparency to make sound decisions.

“To put things into perspective, Google averages about 16 billion queries a day. We handle nearly 1.1 trillion records daily, and 99% of the time, we make a decision in less than 20 microseconds. So scale and latency are absolutely critical to our success,” said Vinil Bhandari, Head of Cloud & Full Stack Engineering.

Continuing its multi-year cloud journey, the NYSE Group wanted to expand its data offerings to include real-time data, plus historical trade and reference data available on AWS Cloud. However, consuming real-time streaming data directly from NYSE required customers to build, manage, and maintain on premises environments or work with a third-party provider for data delivery — a time-consuming and challenging process.

“When we started looking at other streaming solutions, none of the solutions met our needs relating to IO, throughput, and latencies,” Vinil explained. “We required a platform that could deliver stock exchange data with high throughput and very low latency across the financial market almost instantaneously, while allowing for easy integration directly into customer applications.”

Why Redpanda

Innovative data architecture and trusted engineering support

The NYSE selected Redpanda to power NYSE Cloud Streaming, a data distribution platform that enables customers to access real-time market data with sub-hundred-millisecond latency in the cloud.

With NYSE Cloud Streaming, customers can spin up an environment in AWS without the cost or hassle of physical infrastructure, allowing retail broker-dealers, fintech companies, and other organizations to access streaming data directly in the cloud at a lower total cost of ownership.

“In our industry, people talk about nanoseconds and microseconds. Traders need to be on alert at all times. Elon Musk makes a tweet and the markets go haywire,” said Vinil.

He explains that all the flavors of Kafka they evaluated didn’t fare well with bursty traffic since most are built on Java, which impacts the latency profiles. Redpanda, in contrast, has rewritten the Kafka protocol in C++, making it ideally suited for unpredictable workloads and easily scales to meet changing needs.

Besides Redpanda’s innovative architecture, NYSE also highly values the Redpanda user experience and documentation, allowing the organization to self-serve on a regular basis. When higher levels of support are needed, Redpanda is always there to lend a helping hand—engineers collaborating with engineers.

“We found Redpanda’s team very easy to work with. My staff appreciates teaming up with Redpanda engineers. They are not support engineers. They are not documentation engineers. They’re engineers,” said Vinil.

Results 

Empowering customers to make smarter investment decisions

“Redpanda is the only Kafka-compatible data streaming platform that can cost-effectively and reliably manage the massive scale of the NYSE. When milliseconds matter, Redpanda delivers real-time market data so that our customers can make important investment decisions and manage risk,” said Vinil.

Redpanda reduces cost and complexity by providing NYSE:

• Better control over memory and CPU usage
• Lower latency, no JVM, no garbage collection
• Major resource savings with Redpanda Console, skipping the need to build a monitoring front-end
• Tiered storage to allow keeping data in blob storage (e.g., S3)
• Access to the Redpanda Connect ecosystem of 300+ connectors
• A RESTful interface for CRUD operations on the cluster
• rpk command line utility to help with cluster management

The biggest benefit that NYSE business teams are realizing as a result of Redpanda is now a prospect or lead can be onboarded within hours or days.

“Now, whether you’re sitting in Timbuktu or Hong Kong, we can easily open up a data access point via the power of the cloud and immediately provide customers with data wherever they reside across the world,” said Vinil.

As a result, NYSE trusts Redpanda as a strategic partner to deliver real value in actual real time, enabling NYSE’s consumers to get crucial data when millions of dollars are at stake with high-volume trades