Building a reliable architecture for cybersecurity with Redpanda

In an era where cyber threats loom large and the stakes for organizations have never been higher, building a robust security intelligence platform is critical. But designing an effective security intelligence platform isn’t easy. It requires careful consideration of various components, technologies, and best practices to create a comprehensive defense ecosystem.

This post explores a reference blueprint of a security intelligence architecture based on existing SIEM (Security Information and Event Management) solutions and augmented by real-time streaming capabilities. We'll unravel the key elements, integration points, and crucial considerations that can empower organizations to detect, analyze, and respond swiftly to cyber threats.

What is a SIEM system and what is it used for?

Traditionally, enterprises depend on SIEM systems to detect security threats, investigate incidents, and mitigate potential risks.

These systems capture security events and logs from various sources via proprietary agents, including servers, network devices, applications, and firewalls. The captured data is aggregated in a centralized repository, enabling search and investigation, visualization, and setting alerts for detecting attacks. Some commonly used SIEM solutions are Splunk, Elastic, and Sumo Logic.

The challenges of deploying SIEM systems

While SIEM systems offer valuable analytics capabilities, organizations often face several challenges when implementing and utilizing them at scale. These challenges include:

Lack of real-time analysis: Traditional SIEM systems often operate on a batch processing model, where data is collected and analyzed periodically. This time delay between event occurrence and analysis can hinder the ability to respond promptly to security incidents in real time.

Scalability: SIEM systems may face scalability issues when handling large volumes of security events, especially in environments with high data ingestion rates. The storage and processing requirements can quickly become overwhelming, potentially leading to performance degradation or the need for costly hardware upgrades.

High rate of false positives: SIEM systems generate a significant number of alerts and events, many of which may turn out to be false positives. Sorting through these false alerts can be time-consuming and may divert resources from addressing genuine security incidents. It requires careful fine-tuning and customization to reduce false positives and improve the accuracy of alerts.

Limited flexibility: Having several security tools results in separate agent installations, storage systems, alerts, etc. All that contributes towards point-to-point integrations, which will reduce the flexibility of replacing a system in the future, potentially increasing the migration effort.

Adding real-time intelligence to SIEM

As organizations face the above challenges, we must rethink how we use SIEM. The ever-increasing amount of security data requires a new approach to SIEM that prioritizes timely responses, scalability, flexibility, and cost efficiency.

It's important to recognize that current SIEM solutions have been around for a while and have evolved into mature technology with a significant following. Also, building a SIEM system from scratch is a significant effort and demands a serious commitment from organizations. Thus, any alternative solution should complement SIEM systems rather than replace them. Our goal here is to augment SIEM capabilities by harnessing the power of real-time data and stream processing.

This is where Redpanda comes into the picture. To illustrate how it fits in, we'll walk you through a reference architecture for a security intelligence platform based on Redpanda and its streaming ecosystem. This reference architecture is driven by three key factors:

• Responsiveness: Address the shortcomings of batch-oriented SIEM systems by leveraging real-time data and stream processing to detect and respond to threats swiftly.
• Scalability: Accommodate high-throughput data ingestion from a variety of sources without being compromised on performance and reliability.
• Cost-efficiency: As we scale, we minimize costs by reusing existing ecosystem components and maintaining a lower hardware footprint.

Reference architecture for a cybersecurity platform

Now that you know the problem and the proposed solution's design goals, let’s walk through each component in the platform to identify its role and the value they add.

Redpanda - the streaming data platform for developers

Let’s start with Redpanda, which provides a central platform for collecting security events from various upstream sources and allowing downstream systems to process them in real time.

Overall, Redpanda adds these benefits to the solution.

• Scalable event ingestion: Redpanda enables high-throughput, low-latency event ingestion from upstream sources and acts as a buffer, preventing downstream systems from overloading with excessive amounts of data. Furthermore, Redpanda’s design ensures it can sustain high-volume ingestion rates without performance degradation or data loss.
• Cost-efficient event storage: Redpanda’s cloud-first storage engine treats cloud object stores as the default storage tier, enabling cost-efficient and elastic storage for security events in the long term. That means even the actively used topics are stored in the cloud, making this data available for a variety of purposes, such as Tiered Storage, remote read replicas, and disaster recovery, regardless of whether the original cluster is still running. It will also enable future advanced use cases, such as migration to other clouds and curation for downstream applications, making this architecture flexible and future-proof.
• Connectivity to Kafka ecosystem: Redpanda is compatible with Apache Kafka® APIs, enabling seamless integration with existing Kafka clients and ecosystem components, such as Kafka Connect and stream processing engines.

Kafka Connect and Kafka APIs

While Redpanda acts as the repository for real-time security events, we need to wrap the platform with standard-compliant API interfaces for easier integration among components.

In the Kafka ecosystem, Kafka Connect provides reusable integrations to Kafka in the form of Connectors. So we recommend using source and sink connectors of Kafka Connect to simplify the process of ingesting data into and extracting data from Redpanda. That includes integrating with collection agents, SIEM systems, and other destinations.

Kafka Connect also simplifies the management of data pipelines by handling essential tasks such as schema management, data transformation, and error handling. Connectors can automatically handle schema evolution, ensuring compatibility between data sources and sinks. Also, Kafka Connect allows you to focus on configuring and monitoring the data pipelines rather than building custom integration logic, reducing development time and effort.

Collection agents

An agent can be a software or hardware component that can collect security events and forward them to Redpanda. Agents typically source events from network traffic, server and application logs, and firewall logs over protocols, including Syslog and SNMP.

Traditionally, SIEM systems used proprietary agents for event collection. In this new architecture, we recommend reusable and standard-compliant agents that can directly produce events to Redpanda over Kafka API interfaces and Kafka Connect.

SIEM system

The SIEM system uses either Kafka Connect or the Kafka client API to receive security events from Redpanda, detect potential threats, and trigger real-time alerts for further actions. Mainstream SIEM systems like Splunk, Elastic, and SumoLogic can ingest events over Kafka Connect Source connectors, allowing them to be used with Redpanda as well.

Once ingested from Redpanda, the SIEM system indexes the events, allowing security professionals to investigate security incidents by providing interactive search facilities. Further, the indexed data can be visualized in dashboards providing a holistic view of the organization's security posture. For instance, a SIEM tool, such as Elasticsearch, ingests data from Redpanda via Elasticsearch Sink Connector and uses Kibana dashboards to set up alerts and visualizations.

Stream processors

Traditionally, the SIEM systems ingested events directly from the agents. Events coming from agents are noisy–often contain events that do not contribute to incident detection and investigation. That results in SIEM systems raising alarms with too many false positives.

So, we can leverage stream processors—an additional processing layer on top of SIEM systems—to cut through the noise. Stream processes can pre-process events arriving from Redpanda before they are ingested into the SIEM system.

The platform can leverage Kafka-compatible stream processors, such as Apache Spark™, Apache Beam®, Kafka Streams, and Apache Flink®, for stateful and stateless stream processing. Stateless operations include filtering, routing, and transforming events, while stateless operations can join events with external sources for enrichment. For example, a stream processor can add contextual information to events received from an external threat intelligence platform.

Pre-processing security events cuts noise from signals, helps to derive better insights, and has a low hardware footprint on storage and computational resources downstream.

Long-term event retention

Once captured, storing security events in long-term storage is crucial as they serve as flight recorders for organizations.

Many organizations are subject to regulatory compliance standards that mandate the retention of security logs and events for a specified period. Long-term retention ensures that organizations can meet these compliance requirements and provide the necessary audit trails when requested. Also, historical events help comprehensive forensic analysis and investigation of security incidents. It enables security analysts to perform historical data analysis, identify patterns, and uncover indicators of compromise that may have been missed during real-time monitoring.

In terms of implementation, the platform can benefit from a cloud object store, such as S3, for cost-efficient and scalable long-term storage for events. Alternatively, a data warehouse can be considered as well. Cloud platforms, including AWS, Google, and Azure, offer sink connectors on Kafka Connect, allowing events to be moved from Redpanda to long-term storage.

Enabling AI/ML use cases

The platform can stream security events through a machine-learning pipeline for real-time model training and scoring, enabling the building of behavioral profiles of users, systems, and network entities within the organization.

Once trained, the ML models can be deployed alongside the SIEM to detect anomalies of security events in real time.

Conclusion

By powering their SIEM with streaming data, organizations can confidently fortify their cybersecurity defenses and navigate the digital realm with heightened confidence.

By adding Redpanda, the platform benefits from unparalleled scalability in event ingestion. Its Tiered Storage and efficient hardware resource usage help keep costs minimal in the long run. Stream processing, combined with SIEM systems, enables real-time detection of threats and responding to them in a timely manner. Furthermore, by gluing the components with Kafka-compatible APIs, including Kafka Connect, the platform takes advantage of reusing battle-tested data sources and sinks.

Interested in trying it for yourself? Take Redpanda for a spin! Check out our documentation and browse the Redpanda blog for tutorials on how to easily integrate with Redpanda.

If you get stuck, have a question, or just want to chat with our engineers and fellow Redpanda users, join our Redpanda Community on Slack.

‍