Introducing Redpanda Labs - a hub for streaming data examples and experiments

Redpanda is a simple, reliable, and cost-efficient streaming data platform used to power high-performance, mission-critical applications—whether that’s for an enterprise or a side project on your laptop. Over the past year or two, we have seen a proliferation of GitHub projects popping up that use Redpanda in super interesting ways, and we thought it would be nice to start collecting all of these examples in one, easy-to-reference place: Redpanda Labs!

Redpanda Labs is the home for examples, experiments, and research projects that intend to showcase the power of Redpanda as the centerpiece of your streaming data architecture. Some of these projects may make it into the product itself, and some will not. What Redpanda Labs will do is provide examples, guidance, best practices, and most importantly, give you ideas for how you can use Redpanda in your own projects.

Getting started

Redpanda opens up the door to building exciting new streaming data applications that were simply not possible before. To kick things off, we have shared four Labs projects:

1. Data Transforms

The first comes fresh off of the back of the recent Data Transforms Sandbox announcement. Data Transforms takes a radically more efficient approach to stream processing by allowing developers to bring the majority of their data processing tasks in-broker, eliminating the “data ping-pong” associated with stream processing frameworks deployed externally to Redpanda.

Simple tasks like cleansing, scrubbing, filtering, and transcoding can be developed and compiled into WebAssembly (Wasm) modules, and then deployed to Redpanda to run directly on your topics. The Labs Data Transforms project contains a set of data transforms that you can modify, build, and deploy to your cluster.

2. Redpanda Edge Agent

The second project is Redpanda Edge Agent. One of the many benefits of Redpanda is its lightweight form factor. Being a compact binary application that can run on x86_64 or ARM platforms with no external dependencies (no JVM here), means that Redpanda works really well at the edge. It can just as easily be deployed to a bunch of small servers running on the factory floor, as it can be deployed as a single node on a resource-constrained IoT device that is connected to a cellular network.

Redpanda is great at collecting and storing events in these environments, but until now, the missing piece of the puzzle has been how to reliably forward those events from the edge to a central Redpanda cluster. After all, you can’t run MirrorMaker on an IoT device. Redpanda Edge Agent is the answer. It’s a small Go application that can be compiled for x86_64 or ARM and can run alongside Redpanda to reliably forward events from the edge to an upstream Kafka API-compatible cluster.

3. Redpanda Clients

The third project houses a few sample Redpanda clients that are available in different programming languages, including Python, Go, Java, and Javascript. These clients have minimal yet essential information for you to connect to Redpanda, produce messages, and then consume them back.

4. Redpanda Schema Registry samples

The fourth project includes a Python sample demonstrating the use of Redpanda Schema Registry. The sample mimics a clickstream event producer and a consumer written in Python, leveraging the built-in Schema Registry for serializing and deserializing clickstream events as Avro messages.

Excited? Get involved with Redpanda Labs

The best part about Redpanda Labs isn’t just that it’s a dedicated space for the exciting projects we’re working on—it’s also open for input from the Redpanda Community! To get involved, you can simply contribute to existing projects or even share your own. Just fork the repository and send us a PR.

To learn more about Redpanda, check out our upcoming events and on-demand resources. If you haven’t already, join the Redpanda Community on Slack.

‍