Getting started with Apache Camel and Redpanda

Welcome to this introductory guide for new Camel riders and streamers! In this post, we'll show you how to write your first Camel route using the latest JBang release to quickly integrate an API to an Apache Kafka® endpoint—without the hassle of defining Maven or Gradle.

Let’s start by introducing the technologies we’ll be using in this tutorial.

Redpanda

With the high demand for real-time data and the explosion of microservices, organizations need a much more sophisticated network for data to flow seamlessly between systems.

This is where a data mesh architecture comes in. And, a streaming data platform is the backbone for the mesh, since it enables a high throughput of data with low latency, and its asynchronous nature makes it modular and scalable.

Redpanda is a Kafka-compatible streaming data platform that's 10x faster and 6x lower in total costs compared to Kafka. It's JVM-free, ZooKeeper®-free, Jepsen-tested, source available—and my personal preference because it's also easy to use. There’s no need to remember if ZooKeeper needs to be up first or the broker, and vice versa when shutting things down.

Apache Camel

The Apache Camel project is famous for solving connectivity and integration problems between applications with a more configurative approach. It includes component libraries and implements enterprise-ready patterns without the need for developers to reinvent the wheel.

The project has been around for more than a decade, so it already has established and well-tested patterns for developers to leverage. It continues to evolve with the introduction of the latest technologies, like Quarkus for runtime, as well as JBang for ease of development and up-to-date components that developers will need.

A huge reason for Camel’s success is that integration continues to be the biggest challenge for data management. Not to mention it’s also a great implementation framework for creating data pipelines.

JBang

In a nutshell: JBang is a powerful tool that makes it easy for Java developers to quickly write, test, and run small scripts and utilities without having to set up a development environment or project.

Now that you’re familiar with the technologies, let's dig into the example.

Example: Streaming air quality data in real time

This simple integration example retrieves aggregated and harmonized real-time air quality data across the globe. The data is a collection of metrics from various locations that need to be stored and recorded.

It also has to split the list into individual locations, providing the separated single location air quality data for other microservices. Here’s a quick overview of the architecture:

This example will use Apache Camel to create a route that pulls from an Open API endpoint every 30 seconds. It'll also take advantage of the built-in patterns to orchestrate, reformat, and send the data to the desired destination.

Both the list and individual location data will be sent to separate topics in the streaming data platform. This provides better scalability and will support various consumption needs. Due to its developer-friendly nature, you can use Redpanda to quickly spin up a local cluster. And, with its compatibility with the Kafka API, you can easily integrate Camel using the Camel Kafka component.

Setting up

Camel is based on Java. Traditionally, developers would build out a full-blown project with structured folders and dependencies specified. We will be using the JBang-based Camel integration for running Camel routes like scripts. This allows developers to focus on building and iterating quickly.

To start, we’ll install the JBang CLI tools. Follow the instructions to Install JBang, then check your JBang installation by running the following:

jbang version
0.100.0

Install Apache Camel command-line tool within JBang:

jbang app install camel@apache/camel

Check the Camel CLI installation by running:

camel --help

Running Redpanda and its console

There are several ways to install Redpanda, such as installing with the command line tool rpk, or running it with Docker. (For additional install and setup examples, see our quickstart documentation.)

Since this is a simple demo for a local development, nothing wild is going to happen to the cluster, so we can simply run a one-node cluster. For this example, we chose Docker to provision Redpanda and the Redpanda Console.

docker run -d \
--pull=always \
--name=redpanda \
--hostname=redpanda \
--net=redpanda \
-p 8081:8081 \
-p 8082:8082 \
-p 9092:9092 \
-p 9644:9644 \
-v "redpanda1:/var/lib/redpanda/data" \
vectorized/redpanda redpanda start \
--overprovisioned \
--smp 1  \
--memory 1G  \
--reserve-memory 0M \
--pandaproxy-addr INSIDE://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082 \
--advertise-pandaproxy-addr INSIDE://redpanda:28082,OUTSIDE://localhost:8082 \
--kafka-addr INSIDE://redpanda:29092,OUTSIDE://0.0.0.0:9092 \
--advertise-kafka-addr INSIDE://redpanda:29092,OUTSIDE://localhost:9092 \
--rpc-addr 0.0.0.0:33145 \
--advertise-rpc-addr redpanda:33145

‍The Redpanda Console shows you information about deploying, configuring, and even the data streamed. Setting up the Redpanda Console is optional, but it makes working with Redpanda much easier. If you don’t use the Console, you can still administer it using Redpanda command-line tools.

docker run --network=redpanda \
  --name=console \
  -e KAFKA_BROKERS=redpanda:29092 \
  -p 8080:8080 \
  docker.redpanda.com/vectorized/console:latest

Running Camel

First, clone the sample code from this GitHub repository:

git clone https://github.com/weimeilin79/camelpanda101.git
cd camelpanda101

Run the Camel application with the following command:

camel run AirQualityConsumer.java

The AirQualityConsumer.java is the Camel route process with the Splitter and WireTap Enterprise Integration Patterns (EIP).

Splitter breaks out the composite message into a series of individual messages, as shown below:

.split().simple("${body[results]}")

WireTap routes the split messages to a separate pm-data topic, while the original data is forwarded to the all-query topic. This pattern is commonly used for testing or auditing, where the same input needs to be sent to multiple destinations.

wireTap("kafka:all-query?brokers=localhost:9092")
...
to("kafka:pm-data?brokers=localhost:9092");

Similar to the EIPs, the Camel route also helps to transform binary data into textual format with the unmarshal feature:

.unmarshal().json()

Results

To see the resulting message, go to Redpanda Console (http://localhost:8080) select the Topic on the left menu, and you should see two topics listed:

• all-query
• pm-data

Click pm-data, and you’ll see all the Air Pollution data from individual meters. It’ll look something like this:

Next, do a quick search to find the city that interests you. You can do this by entering a city name in the search box next to the Actions drop-down menu.

To export the Camel route into a traditional Java-based project (in this case, Quarkus), simply run the following command:

camel export --runtime=quarkus --gav=org.rp:camel:1.0 --directory=./camelproject

Lastly, under the folder camelproject, there’ll be a Maven-based Quarkus project with files organized in the following manner:

Conclusion

In this tutorial, we covered how to create a simple Camel route integration to orchestrate data with a pre-built EIP from an API endpoint. With JBang, we can develop and test the Camel route like a scripting language. Additionally, we set up Redpanda as the streaming data platform in just a few seconds, and monitored incoming streaming data with the Redpanda console.

Want to keep learning? Take Redpanda's free Community edition for a spin! Make sure to check our documentation to understand the nuts and bolts of the platform, and browse the Redpanda blog to learn all the cool ways you can integrate Redpanda.

If you have questions or just want to chat with our architects, engineers, and other Redpanda users, join the Redpanda Community on Slack.

‍