Streamlining test driven development and CI testing for Kafka developers

Test-driven development has gained popularity among developers as it gives developers instant feedback and can identify defects and problems early. Once the application is developed, during continuous integration (CI), it’s also important to run automatic tests to cover all possible scenarios before it gets built and deployed to detect defects and issues early.

Apache Kafka® provides a distributed, fault-tolerant streaming system that allows applications to communicate with each other asynchronously. Whether you are building microservices or data pipelines, it allows applications to be more loosely-coupled for better scalability and flexibility. But at the same time, it also introduces a lot more complexity to the environment.

This post will cover how to solve the complexity problem by introducing Quarkus, Redpanda, and Testcontainers in a demo that showcases all the components in action.

Test Driven Development (TDD)

TDD is an iterative process that involves writing a test that specifies the desired behavior of the code, then testing your code during and after development. This cycle is repeated continuously to produce clean and reliable code.

Quarkus

Quarkus and JUnit5 set your testing framework. Simply add the dependencies and define your test case with @QuarkusTest and @Test annotations. Then code with Quarkus in dev mode, which allows you to make changes to the code and see the results immediately—without having to restart the application or rebuild the container. This significantly improves productivity and enables continuous testing with the press of a button.

Run mvn quarkus:dev to start the dev mode. You will be prompted with following instructions after dev mode starts:

--
Tests paused, press [r] to resume, [h] for more options>

After making changes to your application while in the mode, test the application by typing r to rerun the entire test sets. You will get instant test feedback.

2023-04-17 10:15:40,311 INFO  [io.qua.test] (Test runner thread) All tests are now passing
--
Press [r] to resume testing, [:] for the terminal, [h] for more options>
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.343 s
[INFO] Finished at: 2023-04-17T10:15:43-04:00
[INFO] ------------------------------------------------------------------------

Set up the development environment

Setting up the environment for development is not difficult, but it does require some patience and work to get all the components set up. For example, to get Kafka running as the streaming platform, it requires knowledge to configure Apache ZooKeeper™ and the brokers and constantly recreating topics for the clean state—along with other databases.

Testcontainers

Testcontainers allow you to define the testing environment in code, ensuring that the environment is consistent and repeatable across multiple runs. Testcontainers works everywhere with access to a Docker environment both in your CI platform and locally.

Even if you don't manage a local Docker installation, Testcontainers Cloud allows you to run containers during Testcontainers tests giving you plenty of resources and simplifying parallelization. There are a wide range of container images that can be used.

Testcontainers create and manage the containerized testing environments. It automatically pulls and starts the configured container services, and tears down the environment. If you have multiple tests running, the resources needed will eventually add up and use up more memory. Even worse, it’ll take more time to get things ready. A bulky streaming system is not ideal.

Redpanda

As a streaming data platform for developers, this simple binary all-in-one form (broker, schema registry, and HTTP proxy) with extreme performance and super lightweight has won over many Kafka developers’ hearts. In fact, Quarkus by default uses Redpanda as a streaming platform for their dev service. Using Redpanda will save you both time and resources to run all tests.

Using Testcontainers and Redpanda, you can define a fast, lightweight streaming service container where an instance of Redpanda broker will be created and run for testing. Simply add the following line of code:

RedpandaContainer redpanda = new RedpandaContainer("docker.redpanda.com/redpandadata/redpanda:latest");

Sometimes you want to share a Redpanda broker instance between multiple tests, so it’s best to allow QuarkusTestResourceLifecycle to take of it. It will call Testcontainers to start up before the test and let it destroy the containers once all tests are done.

Automate testing in Continuous Integration (CI)

Tests should always run during the CI process automatically to ensure that code changes do not break existing functionality or introduce new bugs. But testing Kafka applications can be challenging since most CI runs on separate platforms such as Jenkins, CircleCI, Gitlab, or GitHub. It’s difficult to manage and integrate a Kafka cluster in those environments.

This problem can be easily solved with Testcontainers spinning up lightweight and fast containerized Redpanda brokers for testing. It’s using the same codebase and configurations you set while testing locally. You would not need to change a line of code and push the changes you made for CI. Note that Github Actions and many CI platforms by default refresh the container images for every run. The size of the image will impact the time to run the pipelines. At the time of writing this blog: Redpanda has the size of 129 MB compared to others at 414 MB.

As part of the test, just add the following step to your CI, (we’re using Github Action in this case):

 - name: Build 
      run: mvn build
 - name: Test 
      run: mvn test

The demo

This is a simple application that instantly identifies the age of customers as their data comes in. All customer information is streamed into the “customer” topic, the application will simply filter the underage customer list into a special topic “underage”.

This demo was created with the TDD approach:

1. Define the desired behavior in a test
2. Using Testcontainers to configure the Redpanda
3. Write the application and run the test to verify if it has the expected result

Once it’s done, both the test and application code are committed to the repo, and the continuous integration pipeline will rerun the test to make sure the code changes do not break existing functionality or introduce new bugs.

To set up Redpanda as yours in your test folder, create a new class that implements QuarkusTestResourceLifecycleManager, this manager will manage the lifecycle of the services needed for testing.

And the following command for better testing experience, by default, Redpanda will allocate a fixed amount of CPU core and memory, which is ideal for production but for development and CI environment with limited hardware capacity, Testcontainer runs the Redpanda broker in development mode, where the memory and CPU limits are not enforced, and turning off fsync() for even faster streaming.

private static DockerImageName REDPANDA_IMAGE = DockerImageName

.parse("docker.redpanda.com/redpandadata/redpanda:latest");

Lastly, create a test. In Quarkus, make sure to add the QuarkusTestResourceLifecycleManager to the test by adding the annotation to the Test class.

@QuarkusTest
@QuarkusTestResource(TestResource.class)
public class TestAgeRestriction {...}

Watch our quick video showing how TDD works with Quarkus, Redpanda, and Testcontainers. how TDD works with Quarkus, Redpanda, and Testcontainers You can find all the code in this GitHub repository.

Summary

TDD requires you to write the tests before coding. Running tests can be challenging for Kafka developers, but Quarkus, Redpanda, and Testcontainers made continuous testing possible.

Developers no longer need to set up separate Kafka instances for development and CI. Testcontainers will initiate independent fast and lightweight Redpanda brokers in a container which are smaller and about half of the size compared to Kafka. The same unit tests can be re-used in the CI pipeline without the need to reconfigure and eliminate the need to set up separate brokers just for testing.

To get started with Redpanda, check the documentation and browse the Redpanda blog for tutorials.

To try it for yourself, take Redpanda for a test drive! If have any questions or just want to chat with the team and fellow Redpanda users, join the Redpanda Community on Slack.

‍