A not-so-technical start-up guide for Redpanda with Node.js.

ByMateo CarvajalonDecember 15, 2020
Getting started with Redpanda using Node.js

In this blog, I’m going to show you how to set up Redpanda and use it with Node.js & TypeScript. We will build a very simple terminal chat room where anyone can join. This is an introduction to Redpanda, so if you already have experience with it or similar streaming platforms, you can check out some of our posts on more advanced topics, like this one on consistency, or this one on autotuning.

Node.js set up

If you haven't already, install Node by following the appropriate steps for your OS here.

As you may have heard by now, Redpanda is Kafka® API-compatible, which means that although Redpanda is relatively new, you can leverage the countless client libraries created for Kafka® (if you find something that is not supported, reach out to our team on our slack. In this case we will use kafkajs.

#create and enter the project folder mkdir redpanda-node cd redpanda-node #generate package.json npm init #install needed dependencies npm i -D typescript npm i -D @types/node npm i kafkajs npm i uuid npm i -D @types/uuid #generate tsconfig.json tsc --init

Setting up Redpanda and creating a topic

Follow the Redpanda setup for your OS:

Note: please follow only this section. If you do the production section, it will optimize your machine for Redpanda, which might affect your experience with desktop applications and other services. Be especially careful with sudo rpk tune all you probably don’t want to run that on your personal workstation.

After Redpanda is installed and running, you will create the first topic

$ rpk topic create chat-room --brokers <comma_separated_list_of_brokers> Created topic 'chat-room'. Partitions: 1, replicas: 1, configuration: 'cleanup.policy':'delete'

This will create a topic named chat-room, with one partition and one replica. You can also see all created topics with:

$ rpk topic list --brokers <comma_separated_list_of_brokers> Name Partitions Replicas chat-room 1 1

With that out of the way, we can get started. To have a working chat we need to receive and send messages, so we need to create a consumer and a producer.

Producer set up

// src/producer.ts import {Kafka} from 'kafkajs'; const kafka = new Kafka({ clientId: 'chat-app', brokers: ['127.0.0.1:9092'] }); const producer = kafka.producer(); export function getConnection(user: string){ return producer.connect().then(() => { return (message: string) => { return producer.send({ topic: 'chat-room', // the topic created before messages: [//we send the message and the user who sent it {value: JSON.stringify({message, user})}, ], }) } }) } export function disconnect(){ return producer.disconnect() }

That’s it, a working producer, sending strings entered by the user. Keepin mind that you can send buffers, meaning you can send pretty much anything you want.

Consumer set up

// src/consumer.ts import { v4 as uuidv4 } from 'uuid'; import {Kafka} from 'kafkajs'; const kafka = new Kafka({ clientId: 'chat-app', brokers: ['127.0.0.1:9092'] }); const consumer = kafka.consumer({ groupId: uuidv4() }); // we need a unique groupId I'll explain down export function connect() { return consumer.connect().then(() => consumer.subscribe({topic: 'chat-room'}).then(() => consumer.run({ eachMessage: async ({topic, partition, message}) => { const formattedValue = JSON.parse((message.value as Buffer).toString()); // everything comes as a buffer console.log(`${formattedValue.user}: ${formattedValue.message}`)// print the message }, }) ) ); } export function disconnect() { consumer.disconnect(); }

There you have it. This will get all produced messages and print them. You can have as many consumer groups as you want, but bear in mind that each group will get a message only once. That’s why we have the uuid generated.

Putting everything together

//src/index.ts import readline from 'readline'; import * as Producer from './producer'; import * as Consumer from './consumer'; const rl = readline.createInterface({ input: process.stdin, output: process.stdout }); function start() { console.log('connecting...') Consumer.connect().then(() => { rl.question('enter user name \n', function (username) { // the username to print it along with the messages Producer.getConnection(username).then((sendMessage) => { console.log('connected, press Ctrl+C to exit') rl.on('line', (input) => { readline.moveCursor(process.stdout, 0,-1); // removing the input so you don't get duplicated items in terminal sendMessage(input); }) }) }); }) } start(); // handling shut down process.on('SIGINT', process.exit); process.on('exit', () => { Producer.disconnect(); Consumer.disconnect(); rl.close(); });

Running

tsc && node src/index.js

Run this as many times as you want clients. At least 2 so you can chat between 2 terminals.

Conclusion

Now you have the basic building blocks to work with Redpanda. Try it yourself, there are endless use cases - what we built here was just the simplest of examples.

Stay tuned, we will soon have a cloud where we will handle all the Redpanda technical details, so you can just focus on developing your product. In the meantime, please feel free to reach out to us via Slack or GitHub Discussions.

Join Our Newsletter

We’ll send you a helpful update every so often, but no spam.