Learn how the rpk command line interface makes it easier to install Redpanda and manage your workflows.

ByDaisuke KobayashionMay 24, 2022
5 Ways Redpanda’s command line interface improves developer productivity

Introduction

Redpanda Keeper (rpk) is a single binary application written in Go to interact with your Redpanda clusters. It’s designed to be a Swiss Army Knife CLI application with simple commands that make it easy to use. It is handy not only for developers who want to evaluate Redpanda on their local machine, but also for developers who want to build, maintain, and debug a production streaming system using Redpanda.

This blog highlights five key benefits that rpk offers developers, outlined below:

  1. Easy installation and quickstart on a local machine
  2. Topic, group, and ACL management capabilities
  3. Rich formatting for producing and consuming
  4. Tuning the host to adopt production workload
  5. Easy configuration changes

1. Provides easy installation and quickstart on local machines

As mentioned above, rpk is a single binary application. It is installed with the Redpanda broker, which is contained in all Redpanda packages. It also provides autocompletion, which helps you interact with rpk easily using the tab button. Please review the rpk documentation for more information.

On Linux

If you want to run rpk in a Linux distribution, there are two ways.

To install rpk with the Redpanda server binary, here’s an example on Ubuntu:

$ curl -1sLf 'https://packages.vectorized.io/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash && \ sudo apt install redpanda -y && \ sudo systemctl start redpanda

To install the rpk binary independently,

  • Download the rpk archive on Ubuntu with: curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
  • Unpack the archive to /usr/local/bin or any location in your $PATH with: unzip rpk-linux-amd64.zip

Once it’s completed, run rpk version, which shows the rpk binary version, not the Redpanda server version.

$ rpk version v22.1.2 (rev 449eef2)

Follow the instructions in our Linux quickstart documentation, which will show you how to install rpk on Fedora/RedHat systems too.

On macOS

For individual developers who want to evaluate Redpanda on macOS, there are two ways to install rpk: via Homebrew or by downloading the binary.

To install rpk with Homebrew,

  • Run: brew install redpanda-data/tap/redpanda

To install the rpk binary,

  • Download the rpk archive with: curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-amd64.zip
  • Unpack the archive to /usr/local/bin or any location in your$PATH with:unzip rpk-darwin-amd64.zip

rpk also provides a way to spin up a cluster using Docker on your machine. To create a 3-node cluster, run:rpk container start -n 3, and you are ready to interact with Redpanda! For further steps, please follow the same instructions in the Docker quickstart documentation.

We're in the process of revising therpk container command to make it better. You can expect updated command usage in the future.

On Windows

We don't provide anrpk binary for Windows at this moment. Instead, follow the instructions in the Windows quickstart documentation, which utilizes WSL2 and Docker for Windows to setup a Redpanda cluster on your machine.

2. Simplifies topic, group, and ACL management capabilities

rpk gives you cluster management capabilities for topics, groups, ACLs, and more. For example, in order to manage consumer groups, rpk gives a group subcommand , where you can delete groups, describe groups, list groups, or seek to modify an existing group's offsets:

$ rpk group --help Describe, list, and delete consumer groups and manage their offsets. ... Available Commands: delete Delete groups from brokers. describe Describe group offset status & lag. list List all groups. seek Modify a group's current offsets. ...

rpk makes adding partitions to an existing topic easily:

$ rpk topic add-partitions test.blog --num 10 TOPIC ERROR test.blog OK

Just like groups, you can delete, describe, and list topics, as well, you can create topics with configurations. For example, the following command creates a new topic with 5GiB for retention.bytes, and confirms the topic configurations.

$ rpk topic create test.blog2 -c retention.bytes=5373952000 TOPIC STATUS test.blog2 OK $ rpk topic describe test.blog2 ======= NAME test.blog2 PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE ... retention.bytes 5373952000 DYNAMIC_TOPIC_CONFIG ...

rpk supports topic modification, and we take the viewpoint that incremental (KIP-339) config modification is more bullet proof and the only recommended way to modify configurations on modern brokers:

$ rpk topic alter-config --help Set, delete, add, and remove key/value configs for a topic.

This command allows you to incrementally alter the configuration for multiple topics at a time.

Incremental altering supports four operations:

  1. Setting a key=value pair

  2. Deleting a key's value

  3. Appending a new value to a list-of-values key

  4. Subtracting (removing) an existing value from a list-of-values key

Here’s an example:

$ rpk topic alter-config test.blog2 --set retention.bytes=10747904000 TOPIC STATUS test.blog2 OK

Here are all topic commands:

$ rpk topic --help Create, delete, produce to and consume from Redpanda topics. ... Available Commands: add-partitions Add partitions to existing topics. alter-config Set, delete, add, and remove key/value configs for a topic. consume Consume records from topics. create Create topics. delete Delete topics. describe Describe a topic. list List topics, optionally listing specific topics. produce Produce records to a topic. ...

ACL creation follows roughly the same flags as kafka-acls.sh. Listing and deleting use the same flags as creation, with missing flags defaulting to an all filter (word this better). We make deletion more bulletproof by prompting a confirmation for the ACLs you will be deleting:

$ rpk acl delete --allow-principal User:daisuke --user admin --password admin --sasl-mechanism SCRAM-SHA-256 MATCHES ======= PRINCIPAL HOST RESOURCE-TYPE RESOURCE-NAME RESOURCE-PATTERN-TYPE OPERATION PERMISSION ERROR User:daisuke * TOPIC * LITERAL WRITE ALLOW ? Confirm deletion of the above matching ACLs? (Y/n)

Once you type ‘Y’, the console updates like this:

? Confirm deletion of the above matching ACLs? Yes DELETIONS ========= PRINCIPAL HOST RESOURCE-TYPE RESOURCE-NAME RESOURCE-PATTERN-TYPE OPERATION PERMISSION ERROR User:daisuke * TOPIC * LITERAL WRITE ALLOW

Here are all ACL commands:

$ rpk acl --help Manage ACLs and SASL users. ... Available commands for ACLs: create Create ACLs. delete Delete ACLs. list List ACLs. user Manage SASL users. ...

For more information for available commands, please refer to the rpk documentation.

3. Enables rich formatting on producing and consuming

rpk provides rich formatting capabilities for both producing and consuming. Here are some parts of them as an example.

Create a topic:

$ rpk topic create my.test.topic TOPIC STATUS my.test.topic OK

Produce records from STDIN with specifying the record format as key and value with the-f option:

$ cat <<EOF | rpk topic produce my.test.topic -f '%k,%v\n' key1,value1 key2,value2 key3,value3 EOF Produced to partition 0 at offset 0 with timestamp 1649921155004. Produced to partition 0 at offset 1 with timestamp 1649921155004. Produced to partition 0 at offset 2 with timestamp 1649921155004.

Let’s consume the records. By default, rpk shows them in the following fashion, including their metadata:

$ rpk topic consume my.test.topic { "topic": "my.test.topic", "key": "key1", "value": "value1", "timestamp": 1649921155004, "partition": 0, "offset": 0 } { "topic": "my.test.topic", "key": "key2", "value": "value2", "timestamp": 1649921155004, "partition": 0, "offset": 1 } { "topic": "my.test.topic", "key": "key3", "value": "value3", "timestamp": 1649921155004, "partition": 0, "offset": 2 }

In order to print only the keys and the values, you can use %k and %v to represent keys and values in each record respectively:

$ rpk topic consume my.test.topic -f '%k,%v\n' key1,value1 key2,value2 key3,value3

By default, the timestamp field is printed as Epoch time number value. It can be printed with either

Go formatting or strftime formatting:

$ rpk topic consume my.test.topic -f '%k,%v,%d{go[2006-01-02T15:04:05Z07:00]}\n' key1,value1,2022-04-14T07:25:55Z key2,value2,2022-04-14T07:25:55Z key3,value3,2022-04-14T07:25:55Z $ rpk topic consume my.test.topic -f '%k,%v,%d{strftime[%Y-%m-%d %H:%M:%S]}\n' key1,value1,2022-04-14 07:25:55 key2,value2,2022-04-14 07:25:55 key3,value3,2022-04-14 07:25:55

You can pipe the consumer and the other produce sincerpk topic produce reads from STDIN. In the following example, it consumes keys and values frommy.test.topic and produces them into the other topic,my.production.topic.

$ rpk topic consume my.test.topic -f '%k,%v\n' | rpk topic produce my.production.topic -f '%k,%v\n'

rpk also provides binary encoding capabilities for consuming and producing and data unpacking on consume. The full formatting options are available with rpk topic produce --help or rpk topic consume --help.

4. Tunes the host to adopt production workload

Even when you’re ready to run a production workload on your Redpanda cluster, rpk is continues to be valuable. rpk has a capability of configuring your Linux host to optimize it to run Redpanda most efficiently. Here’s the command, followed by the available options along with brief descriptions.

rpk redpanda tune [command]
Command Description
all Tunes everything at once
aio_events Increases the maximum number of outstanding asynchronous IO operations if the current value is below a certain threshold, 1048576.
ballast_file Creates a large, unused file under /var/lib/redpanda/data to save from disk full
clocksource Sets the clock source to TSC (Time Stamp Counter) to get the time more efficiently via the Virtual Dynamic Shared Object
coredump Enables coredump
cpu Optimizes CPU settings to achieve best performance in I/O intensive workloads. More specifically this performs the following operations:
  • Disable Hyper Threading
  • Sets the ACPI-cpufreq governor to ‘performance’
  • Additionally if system reboot is allowed:
    • Disable Hyper Threading via Kernel boot parameter
    • Disable Intel P-States
    • Disable Intel C-States
    • Disable Turbo Boost
disk_irq Distributes block devices IRQs according to the specified mode. More specifically this performs the following operations:
  • Setup disks IRQs affinity
  • Ban the IRQ Balance service from moving distributed IRQs
disk_nomerges Disables merging adjacent IO requests.
disk_scheduler Sets the preferred I/O scheduler for given block devices and disables I/O operation merging. It can work using both the device name or a directory, then the device where directory is stored will be optimized. It sets either ‘none’ or ‘noop’ scheduler if supported.
disk_write_cache Tunes disk write cache in GCP deployments
fstrim Start the default fstrim systemd service, which runs in the background on a weekly basis and "trims" or "wipes" blocks which are not in use by the filesystem
net Distributes the NIC IRQs and queues according to the specified mode. This performs the following operations:
  • Setup NIC IRQs affinity
  • Setup NIC RPS and RFS
  • Setup NIC XPS
  • Increase socket listen backlog
  • Increase number of remembered connection requests
  • Ban the IRQ Balance service from moving distributed IRQs
swappiness Lowers /proc/sys/vm/swappiness to 1
transparent_hugepages Enables Transparent Hugepages

For more details of the commands, use the help command as follows:

rpk redpanda tune help [command]

If you want to dig deeper intorpk’s autotuning capabilities, check out this blog post.

5. Allows for easy configuration changes

In Redpanda’s latest major version update, Redpanda 22.1, our new centralized cluster configuration system is available and makes it easy to manage and edit the cluster-wide configuration using rpk. rpk centralized configuration carries the following subcommands:

\$ rpk cluster config --help Interact with cluster configuration properties. ... Available Commands: edit Edit cluster configuration properties. export Export cluster configuration. force-reset Forcibly clear a cluster configuration property on this node. get Get a cluster configuration property import Import cluster configuration from a file. lint Remove any deprecated content from redpanda.yaml. set Set a single cluster configuration property status Get configuration status of redpanda nodes. ...

In order to change the broker configuration, simply type rpk cluster config edit, which launches an editor (via $EDITOR) and allows you to edit the configuration on the fly. With this feature, you don’t need to manage the individual configuration files on each broker anymore.

What will you build with rpk and Redpanda?

In summary, rpk makes it easier for devs to interact with Redpanda clusters. Among other benefits, rpk makes it simple to:

  1. Install and start Redpanda on local machines
  2. Manage, topics, groups, and ACLs
  3. Use rich formatting for producing and consuming
  4. Tune the host to adopt production workload
  5. Make configuration changes

To learn more about rpk and how it can make your life simpler, join us on GitHub, or interact with us in our Slack community. For a more detailed list of how you can interact with rpk, view our rpk documentation.

Let's keep in touch

Subscribe and never miss another blog post, announcement, or community event. We hate spam and will never sell your contact information.