TLS configuration for Redpanda and rpk

Prerequisites

In order to set up TLS on Redpanda and rpk, you’ll need a certificate and a key. If you require client authentication, you'll also need a truststore file containing the client certificates.

1. TLS configuration in Redpanda

Redpanda supports two levels of TLS:

• Basic TLS: Encrypt all incoming requests using a certificate and a key file.
• TLS with client authentication: In addition to encrypting all incoming requests, require that the clients include their certificate, checking that it's trusted.

These can be set up for the Apache Kafka®-compatible API, the Redpanda Admin API and the internal RPC API, which is used for communication between all redpanda nodes.

redpanda:

  # Kafka® API configuration
  kafka_api:
  - name: default
    address: <private IP>
    port: 9092

  advertised_kafka_api:
  - name: default
    address: <public IP or domain>
    port: 9092

  kafka_api_tls:
  - name: default
    enabled: true
    require_client_auth: true
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted key file>
    truststore_file: <path to PEM-formatted truststore file>

  # Admin API configuration
  admin_api_tls:
    enabled: true
    require_client_auth: true
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted cert file>
    truststore_file: <path to PEM-formatted cert file>

  # RPC (Internal) API configuration
  rpc_server_tls:
    enabled: true
    require_client_auth: true
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted cert file>
    truststore_file: <path to PEM-formatted cert file>

There are many important things going on here, so let's unpack them.

First off, we're declaring a listener for the broker.

  kafka_api:
  - name: default
    address: <private IP>
    port: 9092

name can be any string you like. The important part is that it matches the name field in redpanda.advertised_kafka_api and redpanda.kafka_api_tls.

address is any IP that the broker can bind to.

Then, we're setting the advertised - or "published" - address.

  advertised_kafka_api:
  - name: default
    address: <public IP or domain>
    port: 9092

If you're trying out this guide locally (i.e. in the same machine), address can be the same as the one we set for the listener in redpanda.kafka_api. However, for a production deployment or if you wanna interact with the broker remotely, it can be the machine's public IP or its domain, if one exists.

After that, there's the TLS config for the listener we declared:

  # A list of TLS configurations for the Kafka API listeners.
  kafka_api_tls:
    # The name of the specific listener this TLS to which this config
    # will be applied. The names must match those in kafka_api.
  - name: default
    # Whether to enable TLS for the Kafka API.
    enabled: true
    # Require client authentication
    require_client_auth: false
    # The path to the server certificate PEM file.
    cert_file: <path to PEM-formatted cert file>
    # The path to the server key PEM file
    key_file: <path to PEM-formatted key file>
    # The path to the truststore PEM file. Only required if client authentication
    # is enabled.
    truststore_file: <path to PEM-formatted truststore file>

admin_api_tls and rpc_server_tls share the same structure as the one above.

Even though you can configure the Kafka API TLS (redpanda.kafka_api_tls), the Admin API TLS (redpanda.admin_api_tls), and the RPC API TLS (redpanda.rpc_server_tls) with the same certificates, we recommend that you use different sets of certificates for each one for improved security.

If you set any of the APIs' require_client_auth field to true, you'll need to set the respective truststore_file. Also, the PEM truststore file it points to should contain the clients' certificates for client authentication.

Besides manually editing the configuration file to configure TLS, you can also use rpk to do it (you may need to run these as root if the config file is restricted):

rpk redpanda config set redpanda '
kafka_api_tls:
- name: default
  enabled: true
  require_client_auth: true
  cert_file: <path to PEM-formatted cert file>
  key_file: <path to PEM-formatted cert file>
  truststore_file: <path to PEM-formatted cert file>

admin_api_tls:
  enabled: true
  require_client_auth: true
  cert_file: <path to PEM-formatted cert file>
  key_file: <path to PEM-formatted cert file>
  truststore_file: <path to PEM-formatted cert file>

rpc_server_tls:
  enabled: true
  require_client_auth: true
  cert_file: <path to PEM-formatted cert file>
  key_file: <path to PEM-formatted cert file>
  truststore_file: <path to PEM-formatted cert file>

' --format yaml

Note: After changing the configuration you'll need to restart the Redpanda broker (e.g. by running sudo systemctl restart redpanda, if you're using systemd).

2. TLS configuration in rpk

To be able to use rpk topic with a broker with TLS enabled, you'll need to update the rpk config in your config file.

For the Kafka API:

rpk:

  kafka_api:
    brokers:
    - <broker address>

    # The presence of the tls field lets rpk know it should enable TLS
    tls: {}

If the server's certificate is self-signed, you'll also need to set rpk.kafka_api.truststore_file so that it points to a PEM-formatted file containing the certificates of the brokers to which you'll make requests:

rpk:

  kafka_api:
    brokers:
    - <broker address>

    tls:
      truststore_file: <path to PEM-formatted truststore file>

If redpanda.kafka_api_tls.require_client_auth was set to true in the broker, you will need to set the client certificate and key:

rpk:

  kafka_api:
    brokers:
    - <broker address>

    tls:
      cert_file: <path to PEM-formatted cert file>
      key_file: <path to PEM-formatted key file>

For the Redpanda Admin API:

The Redpanda Admin API is used by rpk in commands such as rpk acl create and you configure it much like rpk.kafka_api. However, the Redpanda Admin API requires that the Admin API addresses for all of the nodes are listed:

rpk:

  admin_api:
    addresses:
    - <node 1 Admin API address>
#   - <node 2 Admin API address>
#   ...
#   - <node n Admin API address>

    tls:
      cert_file: <path to PEM-formatted cert file>
      key_file: <path to PEM-formatted key file>
      truststore_file: <path to PEM-formatted truststore file>

It's strongly encouraged that the certificate used for rpk and other clients be different from the ones used in the brokers.

Changing the configuration in the rpk section doesn't require restarting Redpanda, so you may test your settings right away with any of the rpk topic subcommands, such as

rpk topic list --brokers <broker IP>:<port>

2.1 TLS configuration via the CLI

You can also set the TLS configuration using the specific flags in rpk topic commands: --tls-enabled, --tls-cert, --tls-key & --tls-truststore. For rpk acl commands, --admin-api-tls-cert, --admin-api-tls-key & --admin-api-tls-truststore are also available for the Redpanda Admin API-specific TLS configuration.

3. Configuring your current Kafka client

The best news is, since Redpanda is API-compatible with Kafka, you probably won't need to modify your TLS settings unless you changed the location or format of any of the files involved.

security.protocol=SSL
ssl.truststore.location=<path to truststore file>
ssl.truststore.password=<truststore password>
ssl.keystore.location=<path to keystore file>
ssl.keystore.password=<keystore password>
ssl.key.password=<key password>

Conclusion

Redpanda leverages TLS to ensure encrypted communication, both internally on its RPC API, as well as externally on its Kafka-compatible API listeners. Additionally, you can require client authentication via mutual TLS to make sure only known clients are able to make requests.

You can configure rpk and any other clients to be able to make encrypted requests to a remote node’s API.

Don't forget to follow @redpandadata on Twitter, or join our Slack community, and stay tuned for more step-by-step guides from our team!

‍