Comparing web user interfaces for managing Apache Kafka

Kafka web UIs are graphical user interfaces (GUIs) that provide access to Apache Kafka® clusters. They allow users to configure, monitor, and manage Kafka from a web browser, providing a powerful and intuitive way to interact with it. If you're used to the reliable command line interface (CLI), you may be wondering if it's worth using a web UI to manage Apache Kafka.

The short answer is: yes, definitely.

Among the advantages of these UI tools over traditional CLIs are:

• Convenience: Kafka web UIs provide an intuitive graphical interface that allows users to quickly and easily interact with Kafka. They offer convenient access to features such as topic management, consumer groups, key metrics, cluster management, and more.
• Mobile access: Kafka web UIs are accessible from anywhere, allowing users to manage their Kafka clusters from any device, including mobile phones and tablets. This makes Kafka web UIs an incredibly convenient way to manage Kafka on the go.
• Observability: Most web UIs feature some form of Kafka metrics visualization; some even offer a complete observability solution with monitoring and alerting for metrics, logs, and traces.

Intrigued by these benefits? This opens the door to two questions. First, what web UI choices do you have? And second, what's the best way to compare these tools in order to choose the best one?

Let's start by answering the second question. In our opinion, the aspects that you should consider to determine which Kafka web UI best suits your needs are those related to usability, security, observability, schema support, and maintainability. So, this article compares tools based on the following criteria:

• User experience (UX) and user interface (UI) quality
• Authentication and authorization
• Schema support (Avro, Protobuf)
• Manageability features, such as:
  • Multicluster management
  • Kafka brokers
  • Topics
  • Users
  • ACLs
• Observability (monitoring, tracing)
• Maintenance features, such as:
  • Adding brokers
  • Partition increases
  • Rebalancing
  • Replica changes

Regarding the web UIs available for Kafka, the most relevant today are UI for Apache Kafka, Conduktor, and Redpanda Console, so they will be the ones included in this comparison. Additionally, even though it is not a web UI tool, we'll also include Kafka Tools by LinkedIn (more on the reasoning behind that shortly). With that said, let's dig into the pros and cons of each of these tools.

Kafka Tools

Kafka Tools by LinkedIn is an excellent choice for Kafka users. It offers a robust set of features that aid in managing and monitoring Kafka clusters. It's not a UI, though, so why is it included in this comparison?

The main reason is to provide expert users—those accustomed to managing Kafka from the command line—with a clear perspective of the differences between using a UI and the CLI. Not with the intention of creating a "CLI vs. UI" debate, but rather so that each engineer judges which tool is more suitable for a given task. Last but not least, nothing prevents you from using Kafka Tools in conjunction with a web UI, so it doesn't hurt to include it.

Having explained why Kafka Tools is included, let's briefly review its strengths compared to web UIs:

• Observability: To be fair, this is perhaps an area where many UIs fail. Many Kafka web UIs only offer limited metrics monitoring. That doesn't add much value, since you can manually configure an observability stack from the command line and get better results. This is a point to which we will pay special attention during the review.
• Maintenance features: This is another aspect where most web UIs leave a lot to be desired. Many web UI and even Kafka administration platforms rely on CLI tools for maintenance tasks such as adding brokers, increasing the number of partitions, rebalancing, or changing replicas. This defies the point of installing a graphical user interface. Furthermore, CLIs are especially useful for automating maintenance tasks through scripts, something that far exceeds the convenience of a UI.

Some drawbacks of Kafka Tools include:

• User experience (UX) and user interface (UI) quality: This is the biggest difference between the two approaches, since CLIs do not have a UI. That said, in terms of user experience, it's debatable which approach is better since it depends on the value the UI adds to facilitate common tasks.
• Authentication and authorization: From the command line, you can set SSL and SASL to authenticate clients and brokers. Similarly, you can use access control lists (ACLs) to restrict access to certain topics. The problem? These configurations tend to be complex, so doing them from the command line is time-consuming and challenging.
• Schema support: Kafka supports ingesting data in multiple formats, including Avro and Protobuf. However, managing the schema registry as well as deserializing Protobuf messages from the command line is inconvenient compared to doing so using a UI.
• Manageability features: No one disputes that from the command line, you can manage multiple clusters, Kafka brokers, topics, users, consumer groups, and ACLs. But how much effort does it take? Even using text expanders, it doesn't compare to the ease offered by web UIs.

Overall, Kafka Tools is still a valid option for managing Kafka clusters. This may change over time as web UIs continue to evolve. However, the truth is, depending on the number of clusters your team manages and the volume of traffic they handle, Kafka Tools could be all you need.

UI for Apache Kafka

Kafka UI—or, as its developer Provectus calls it, UI for Apache Kafka—is a free, open source web UI that stands out for being lightweight and easy to use. However, make no mistake: the fact that it is lightweight does not mean that it lacks features. Quite the opposite, in fact. UI for Apache Kafka includes many features that make it an exciting alternative for many DevOps teams.

Some of these features and strengths include:

• User experience (UX) and user interface (UI) quality: While this is a subjective topic, we believe that the minimalist interface offered by UI for Apache Kafka has the advantage of being intuitive and easy to use, especially for new users. That said, power users running multiple clusters may prefer more feature-rich dashboards like the ones offered by Conduktor or Redpanda Console.
• Schema support: The schema registry in UI for Apache Kafka supports both Apache Avro and Protobuf. Furthermore, adding both Avro and Protobuf schemas for the topic in the schema registry can be easily done from the graphical interface.
• Manageability features: In terms of manageability features, we have mixed feelings. On the one hand, Kafka UI supports multicluster management and visualization of Kafka brokers, topics, and consumer groups. However, it does not support access control lists (ACLs). More on security shortly.

Some drawbacks of UI for Apache Kafka include:

• Authentication and authorization: Security is at the top of the list for many companies today. In that sense, Kafka UI offers basic support for role-based access control (RBAC), which is a point in its favor in terms of authorization mechanisms. However, in terms of authentication, you must configure each OAuth 2.0 provider manually. At the time of writing this review, the documentation only offers examples for Cognito, Google, and GitHub, as well as a short guide on how to configure single sign-on (SSO).
• Observability: While UI for Apache Kafka allows you to view topics and partition assignments, controller status, partition count, replication status, and many other key metrics, it is far from a true observability solution. To do this, it should have tracing, monitoring, and alerting capabilities, which UI for Apache Kafka lacks.
• Maintenance features: UI for Apache Kafka allows you to view brokers, topics, and consumer groups. It also allows you to browse messages, create topics, navigate between topics and consumers, produce and send messages, and add schemas to the registry. Unfortunately, it lacks maintenance features like adding brokers, increasing partitions, rebalancing, and replica changes.

In general terms, UI for Apache Kafka is a solution that has a lot to offer, especially to novice users and teams that manage a few Kafka clusters. It's easy to install both locally and remotely and also has an intuitive graphical user interface, which is important for those who are not used to managing clusters from the UI. Another positive aspect is that it is open source, which allows you to modify it to your needs.

Conduktor

Conduktor for Apache Kafka is a comprehensive platform that enables users to easily manage, monitor, and analyze their Kafka clusters. It provides a powerful user interface to manage Kafka resources, perform analytics, and monitor tasks with built-in tools. Furthermore, Conduktor includes enterprise-grade features such as data masking, cold storage, multitenancy, audit logs, message encryption, and many other features that make it an excellent solution for large enterprises that must meet strict compliance and governance regulations.

Some of the features and strengths of Conduktor include:

• User experience (UX) and user interface (UI) quality: Both the UI and UX of Conduktor are top-notch. Through its intuitive user interface, your team can manage, test, and monitor Kafka clusters. Additionally, the UI facilitates the assurance of data quality, security, and data governance.
• Authentication and authorization: Conduktor offers several user authentication options, including local users, an LDAP server, LDAPS, Auth0, Okta, Keycloak, and Azure. It also supports single sign-on (SSO) protocols such as SAML, OpenID Connect (OIDC), Google Workspace, and Microsoft Azure AD. Furthermore, the platform enables your organization to enforce granular permissions for both users and groups through its robust RBAC feature.
• Schema support: Through its UI, Conduktor allows you to easily create and manage schemas in the schema registry using Avro, JSON, and Protobuf formats. Furthermore, schemas can be applied in different ways: to topics, to records, to both, or through a custom method.
• Manageability features: When it comes to manageability features, Conduktor checks all the boxes. It supports multicluster management, allows your team to manage and configure Kafka brokers and topics, and even knows which topic partitions are hosted on which brokers. On the security side, Conduktor also allows you to list, view, and create new ACLs.

Some drawbacks of Conduktor include:

• Observability: There is no doubt that the monitoring provided by Conduktor is superior to that of UI for Apache Kafka. Among the metrics that can be monitored are cluster health state, broker health state, partition health state, CPU usage, consumer group lag, consumer group state, and countless other application metrics. Furthermore, Conduktor supports real-time notifications and custom alerts through Slack. However, it lacks a crucial component to be considered a complete observability platform: tracing.
• Maintenance features: While Conduktor provides the ability to create and manage topics easily, once you've chosen the replication factor and partition count, there's little you can do. This lack of maintenance features is possibly its biggest weakness.

Overall, the Conduktor platform for Apache Kafka provides a feature-rich solution for managing, monitoring, testing, and analyzing Apache Kafka clusters. It has an intuitive UI, advanced monitoring capabilities, and robust security, making it an ideal platform for large companies looking to deploy and maintain Apache Kafka clusters conveniently.

Lenses for Kafka (by Lenses.io)

Lenses offers a Developer Experience for engineers to find, explore, govern, process, and integrate real-time data across a multi-Kafka environment. This gives autonomy to developers working with Kafka without dependence on centralized platform engineering teams.

Lenses acts as an “operating fabric” over an existing multi-Kafka environment spanning different Kafka vendors and deployments. It unifies the experience for developers in large enterprises to get more done, especially with its fine-grained RBAC, auditing, data masking, and data observability with SQL.

Lenses is the most suitable Kafka UI and API for teams that plan to grow and proliferate their Kafka development enterprise-wide to thousands of users.

Some of the features and strengths of Lenses include:

• A powerful Kafka UI and data catalog that intelligently discovers metadata across multiple Kafka environments.
• The Lenses Topology feature maps the data lineage of applications.
• Platform teams can enable developers by offering self-service controls for managing topics, schemas, ACLs, quotas, and alerts.
• Its superpower is its global SQL capabilities for rapid on-the-spot troubleshooting and data exploration across multiple Kafka clusters. This queries data directly in the topic (no need to move it).
• The same SQL can also define Kubernetes-based SQL stream processing applications to transform, reshape, aggregate, and join data in Kafka.
• Its IAM and RBAC model is granular and designed to give enterprises the confidence to shift-left and decentralize their data operations. It also includes data masking rules when exploring data.
• Finally, Lenses offers a suite of fully Open-source Kafka Connect connectors that are available with optional enterprise support. Their Kafka Connect connectors can be managed and deployed in a self-service manner.

Some drawbacks of Lenses.io include:

• Owing to the direction of their roadmap, Lenses is designed for companies that plan to scale their Kafka estate significantly and may have many Kafka clusters spanning different clouds & vendors. For small or completely homogenous deployments, there may be vendor-specific or more rudimentary Kafka UIs that better fit.

For pilot projects or pre-production clusters, Lenses offers a free Kafka docker box.

Redpanda Console

Formally speaking, Redpanda Console is more than just a web UI for Kafka. Redpanda is actually an API-compatible replacement for Kafka. This makes Redpanda an interesting solution for managing Apache Kafka environments. For one thing, it provides your team with a powerful and intuitive UI that you can use right away in your existing infrastructure. You could also migrate to Redpanda without application code changes and get rid of Kafka performance limitations imposed by JVM and ZooKeeper dependencies. For reference, Redpanda is up to 6x more cost-effective than Apache Kafka, 10x faster, and provides zero data loss without degrading performance.

Some of the features and strengths of Redpanda Console include:

• User experience (UX) and user interface (UI) quality: Redpanda Console provides your organization with a modern, powerful, and yet easy-to-use graphical interface. Plus, since the user interface was built with both developers and administrators in mind, Redpanda Console provides your team with a user experience that's second to none.
• Authentication and authorization: Redpanda gives your organization full control and flexibility in terms of authentication and authorization. You can configure different listeners, each with its own authentication settings. Redpanda also offers support for SASL and SCRAM authentication methods, which makes it an excellent option for large enterprises. Furthermore, with an enterprise license, your organization can also connect with popular identity providers that support OAuth and thus leverage the benefits of single sign-on (SSO) users. Regarding authorization mechanisms, Redpanda allows you to configure fine-grained access to provisioned users through ACLs, which can work in conjunction with SASL/SCRAM and with mTLS for greater security.
• Schema support: One notable advantage of Redpanda is that it includes the schema registry in its binary, which allows it to run immediately with the default configuration. As for schemas, Redpanda supports both Avro and Protobuf serialization formats, as well as Protobuf message deserialization into JSON format.
• Manageability features: When it comes to manageability features, Redpanda really shines. Through its beautiful UI, you may perform a variety of cluster administration tasks, carry out multicluster management, view brokers, view and create topics, examine consumer group data, configure listeners, set up ACLs, manage nodes, and more.
• Observability: Another differentiating aspect of Redpanda is its approach toward observability. It easily integrates with Prometheus to provide an unbeatable monitoring and alerting system. What we love about it is that it opens up the possibility of using Grafana to achieve a true observability solution that includes metrics, logs, and traces.

Some drawbacks of Redpanda Console include:

• Maintenance features: This is possibly Redpanda's Achilles' heel, at least for now. Redpanda doesn’t support setting request quotas or triggerring partition rebalancing through the UI. That being said, Redpanda automatically rebalances partition distribution when nodes are added or decommissioned in high availability (HA) deployments.

As you can see, Redpanda Console is one of the most feature-rich and user-friendly web UIs available for managing Apache Kafka clusters. It provides your team with built-in support to deploy and manage Kafka Connect connectors, it's easy to deploy, both on-premise and in the cloud, and it has incredible message filtering support. Put another way, Redpanda allows you to perform almost any type of management and monitoring task more quickly and intuitively. This makes Redpanda ideal for companies of all sizes.

Summary: comparing Web UIs to manage Kafka clusters

We’ve covered a lot of features in this comparison, so to make it easy on you, the table below summarizes everything above. This table uses a score on a scale of 1 to 5 for each category used in the comparison, with 5 being the maximum score.

As you can see, there’s no perfect solution. That said, Kafka Tools is the best option when it comes to maintenance features, while Redpanda and Conduktor excel in multiple categories.

Conclusion

In this comparison, we reviewed various web UI options for managing your Kafka clusters. While Kafka Tools isn't really a user interface, including it allowed us to talk about the advantages of graphical user interfaces over command line tools. In short, Kafka UI is ideal for those who prefer to work with a simple and lightweight dashboard, whereas Conduktor is a platform with a strong enterprise focus and few downsides. Finally, we summarized the virtues of Redpanda, a platform that goes far beyond a simple graphical interface.

To learn more about Redpanda, check out our documentation to understand the nuts and bolts of the platform, and browse the Redpanda blog for tutorials and guides on how to easily integrate with Redpanda. You can also take Redpanda's free Community edition for a test drive.

If you get stuck, have a question, or just want to chat with our solution architects, core engineers, and fellow Redpanda users, join our Redpanda Community on Slack.