Designs Valley

Get Started: How to Install & Run Apache Kafka Using Docker

A dependable streaming platform is critical in the age of big data. That’s when Kafka enters the picture. Today, you’ll learn how to set up Kafka on your machine and create your first topic. 

Kafka Installation Methodologies

Kafka may be installed on every operating system, including Windows, Mac OS X, and Linux, but the method varies. Rather than covering all of them, install Kafka on a virtual machine and utilize Linux Ubuntu as the preferred distribution.

However, because you may be using a MacBook Pro with an M1 processor, managing virtual machines is a bit of a pain. VirtualBox will not accept ISO images.

As a result, Kafka Docker will be your go-to solution. I believe it’s an even better option because you don’t have to install the tools manually. Instead, you’ll create a single Docker compose file to do everything for you.

What is Kafka?

Kafka is a kind of event-streaming platform. It gives users the ability to collect, store, and process data to create real-time event-driven applications. It’s written in Java and Scala, but you don’t need to know either of these languages to use Kafka. 

Zookeeper

Used to keep track of node status, administer a Kafka cluster, and keep track of topics and messages to manage a Kafka cluster, track node status, and maintain a list of topics and messages. Kalfa console producer is also used to write the records to Kafka topics from the command line. Kafka version 2.8.0 introduced early access to a Kafka version without Zookeeper, but it’s not ready yet for production environments.

Kafka broker

Brokers make up a Kafka Cluster. They deal with producers and consumers and ensure that data is replicated throughout the cluster.

What is Docker?

Containers can be built, deployed, and managed using this open-source platform. It lets you put your apps into containers, making application deployment much easier. That way, if the application works on your machine, you can be confident that it will work on every deployed system.

You now have a high level of working knowledge of Kafka, Zookeeper, and Docker concepts. 

Docker is used to set up Kafka.

Image credit: Unsplash 

To get Kafka working, you’ll need two Docker images:

wurstmeister/zookeeper

wurstmeister/kafka

You won’t need to manually download them because docker-compose.yml will take care of it. So you can copy the code to your machine, here it is:

version: ‘3’

services:

  zookeeper:

    image: wurstmeister/zookeeper

    container_name: zookeeper

    ports:

      “2181:2181”

  kafka:

    image: wurstmeister/kafka

    container_name: kafka

    ports:

      “9092:9092”

    environment:

      KAFKA_ADVERTISED_HOST_NAME: localhost

      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 

If 2181 or 9092 aren’t available on your PC, make sure you change the ports. You can now navigate to the location where you saved the docker-compose.yml file in a Terminal. To retrieve the images and construct containers, run the following command:

docker-compose -f  docker-compose.yml -d up

The -d option causes Zookeeper and Kafka to run in the background, allowing you to access the Terminal once they’re up and running.

The download and configuration process should now be displayed on the Terminal.

That’s the end of the story! To verify that both are running, use the docker ps command.

For single-node setup

Image credit: Unsplash

Let’s start by learning how to set up a single-node Kafka broker, which will suffice for most local development needs. 

1. Configuration of docker-compose.yml

We’ll need to start a Zookeeper server first before we can start an Apache Kafka server.

This dependency can be set in a docker-compose.yml file, ensuring that the Zookeeper server begins before the Kafka server and shuts after it. 

version: ‘2’

services:

  zookeeper:

    image: confluentinc/cp-zookeeper:latest

    environment:

      ZOOKEEPER_CLIENT_PORT: 2181

      ZOOKEEPER_TICK_TIME: 2000

    ports:

      – 22181:2181

  kafka:

    image: confluentinc/cp-kafka:latest

    depends_on:

      – zookeeper

    ports:

      – 29092:29092

    environment:

      KAFKA_BROKER_ID: 1

      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

2. Start Kafka server

Using the docker-compose command, let’s start the Kafka server:

$ docker-compose up -d

Creating network “kafka_default” with the default driver

Creating kafka_zookeeper_1 … done

Creating kafka_kafka_1     … done

$ nc -z localhost 22181

Connection to localhost port 22181 [tcp/*] succeeded!

$ nc -z localhost 29092

Connection to localhost port 29092 [tcp/*] succeeded!

Multi-node setup

Let’s add a multi-node Kafka cluster to our docker-compose.yml configuration. 

1. docker-compose.yml Configuration

Redundancy for both Zookeeper and Kafka servers is required in an Apache Kafka cluster arrangement.

So, for Zookeeper and Kafka services, let’s add settings for one more node each:

version: ‘2’

services:

  zookeeper-1:

    image: confluentinc/cp-zookeeper:latest

    environment:

      ZOOKEEPER_CLIENT_PORT: 2181

      ZOOKEEPER_TICK_TIME: 2000

    ports:

      – 22181:2181

  zookeeper-2:

    image: confluentinc/cp-zookeeper:latest

    environment:

      ZOOKEEPER_CLIENT_PORT: 2181

      ZOOKEEPER_TICK_TIME: 2000

    ports:

      – 32181:2181

  kafka-1:

    image: confluentinc/cp-kafka:latest

    depends_on:

      – zookeeper-1

      – zookeeper-2

    ports:

      – 29092:29092

    environment:

      KAFKA_BROKER_ID: 1

      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092,PLAINTEXT_HOST://localhost:29092

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  kafka-2:

    image: confluentinc/cp-kafka:latest

    depends_on:

      – zookeeper-1

      – zookeeper-2

    ports:

      – 39092:39092

    environment:

      KAFKA_BROKER_ID: 2

      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9092,PLAINTEXT_HOST://localhost:39092

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

2. Kafka cluster setup

Let’s use the docker-compose command to create the cluster:

Let’s connect to the cluster using the Kafka Tool by specifying comma-separated values for the Kafka servers and their corresponding ports after the cluster is up and running.

$ docker-compose up -d

Creating network “kafka_default” with the default driver

Creating kafka_zookeeper-1_1 … done

Creating kafka_zookeeper-2_1 … done

Creating kafka_kafka-2_1     … done

Creating kafka_kafka-1_1     … done

Conclusion

In this article, we created single-node and multi-node Apache Kafka configurations using Docker technology. We also used the Kafka Tool to connect to the broker server and visualize the information.

About The Author

Scroll to Top