Image credit: Unsplash

A dependable streaming platform is critical in the age of big data. That’s when Kafka enters the picture. Today, you’ll learn how to set up Kafka on your machine and create your first topic. 

Kafka Installation Methodologies

Kafka may be installed on every operating system, including Windows, Mac OS X, and Linux, but the method varies. Rather than covering all of them, install Kafka on a virtual machine and utilize Linux Ubuntu as the preferred distribution.

However, because you may be using a MacBook Pro with the M1 processor, managing virtual machines is a bit of a pain. VirtualBox will not accept ISO images.

As a result, Kafka Docker will be your go-to solution. Because you don’t have to manually install the tools, I believe it’s an even better option. Instead, you’ll create a single Docker compose file that will do everything for you

What is Kafka?

Kafka is a kind of event-streaming platform. It gives users the ability to collect, store, and process data in order to create real-time event-driven applications. It’s written in Java and Scala, but you don’t need to know either of these languages to use Kafka. 

Zookeeper

Used to keep track of node status, administer a Kafka cluster, and keep track of topics and messages to manage a Kafka cluster, track node status, and maintain a list of topics and messages. Kalfa console producer also is used to write the records to Kafka topics all the way from the command line. Kafka version 2.8.0 introduced early access to a Kafka version without Zookeeper, but it’s not ready yet for production environments.

Kafka broker

Brokers are what make up a Kafka Cluster. They deal with producers and consumers, as well as ensure that data is replicated throughout the cluster.

What is Docker?

Containers can be built, deployed, and managed using this open-source platform. It allows you to put your apps into containers, making application deployment much easier. That way, if the application works on your machine, you can be confident that it will work on every system to which it is deployed.

You now have a working knowledge of Kafka, Zookeeper, and Docker concepts at a high level. 

Docker is used to setting up Kafka.

Image credit: Unsplash 

To get Kafka working, you’ll need two Docker images:

wurstmeister/zookeeper

wurstmeister/kafka

You won’t need to manually download them because docker-compose.yml will take care of it. So you can copy the code to your machine, here it is:

version: ‘3’

services:

  zookeeper:

    image: wurstmeister/zookeeper

    container_name: zookeeper

    ports:

      “2181:2181”

  kafka:

    image: wurstmeister/kafka

    container_name: kafka

    ports:

      “9092:9092”

    environment:

      KAFKA_ADVERTISED_HOST_NAME: localhost

      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 

If 2181 or 9092 aren’t available on your PC, make sure you change the ports. You can now navigate to the location where you saved the docker-compose.yml file in a Terminal. To retrieve the images and construct containers, run the following command:

docker-compose -f  docker-compose.yml -d up

The -d option causes Zookeeper and Kafka to run in the background, allowing you to access the Terminal once they’re up and running.

The download and configuration process should now be displayed on the Terminal.

That’s the end of the story! To verify that both are running, use the docker ps command.

For single-node setup

Image credit: Unsplash

Let’s start by learning how to set up a single node Kafka broker, which will suffice for most local development needs. 

1. Configuration of docker-compose.yml

We’ll need to start a Zookeeper server first before we can start an Apache Kafka server.

This dependency can be set in a docker-compose.yml file, ensuring that the Zookeeper server begins before the Kafka server and shuts after it. 

version: ‘2’

services:

  zookeeper:

    image: confluentinc/cp-zookeeper:latest

    environment:

      ZOOKEEPER_CLIENT_PORT: 2181

      ZOOKEEPER_TICK_TIME: 2000

    ports:

      – 22181:2181

  kafka:

    image: confluentinc/cp-kafka:latest

    depends_on:

      – zookeeper

    ports:

      – 29092:29092

    environment:

      KAFKA_BROKER_ID: 1

      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

2. Start Kafka server

Using the docker-compose command, let’s start the Kafka server:

$ docker-compose up -d

Creating network “kafka_default” with the default driver

Creating kafka_zookeeper_1 … done

Creating kafka_kafka_1     … done

$ nc -z localhost 22181

Connection to localhost port 22181 [tcp/*] succeeded!

$ nc -z localhost 29092

Connection to localhost port 29092 [tcp/*] succeeded!

Multi-node setup

Let’s add a multi-node Kafka cluster to our docker-compose.yml configuration. 

1. docker-compose.yml Configuration

Redundancy for both Zookeeper and Kafka servers is required in an Apache Kafka cluster arrangement.

So, for Zookeeper and Kafka services, let’s add settings for one more node each:

version: ‘2’

services:

  zookeeper-1:

    image: confluentinc/cp-zookeeper:latest

    environment:

      ZOOKEEPER_CLIENT_PORT: 2181

      ZOOKEEPER_TICK_TIME: 2000

    ports:

      – 22181:2181

  zookeeper-2:

    image: confluentinc/cp-zookeeper:latest

    environment:

      ZOOKEEPER_CLIENT_PORT: 2181

      ZOOKEEPER_TICK_TIME: 2000

    ports:

      – 32181:2181

  kafka-1:

    image: confluentinc/cp-kafka:latest

    depends_on:

      – zookeeper-1

      – zookeeper-2

    ports:

      – 29092:29092

    environment:

      KAFKA_BROKER_ID: 1

      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092,PLAINTEXT_HOST://localhost:29092

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  kafka-2:

    image: confluentinc/cp-kafka:latest

    depends_on:

      – zookeeper-1

      – zookeeper-2

    ports:

      – 39092:39092

    environment:

      KAFKA_BROKER_ID: 2

      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181

      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9092,PLAINTEXT_HOST://localhost:39092

      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

2. Kafka cluster setup

Let’s use the docker-compose command to create the cluster:

Let’s connect to the cluster using the Kafka Tool by specifying comma-separated values for the Kafka servers and their corresponding ports after the cluster is up and running.

$ docker-compose up -d

Creating network “kafka_default” with the default driver

Creating kafka_zookeeper-1_1 … done

Creating kafka_zookeeper-2_1 … done

Creating kafka_kafka-2_1     … done

Creating kafka_kafka-1_1     … done

Conclusion

We created single node and multi-node Apache Kafka configurations using Docker technology in this article. We also used the Kafka Tool to connect to the broker server and visualize the information.

About The Author

Shahzad Ahmad Mirza

Shahzad Ahmad Mirza is a web developer, entrepreneur, and trainer based in Lahore, Pakistan. He started his career in 2000 and founded his web development agency, Designs Valley, in 2012. Mirza also runs a YouTube channel, “Learn With Shahzad Ahmad Mirza,” where he shares his web programming and internet marketing expertise. He has trained over 50,000 students, many of whom have become successful digital marketers, programmers, and freelancers. He also created the GBOB (Guest Blog Posting Business) course, which teaches individuals how to make money online.