Create a Kafka multi-broker cluster

This section describes the creation of a multi-broker Kafka cluster with brokers located on different hosts. In this scenario:

• One server hosts the Zookeeper server and a Kafka broker
• The second server hosts a a second Kafka broker
• The third server hosts a producer and a consumer

NOTE: Before beginning, ensure that ports 2181 (Zookeeper) and 9092 (Kafka) are open on the first server and port 9092 (Kafka) is open on the second server. Also ensure that remote connections are possible between the three servers (instructions).

Configuring the first server (Zookeeper manager and Kafka broker)

The default configuration may be used as is. However, you must perform the steps below:

• Delete the contents of the Zookeeper and Kafka temporary directories

  $ sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs
  $ sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper
  

• Restart the Kafka and Zookeeper services.

  $ sudo /opt/bitnami/ctlscript.sh restart kafka
  $ sudo /opt/bitnami/ctlscript.sh restart zookeeper
  

Configuring the second server (Kafka broker)

• Edit the /opt/bitnami/kafka/config/server.properties configuration file and update the broker.id parameter.

  broker.id = 1
  

  This broker id must be unique in the Kafka ecosystem.

• In the same file, update the zookeeper.connect parameter to reflect the public IP address of the first server.

  zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
  

• Delete the contents of the Zookeeper and Kafka temporary directories

  $ sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs
  $ sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper
  

• Stop the Zookeeper service.

  $ sudo /opt/bitnami/ctlscript.sh stop zookeeper
  

• Restart the Kafka service.

  $ sudo /opt/bitnami/ctlscript.sh restart kafka
  

Configuring the third server (Kafka message producer/consumer)

• Edit the /opt/bitnami/kafka/config/producer.properties file and update the metadata.broker.list parameter with the public IP addresses of the two brokers:

  metadata.broker.list=PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092, PUBLIC_IP_ADDRESS_OF_SECOND_KAFKA_BROKER:9092
  

• Edit the /opt/bitnami/kafka/config/consumer.properties file and update the zookeeper.connect parameter to reflect the public IP address of the first server.

  zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
  

• Since this host only serves as a producer and a consumer, stop the Kafka and Zookeeper services:

  $ sudo /opt/bitnami/ctlscript.sh stop kafka
  $ sudo /opt/bitnami/ctlscript.sh stop zookeeper
  

Testing the cluster

NOTE: The following commands should be executed on the third server (Kafka message producer/consumer).

• Export the authentication configuration:

  $ export KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/conf/kafka_jaas.conf"
  

• Create a new topic.

  $ /opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --replication-factor 2 --partitions 1 --topic multiBroker
  

• Produce some messages by running the command below and then entering some messages, each on a separate line. Enter Ctrl-D to end.

  $ /opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --producer.config /opt/bitnami/kafka/conf/producer.properties --topic multiBroker
  this is a message
  this is another message
  

• Consume the messages. The consumer will connect to the cluster and retrieve and display the messages you entered in the previous step.

  $ /opt/bitnami/kafka/bin/kafka-console-consumer.sh --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --topic multiBroker --from-beginning
  this is a message
  this is another message