Create a Kafka multi-broker cluster
This section describes the creation of a multi-broker Kafka cluster with brokers located on different hosts. In this scenario:
- One server hosts the Zookeeper server and a Kafka broker
- The second server hosts a a second Kafka broker
- The third server hosts a producer and a consumer
NOTE: Before beginning, ensure that ports 2181 (Zookeeper) and 9092 (Kafka) are open on the first server and port 9092 (Kafka) is open on the second server. Also ensure that remote connections are possible between the three servers (instructions).
Configuring the first server (Zookeeper manager and Kafka broker)
The default configuration may be used as is. However, you must perform the steps below:
Delete the contents of the Zookeeper and Kafka temporary directories
$ sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs $ sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper
Restart the Kafka and Zookeeper services.
$ sudo /opt/bitnami/ctlscript.sh restart kafka $ sudo /opt/bitnami/ctlscript.sh restart zookeeper
Configuring the second server (Kafka broker)
Edit the /opt/bitnami/kafka/config/server.properties configuration file and update the broker.id parameter.
broker.id = 1
This broker id must be unique in the Kafka ecosystem.
In the same file, update the zookeeper.connect parameter to reflect the public IP address of the first server.
zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
Delete the contents of the Zookeeper and Kafka temporary directories
$ sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs $ sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper
Stop the Zookeeper service.
$ sudo /opt/bitnami/ctlscript.sh stop zookeeper
Restart the Kafka service.
$ sudo /opt/bitnami/ctlscript.sh restart kafka
Configuring the third server (Kafka message producer/consumer)
Edit the /opt/bitnami/kafka/config/producer.properties file and update the metadata.broker.list parameter with the public IP addresses of the two brokers:
metadata.broker.list=PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092, PUBLIC_IP_ADDRESS_OF_SECOND_KAFKA_BROKER:9092
Edit the /opt/bitnami/kafka/config/consumer.properties file and update the zookeeper.connect parameter to reflect the public IP address of the first server.
zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
Since this host only serves as a producer and a consumer, stop the Kafka and Zookeeper services:
$ sudo /opt/bitnami/ctlscript.sh stop kafka $ sudo /opt/bitnami/ctlscript.sh stop zookeeper
Testing the cluster
NOTE: The following commands should be executed on the third server (Kafka message producer/consumer).
Export the authentication configuration:
$ export KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/conf/kafka_jaas.conf"
Create a new topic.
$ /opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --replication-factor 2 --partitions 1 --topic multiBroker
Produce some messages by running the command below and then entering some messages, each on a separate line. Enter Ctrl-D to end.
$ /opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --producer.config /opt/bitnami/kafka/conf/producer.properties --topic multiBroker this is a message this is another message
Consume the messages. The consumer will connect to the cluster and retrieve and display the messages you entered in the previous step.
$ /opt/bitnami/kafka/bin/kafka-console-consumer.sh --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --topic multiBroker --from-beginning this is a message this is another message