Create a Kafka multi-broker cluster

This section describes the creation of a multi-broker Kafka cluster with brokers located on different hosts. In this scenario:

• One server hosts the Zookeeper server and a Kafka broker
• The second server hosts a a second Kafka broker
• The third server hosts a producer and a consumer

NOTE: Before beginning, ensure that ports 2181 (Zookeeper) and 9092 (Kafka) are open on the first server and port 9092 (Kafka) is open on the second server. Also ensure that remote connections are possible between the three servers (instructions).

Configuring the first server (Zookeeper manager and Kafka broker)

The default configuration may be used as is. However, you must perform the steps below:

• Restart the Kafka and Zookeeper services.

    $ sudo service bitnami restart kafka
    $ sudo service bitnami restart zookeeper
  

Configuring the second server (Kafka broker)

• Edit the /opt/bitnami/kafka/config/server.properties configuration file and update the broker.id parameter.

    broker.id = 1
  

  This broker id must be unique in the Kafka ecosystem.

• In the same file, update the zookeeper.connect parameter to reflect the public IP address of the first server.

    zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
  

• Stop the Zookeeper service.

    $ sudo service bitnami stop zookeeper
  

• Restart the Kafka service.

    $ sudo service bitnami restart kafka
  

Configuring the third server (Kafka message producer/consumer)

• Edit the /opt/bitnami/kafka/config/producer.properties file and update the metadata.broker.list parameter with the public IP addresses of the two brokers:

    metadata.broker.list=PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092, PUBLIC_IP_ADDRESS_OF_SECOND_KAFKA_BROKER:9092
  

• Edit the /opt/bitnami/kafka/config/consumer.properties file and update the zookeeper.connect parameter to reflect the public IP address of the first server.

    zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
  

• Since this host only serves as a producer and a consumer, stop the Kafka and Zookeeper services:

    $ sudo service bitnami stop kafka
    $ sudo service bitnami stop zookeeper
  

Testing the cluster

NOTE: The following commands should be executed on the third server (Kafka message producer/consumer).

• Export the authentication configuration:

    $ export KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/config/kafka_jaas.conf"
  

• Create a new topic.

    $ /opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --replication-factor 2 --partitions 1 --topic multiBroker
  

• Produce some messages by running the command below and then entering some messages, each on a separate line. Enter Ctrl-D to end.

    $ /opt/bitnami/kafka/bin/kafka-console-producer.sh --bootstrap-server PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --producer.config /opt/bitnami/kafka/config/producer.properties --topic multiBroker
    this is a message
    this is another message
  

• Consume the messages. The consumer will connect to the cluster and retrieve and display the messages you entered in the previous step.

    $ /opt/bitnami/kafka/bin/kafka-console-consumer.sh --bootstrap-server PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --consumer.config /opt/bitnami/kafka/config/consumer.properties --topic multiBroker --from-beginning
    this is a message
    this is another message