Bitnami Kafka Installer

NOTE: Before running the commands shown on this page, you should load the Bitnami stack environment by executing the installdir/use_APPNAME script (Linux and Mac OS X) or by clicking the shortcut in the Start Menu under "Start -> Bitnami APPNAME Stack -> Application console" (Windows). Learn more.
NOTE: When running the commands shown on this page, replace the installdir placeholder with the full installation directory for your Bitnami stack.

Description

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

First steps with the Bitnami Kafka Stack

Welcome to your new Bitnami application! Here are a few questions (and answers!) you might need when first starting with your application.

What are the system requirements?

Before you download and install your application, check that your system meets these requirements.

How do I install the Bitnami Kafka Stack?

Windows, OS X and Linux installer
  • Download the executable file for the Bitnami Kafka Stack from the Bitnami website.

  • Run the downloaded file:

    • On Linux, give the installer executable permissions and run the installation file in the console.
    • On other platforms, double-click the installer and follow the instructions shown.

Check the FAQ instructions on how to download and install a Bitnami Stack for more details.

The application will be installed to the following default directories:

Operating System Directory
Windows C:\Bitnami\APPNAME-VERSION
Mac OS X /Applications/APPNAME-VERSION
Linux /opt/APPNAME-VERSION (running as root user)
OS X VM
  • Download the OS X VM file for the Bitnami Kafka Stack from the Bitnami website.
  • Begin the installation process by double-clicking the image file and dragging the WordPress OS X VM icon to the Applications folder.
  • Launch the VM by double-clicking the icon in the Applications folder.

What credentials do I need?

You need application credentials, consisting of a username and password. These credentials allow you to log in to your new Bitnami application.

What is the administrator username set for me to log in to the application for the first time?

  • For Windows, Linux and OS X installers, the username was configured by you when you first installed the application.
  • For OS X VMs, the username can be obtained by clicking the Bitnami badge at the bottom right corner of the application welcome page.

What is the administrator password?

  • For Windows, Linux and OS X installers, the password was configured by you when you first installed the application.
  • For OS X VMs, the password can be obtained by clicking the Bitnami badge at the bottom right corner of the application welcome page.

How to start or stop the services?

Linux

Bitnami native installers include a graphical tool to manage services. This tool is named manager-linux-x64.run on Linux and is located in the installation directory. To use this tool, double-click the file and then use the graphical interface to start, stop or restart services. Server log messages can be checked in the "Server Events" tab.

Management tool

The native installer also includes a command-line script to start, stop and restart applications, named ctlscript.sh. This script can be found in the installation directory and accepts the options start, stop, restart, and status. To use it, log in to the server console and execute it following the examples below:

  • Call it without any service names to start all services:

      $ sudo installdir/ctlscript.sh start
    
  • Use it to restart a specific service only by passing the service name as argument - for example, mysql, postgresql or apache:

      $ sudo installdir/ctlscript.sh restart mysql
      $ sudo installdir/ctlscript.sh restart postgresql
      $ sudo installdir/ctlscript.sh restart apache
    
  • Obtain current status of all services:

      $ installdir/ctlscript.sh status
    

The list of available services varies depending on the required components for each application.

Mac OS X

Bitnami native installers include a graphical tool to manage services. This tool is named manager-osx on Mac OS X and is located in the installation directory. To use this tool, double-click the file and then use the graphical interface to start, stop or restart services. Server log messages can be checked in the "Server Events" tab.

Management tool

The native installer also includes a command-line script to start, stop and restart applications, named ctlscript.sh. This script can be found in the installation directory and accepts the options start, stop, restart, and status. To use it, log in to the server console and execute it following the examples below:

  • Call it without any service names to start all services:

    $ sudo installdir/ctlscript.sh start
    
  • Use it to restart a specific service only by passing the service name as argument - for example, mysql or apache:

     $ sudo installdir/ctlscript.sh restart mysql
     $ sudo installdir/ctlscript.sh restart apache
    
  • Obtain current status of all services:

     $ installdir/ctlscript.sh status
    

The list of available services varies depending on the required components for each application.

NOTE: If you are using the stack manager for Mac OS X-VM, please check the following blog post to learn how to manage services from its graphical tool.

Windows

Bitnami native installers include a graphical tool to manage services. This tool is named manager-windows.exe on Windows and is located in the installation directory. To use this tool, double-click the file and then use the graphical interface to start, stop or restart services. Server log messages can be checked in the "Server Events" tab.

Management tool

The Windows native installer creates shortcuts to start and stop services created in the Start Menu, under "Programs -> Bitnami APPNAME Stack -> Bitnami Service". Servers can also be managed from the Windows "Services" control panel. Services are named using the format APPNAMESERVICENAME, where APPNAME is a placeholder for the application name and SERVICENAME is a placeholder for the service name. For example, the native installer for the Bitnami WordPress Stack installs services named wordpressApache and wordpressMySQL.

These services will be automatically started during boot. To modify this behaviour, refer to the section on disabling services on Windows.

What is the default configuration?

Kafka default configuration

Kafka configuration files

The Kafka configuration files are located at the installdir/kafka/config/ directory.

Kafka ports

The Kafka server has a single broker running on port 9092. Only conections from localhost are permitted.

Kafka log files

The Kafka log files are created at the installdir/kafka/logs/ directory.

Zookeeper default configuration

Zookeeper configuration files

The Zookeeper configuration files are located at the installdir/zookeeper/conf/ directory.

Zookeeper ports

By default, the Zookeeper server runs on port 2181. Only conections from localhost are permitted.

How to connect to Kafka from a different machine?

IMPORTANT: Bitnami Native Installers do not modify the firewall configuration of your computer, therefore the Kafka ports could be open which is a significant security risk. You are strongly advised to close the Kafka ports (refer to the FAQ for more information on this).

Once you have an active SSH tunnel or if you did not close the port for remote access, perform these additional steps:

  • Edit your Zookeeper configuration file (installdir/zookeeper/conf/zoo.cfg) and comment out the following line:

     clientPortAddress=localhost
    
  • Edit your Kafka configuration file (installdir/kafka/config/server.properties). If necessary, uncomment the following line and change the value of the parameter to the public IP address of the server:

     #advertised.host.name=<hostname routable by clients>
    
  • Restart the server to reload the configuration files.

     $ sudo installdir/ctlscript.sh restart
    

How to create a Kafka multi-broker cluster?

This section describes the creation of a multi-broker Kafka cluster with brokers located on different hosts. In this scenario:

  • One server hosts the Zookeeper server and a Kafka broker
  • The second server hosts a a second Kafka broker
  • The third server hosts a producer and a consumer

Kafka cluster

NOTE: Before beginning, ensure that ports 2181 (Zookeeper) and 9092 (Kafka) are open on the first server and port 9092 (Kafka) is open on the second server. Also ensure that remote connections are possible between the three servers (instructions).

Configuring the first server (Zookeeper manager and Kafka broker)

The default configuration may be used as is. However, you must perform the steps below:

  • Delete the contents of the Zookeeper and Kafka temporary directories

     $ sudo rm -rf installdir/kafka/tmp/kafka-logs
     $ sudo rm -rf installdir/zookeeper/tmp/zookeeper
    
  • Restart the Kafka and Zookeeper services.

     $ sudo installdir/ctlscript.sh restart kafka
     $ sudo installdir/ctlscript.sh restart zookeeper
    

Configuring the second server (Kafka broker)

  • Edit the installdir/kafka/config/server.properties configuration file and update the broker.id parameter.

     broker.id = 1
    

    This broker id must be unique in the Kafka ecosystem.

  • In the same file, update the zookeeper.connect parameter to reflect the public IP address of the first server.

     zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
    
  • Delete the contents of the Zookeeper and Kafka temporary directories

     $ sudo rm -rf installdir/kafka/tmp/kafka-logs
     $ sudo rm -rf installdir/zookeeper/tmp/zookeeper
    
  • Stop the Zookeeper service.

     $ sudo installdir/ctlscript.sh stop zookeeper
    
  • Restart the Kafka service.

     $ sudo installdir/ctlscript.sh restart kafka
    

Configuring the third server (Kafka message producer/consumer)

  • Edit the installdir/kafka/config/producer.properties file and update the metadata.broker.list parameter with the public IP addresses of the two brokers:

     metadata.broker.list=PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092, PUBLIC_IP_ADDRESS_OF_SECOND_KAFKA_BROKER:9092
    
  • Edit the installdir/kafka/config/consumer.properties file and update the zookeeper.connect parameter to reflect the public IP address of the first server.

     zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
    
  • Since this host only serves as a producer and a consumer, stop the Kafka and Zookeeper services:

     $ sudo installdir/ctlscript.sh stop kafka
     $ sudo installdir/ctlscript.sh stop zookeeper
    

Testing the cluster

NOTE: The following commands should be executed on the third server (Kafka message producer/consumer).
  • Create a new topic.

     $ installdir/kafka/bin/kafka-topics.sh --create --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --replication-factor 2 --partitions 1 --topic multiBroker
    
  • Produce some messages by running the command below and then entering some messages, each on a separate line. Enter Ctrl-C to end.

     $ installdir/kafka/bin/kafka-console-producer.sh --broker-list PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --topic multiBroker
     this is a message
     this is another message
     ^C
    
  • Consume the messages. The consumer will connect to the cluster and retrieve and display the messages you entered in the previous step.

     $ installdir/kafka/bin/kafka-console-consumer.sh --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --topic multiBroker --from-beginning
     this is a message
     this is another message
     ^C
    

How can I run a command in the Bitnami Kafka Stack?

Load the Bitnami Console and then run the command as usual.

How to run a Kafka producer and consumer from the server itself?

You can run the following example to publish and collect your first message:

  • Declare a new topic. The Kafka server is configured to use the server's public IP address:

     $ installdir/kafka/bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test
    

    We use --replication-factor to indicate how many servers are going to have a copy of the logs, and --partitions to choose the number of partitions for the topic we are creating.

  • Start a new producer on the same Kafka server and generates a message in the topic. Remember to replace localhost with your server's public IP address. Enter CTRL-D to end the message.

     $ installdir/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
    
     this is my first message 
    
  • Collect and display the first message in the consumer:

     $ installdir/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
    

How to create a full backup of Kafka?

The Bitnami Kafka Stack is self-contained and the simplest option for performing a backup is to copy or compress the Bitnami stack installation directory. To do so in a safe manner, you will need to stop all servers, so this method may not be appropriate if you have people accessing the application continuously.

NOTE: If you want to create only a database backup, refer to these instructions for MySQL and PostgreSQL.

Backup on Linux and Mac OS X

Follow these steps:

  • Change to the directory in which you wish to save your backup.

     $ cd /your/directory
    
  • Stop all servers.

     $ sudo installdir/ctlscript.sh stop
    
  • Create a compressed file with the stack contents.

     $ sudo tar -pczvf application-backup.tar.gz installdir
    
  • Start all servers.

     $ sudo installdir/ctlscript.sh start
    
  • Download or transfer the application-backup.tar.gz file to a safe location.

You should now download or transfer the application-backup.tar.gz file to a safe location.

Backup on Windows

Follow these steps:

  • Stop all servers using the shortcuts in the Start Menu or the graphical manager tool.

  • Create a compressed file with the stack contents. You can use a graphical tool like 7-Zip or WinZip or just right-click the folder, click "Send to", and select the "Compressed (zipped) folder" option.

  • Download or transfer the compressed file to a safe location.

  • Start all servers using the shortcuts in the Start Menu or the graphical manager tool.

Restore on Linux and Mac OS X

Follow these steps:

  • Change to the directory containing your backup:

    $ cd /your/directory
    
  • Stop all servers.

    $ sudo installdir/ctlscript.sh stop
    
  • Move the current stack to a different location:

    $ sudo mv installdir /tmp/bitnami-backup
    
  • Uncompress the backup file to the original directory:

    $ sudo tar -pxzvf application-backup.tar.gz -C /
    
  • Start all servers.

    $ sudo installdir/ctlscript.sh start
    
IMPORTANT: When restoring, remember to maintain the original permissions for the files and folders. For example, if you originally installed the stack as the root user on Linux, make sure that the restored files are owned by root as well.

Restore on Windows

  • Change to the directory containing your backup:

       $ cd /your/directory
    
  • Stop all servers using the shortcuts in the Start Menu or the graphical manager tool.

  • Uninstall the previous services by executing the following command:

        $ serviceinstall.bat
    
  • Create a safe folder named Backups in the desktop and move the current stack to it. Remember to replace PATH with the right location of your folder:

        $ move installdir \PATH\Backups
    
  • Uncompress the backup file using a tool like 7-Zip or Winzip or just double-click the .zip file to uncompress it, and move it to the original directory.
  • Install services by running the following commands from an elevated command prompt:

      $ cd installdir
      $ serviceinstall.bat INSTALL
    
  • Start all servers using the shortcuts in the Start Menu or the graphical manager tool.

How to debug Kafka and Zookeeper errors?

The main Kafka log file is created at installdir/kafka/logs/server.log.

The main Zookeeper log file is created at installdir/zookeeper/tmp/zookeeper.out.

nativeInstaller

Bitnami Documentation