Deploy your Bitnami Kafka Stack on AWS Cloud now! Launch Now

Bitnami Kafka for AWS Cloud

Description

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

First steps with the Bitnami Kafka Stack

Welcome to your new Bitnami application running on Amazon Web Services! Here are a few questions (and answers!) you might need when first starting with your application.

What credentials do I need?

You need two sets of credentials:

  • The application credentials that allow you to log in to your new Bitnami application. These credentials consist of a username and password.
  • The server credentials that allow you to log in to your AWS Cloud server using an SSH client and execute commands on the server using the command line. These credentials consist of an SSH username and key.

Watch the following video to learn quickly how to obtain the application credentials of those applications deployed using the AWS Console:

What is the administrator username set for me to log in to the application for the first time?

Username: user

How do I get my SSH key or password?

SSH username: bitnami

How to start or stop the services?

Each Bitnami stack includes a control script that lets you easily stop, start and restart services. The script is located at /opt/bitnami/ctlscript.sh. Call it without any service name arguments to start all services:

$ sudo /opt/bitnami/ctlscript.sh start

Or use it to restart a single service, such as Apache only, by passing the service name as argument:

$ sudo /opt/bitnami/ctlscript.sh restart apache

Use this script to stop all services:

$ sudo /opt/bitnami/ctlscript.sh stop

Restart the services by running the script without any arguments:

$ sudo /opt/bitnami/ctlscript.sh restart

Obtain a list of available services and operations by running the script without any arguments:

$ sudo /opt/bitnami/ctlscript.sh

What is the default configuration?

Kafka default configuration

Kafka configuration files

The Kafka configuration files are located at the /opt/bitnami/kafka/config/ directory.

Kafka ports

The Kafka server has a single broker running on port 9092. Only conections from localhost are permitted.

Kafka log files

The Kafka log files are created at the /opt/bitnami/kafka/logs/ directory.

Zookeeper default configuration

Zookeeper configuration files

The Zookeeper configuration files are located at the /opt/bitnami/zookeeper/conf/ directory.

Zookeeper ports

By default, the Zookeeper server runs on port 2181. Only conections from localhost are permitted.

What are the default ports?

A port is an endpoint of communication in an operating system that identifies a specific process or a type of service. Bitnami stacks include several services or servers that require a port.

IMPORTANT: Making this application's network ports public is a significant security risk. You are strongly advised to only allow access to those ports from trusted networks. If, for development purposes, you need to access from outside of a trusted network, please do not allow access to those ports via a public IP address. Instead, use a secure channel such as a VPN or an SSH tunnel. Follow these instructions to remotely connect safely and reliably.

Port 22 is the default port for SSH connections.

The Kafka access port is 9092. This port is closed by default. You must open it to enable remote access.

How to upload files to the server with SFTP?

NOTE: Bitnami applications can be found in /opt/bitnami/apps.

The first step is to ensure that you have an SSH key for your server.

If you are using the Bitnami Launchpad for AWS Cloud, download the SSH key for your server in .ppk format (for FileZilla or WinSCP) or in .pem format (for Cyberduck) from the Launchpad detail page for your server.

SSH keys

Although you can use any SFTP/SCP client to transfer files to your server, this guide documents FileZilla (Windows, Linux and Mac OS X), WinSCP (Windows) and Cyberduck (Mac OS X).

Using an SSH Key

Once you have your server's SSH key, choose your preferred application and follow the steps below to connect to the server using SFTP.

FileZilla
IMPORTANT: To use FileZilla, your server private key should be in PPK format.

Follow these steps:

  • Download and install FileZilla.
  • Launch FileZilla and use the "Edit -> Settings" command to bring up FileZilla's configuration settings.
  • Within the "Connection -> SFTP" section, use the "Add keyfile" command to select the private key file for the server. FileZilla will use this private key to log in to the server.

    FileZilla configuration

  • Use the "File -> Site Manager -> New Site" command to bring up the FileZilla Site Manager, where you can set up a connection to your server.
  • Enter your server host name and specify bitnami as the user name.
  • Select "SFTP" as the protocol and "Ask for password" as the logon type.

    FileZilla configuration

  • Use the "Connect" button to connect to the server and begin an SFTP session. You might need to accept the server key, by clicking "Yes" or "OK" to proceed.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

If you have problems accessing your server, get extra information by use the "Edit -> Settings -> Debug" menu to activate FileZilla's debug log.

FileZilla debug log

WinSCP
IMPORTANT: To use WinSCP, your server private key should be in PPK format.

Follow these steps:

  • Download and install WinSCP.
  • Launch WinSCP and in the "Session" panel, select "SCP" as the file protocol.
  • Enter your server host name and specify bitnami as the user name.

    WinSCP configuration

  • Click the "Advanced…" button and within the "SSH -> Authentication -> Authentication parameters" section, select the private key file for the server. WinSCP will use this private key to log in to the server.

    WinSCP configuration

  • From the "Session" panel, use the "Login" button to connect to the server and begin an SCP session.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

If you need to upload files to a location where the bitnami user doesn't have write permissions, you have two options:

  • Once you have configured WinSCP as described above, click the "Advanced…" button and within the "Environment -> Shell" panel, select sudo su - as your shell. This will allow you to upload files using the administrator account.

    WinSCP configuration

  • Upload the files to the /home/bitnami directory as usual. Then, connect via SSH and move the files to the desired location with the sudo command, as shown below:

     $ sudo mv /home/bitnami/uploaded-file /path/to/desired/location/
    
Cyberduck
IMPORTANT: To use Cyberduck, your server private key should be in PEM format.

Follow these steps:

  • Select the "Open Connection" command and specify "SFTP" as the connection protocol.

    Cyberduck configuration

  • In the connection details panel, under the "More Options" section, enable the "Use Public Key Authentication" option and specify the path to the private key file for the server.

    Cyberduck configuration

  • Use the "Connect" button to connect to the server and begin an SFTP session.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

How to connect to Kafka from a different machine?

For security reasons, the Kafka ports in this solution cannot be accessed over a public IP address. To connect to Kafka and Zookeeper from a different machine, you must open ports 9092 and 2181 for remote access. Refer to the FAQ for more information on this.

IMPORTANT: Making this application's network ports public is a significant security risk. You are strongly advised to only allow access to those ports from trusted networks. If, for development purposes, you need to access from outside of a trusted network, please do not allow access to those ports via a public IP address. Instead, use a secure channel such as a VPN or an SSH tunnel. Follow these instructions to remotely connect safely and reliably.

Once you have added the firewall rule and opened the ports, perform these additional steps:

  • Edit your Zookeeper configuration file (/opt/bitnami/zookeeper/conf/zoo.cfg) and comment out the following line:

      clientPortAddress=localhost
    
  • Edit your Kafka configuration file (/opt/bitnami/kafka/config/server.properties). If necessary, uncomment the following line and change the value of the parameter to the public IP address of the server:

      #advertised.host.name=<hostname routable by clients>
    
  • Restart the server to reload the configuration files.

      $ sudo /opt/bitnami/ctlscript.sh restart
    

How to create a Kafka multi-broker cluster?

This section describes the creation of a multi-broker Kafka cluster with brokers located on different hosts. In this scenario:

  • One server hosts the Zookeeper server and a Kafka broker
  • The second server hosts a a second Kafka broker
  • The third server hosts a producer and a consumer

Kafka cluster

NOTE: Before beginning, ensure that ports 2181 (Zookeeper) and 9092 (Kafka) are open on the first server and port 9092 (Kafka) is open on the second server. Also ensure that remote connections are possible between the three servers (instructions).

Configuring the first server (Zookeeper manager and Kafka broker)

The default configuration may be used as is. However, you must perform the steps below:

  • Delete the contents of the Zookeeper and Kafka temporary directories

     $ sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs
     $ sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper
    
  • Restart the Kafka and Zookeeper services.

     $ sudo /opt/bitnami/ctlscript.sh restart kafka
     $ sudo /opt/bitnami/ctlscript.sh restart zookeeper
    

Configuring the second server (Kafka broker)

  • Edit the /opt/bitnami/kafka/config/server.properties configuration file and update the broker.id parameter.

     broker.id = 1
    

    This broker id must be unique in the Kafka ecosystem.

  • In the same file, update the zookeeper.connect parameter to reflect the public IP address of the first server.

     zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
    
  • Delete the contents of the Zookeeper and Kafka temporary directories

     $ sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs
     $ sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper
    
  • Stop the Zookeeper service.

     $ sudo /opt/bitnami/ctlscript.sh stop zookeeper
    
  • Restart the Kafka service.

     $ sudo /opt/bitnami/ctlscript.sh restart kafka
    

Configuring the third server (Kafka message producer/consumer)

  • Edit the /opt/bitnami/kafka/config/producer.properties file and update the metadata.broker.list parameter with the public IP addresses of the two brokers:

     metadata.broker.list=PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092, PUBLIC_IP_ADDRESS_OF_SECOND_KAFKA_BROKER:9092
    
  • Edit the /opt/bitnami/kafka/config/consumer.properties file and update the zookeeper.connect parameter to reflect the public IP address of the first server.

     zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181
    
  • Since this host only serves as a producer and a consumer, stop the Kafka and Zookeeper services:

     $ sudo /opt/bitnami/ctlscript.sh stop kafka
     $ sudo /opt/bitnami/ctlscript.sh stop zookeeper
    

Testing the cluster

NOTE: The following commands should be executed on the third server (Kafka message producer/consumer).
  • Create a new topic.

     $ /opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --replication-factor 2 --partitions 1 --topic multiBroker
    
  • Produce some messages by running the command below and then entering some messages, each on a separate line. Enter Ctrl-C to end.

     $ /opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --topic multiBroker
     this is a message
     this is another message
     ^C
    
  • Consume the messages. The consumer will connect to the cluster and retrieve and display the messages you entered in the previous step.

     $ /opt/bitnami/kafka/bin/kafka-console-consumer.sh --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --topic multiBroker --from-beginning
     this is a message
     this is another message
     ^C
    

How can I run a command in the Bitnami Kafka Stack?

Log in to the server console as the bitnami user and run the command as usual. The required environment is automatically loaded for the bitnami user.

How to run a Kafka producer and consumer from the server itself?

You can run the following example to publish and collect your first message:

  • Declare a new topic. The Kafka server is configured to use the server's public IP address:

     $ /opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test
    

    We use --replication-factor to indicate how many servers are going to have a copy of the logs, and --partitions to choose the number of partitions for the topic we are creating.

  • Start a new producer on the same Kafka server and generates a message in the topic. Remember to replace SERVER-IP with your server's public IP address. Enter CTRL-D to end the message.

     $ /opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list SERVER-IP:9092 --topic test
    
     this is my first message 
    
  • Collect and display the first message in the consumer:

     $ /opt/bitnami/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
    

How to create a full backup of Kafka?

Backup

The Bitnami Kafka Stack is self-contained and the simplest option for performing a backup is to copy or compress the Bitnami stack installation directory. To do so in a safe manner, you will need to stop all servers, so this method may not be appropriate if you have people accessing the application continuously.

Follow these steps:

  • Change to the directory in which you wish to save your backup:

      $ cd /your/directory
    
  • Stop all servers:

      $ sudo /opt/bitnami/ctlscript.sh stop
    
  • Create a compressed file with the stack contents:

      $ sudo tar -pczvf application-backup.tar.gz /opt/bitnami
    
  • Restart all servers:

      $ sudo /opt/bitnami/ctlscript.sh start
    

You should now download or transfer the application-backup.tar.gz file to a safe location.

Restore

Follow these steps:

  • Change to the directory containing your backup:

      $ cd /your/directory
    
  • Stop all servers:

      $ sudo /opt/bitnami/ctlscript.sh stop
    
  • Move the current stack to a different location:

      $ sudo mv /opt/bitnami /tmp/bitnami-backup
    
  • Uncompress the backup file to the original directoryv

      $ sudo tar -pxzvf application-backup.tar.gz -C /
    
  • Start all servers:

      $ sudo /opt/bitnami/ctlscript.sh start
    

If you want to create only a database backup, refer to these instructions for MySQL and PostgreSQL.

How to debug Kafka and Zookeeper errors?

The main Kafka log file is created at /opt/bitnami/kafka/logs/server.log.

The main Zookeeper log file is created at /opt/bitnami/zookeeper/tmp/zookeeper.out.

aws

Bitnami Documentation