Bitnami ZooKeeper for Google Cloud Platform

Description

ZooKeeper provides a reliable, centralized register of configuration data and services for distributed applications.

First steps with the Bitnami ZooKeeper Stack

Welcome to your new Bitnami application running on Google Cloud Platform! Here are a few questions (and answers!) you might need when first starting with your application.

What credentials do I need?

You need two sets of credentials:

The application credentials that allow you to log in to your new Bitnami application. These credentials consist of a username and password.
The server credentials that allow you to log in to your Google Cloud Platform server using an SSH client and execute commands on the server using the command line. These credentials consist of an SSH username and key.

What is the administrator username set for me to log in to the application for the first time?

Username: zkcli

What SSH username should I use for secure shell access to my application?

SSH username: bitnami

How to start or stop the services?

Each Bitnami stack includes a control script that lets you easily stop, start and restart services. The script is located at /opt/bitnami/ctlscript.sh. Call it without any service name arguments to start all services:

$ sudo /opt/bitnami/ctlscript.sh start

Or use it to restart a single service, such as Apache only, by passing the service name as argument:

$ sudo /opt/bitnami/ctlscript.sh restart apache

Use this script to stop all services:

$ sudo /opt/bitnami/ctlscript.sh stop

Restart the services by running the script without any arguments:

$ sudo /opt/bitnami/ctlscript.sh restart

Obtain a list of available services and operations by running the script without any arguments:

$ sudo /opt/bitnami/ctlscript.sh

ZooKeeper default configuration

ZooKeeper configuration files

The ZooKeeper configuration files are located at the /opt/bitnami/zookeeper/conf/ directory.

ZooKeeper ports

By default, the ZooKeeper server runs on port 2181. Only conections from localhost are permitted.

ZooKeeper log files

The ZooKeeper log file is /opt/bitnami/zookeeper/logs/zookeeper.out.

What are the default ports?

A port is an endpoint of communication in an operating system that identifies a specific process or a type of service. Bitnami stacks include several services or servers that require a port.

IMPORTANT: Making this application's network ports public is a significant security risk. You are strongly advised to only allow access to those ports from trusted networks. If, for development purposes, you need to access from outside of a trusted network, please do not allow access to those ports via a public IP address. Instead, use a secure channel such as a VPN or an SSH tunnel. Follow these instructions to remotely connect safely and reliably.

Port 22 is the default port for SSH connections.

The ZooKeeper access port is 2181. This port is closed by default. You must open it to enable remote access.

How to upload files to the server with SFTP?

NOTE: Bitnami applications can be found in /opt/bitnami/apps.

If you are using the Bitnami Launchpad for Google Cloud Platform, obtain your server SSH key by following these steps:
- Browse to the Bitnami Launchpad for Google Cloud Platform dashboard and sign in if required using your Bitnami account.
- Select the "Virtual Machines" menu item.
- Select your cloud server from the resulting list.
- Download the SSH key for your server in PPK or PEM format. Note the server IP address on the same page.

If you are using the GCP Marketplace, you will need to generate and add your SSH key manually using these instructions.
Generate SSH key pair by executing the following commands:

NOTE: Replace USERNAME in the commands below with your Google Cloud platform username.

	$ sudo su USERNAME
	$ ssh-keygen -t rsa -f ~/.ssh/my-ssh-key -C USERNAME

Enter the passphrase twice. The SSH key pair will be generated and saved in /home/USERNAME/.ssh/my-ssh-key and /home/USERNAME/.ssh/my-ssh-key.pub.

Although you can use any SFTP/SCP client to transfer files to your server, this guide documents FileZilla (Windows, Linux and Mac OS X), WinSCP (Windows) and Cyberduck (Mac OS X).

Using an SSH Key

Once you have your server's SSH key, choose your preferred application and follow the steps below to connect to the server using SFTP.

FileZilla

IMPORTANT: To use FileZilla, your server private key should be in PPK format.

Watch the following video to learn how to upload files to your Google Cloud server with SFTP using FileZilla. The example below shows a server launched using the Bitnami launchpad, but this video is applicable to those servers launched using the GCP Marketplace.

Or you can follow these steps:

Download and install FileZilla.
Launch FileZilla and use the "Edit -> Settings" command to bring up FileZilla's configuration settings.
Within the "Connection -> SFTP" section, use the "Add keyfile" command to select the private key file for the server. FileZilla will use this private key to log in to the server.
Use the "File -> Site Manager -> New Site" command to bring up the FileZilla Site Manager, where you can set up a connection to your server.
Enter your server host name and specify bitnami as the user name.
Select "SFTP" as the protocol and "Ask for password" as the logon type.
Use the "Connect" button to connect to the server and begin an SFTP session. You might need to accept the server key, by clicking "Yes" or "OK" to proceed.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

If you have problems accessing your server, get extra information by use the "Edit -> Settings -> Debug" menu to activate FileZilla's debug log.

WinSCP

IMPORTANT: To use WinSCP, your server private key should be in PPK format.

Follow these steps:

Download and install WinSCP.
Launch WinSCP and in the "Session" panel, select "SCP" as the file protocol.
Enter your server host name and specify bitnami as the user name.
Click the "Advanced…" button and within the "SSH -> Authentication -> Authentication parameters" section, select the private key file for the server. WinSCP will use this private key to log in to the server.
From the "Session" panel, use the "Login" button to connect to the server and begin an SCP session.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

If you need to upload files to a location where the bitnami user doesn't have write permissions, you have two options:

Once you have configured WinSCP as described above, click the "Advanced…" button and within the "Environment -> Shell" panel, select sudo su - as your shell. This will allow you to upload files using the administrator account.
Upload the files to the /home/bitnami directory as usual. Then, connect via SSH and move the files to the desired location with the sudo command, as shown below:
```
 $ sudo mv /home/bitnami/uploaded-file /path/to/desired/location/
```

Cyberduck

IMPORTANT: To use Cyberduck, your server private key should be in PEM format.

Follow these steps:

Select the "Open Connection" command and specify "SFTP" as the connection protocol.
In the connection details panel, under the "More Options" section, enable the "Use Public Key Authentication" option and specify the path to the private key file for the server.
Use the "Connect" button to connect to the server and begin an SFTP session.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

How to connect instances hosted in separate virtual networks or VPCs?

The Google Cloud Platform makes it possible to connect instances hosted in separate Virtual Private Clouds (VPCs), even if those instances belong to different projects or are hosted in different regions. This feature, known as VPC Network Peering, can result in better security (as services do not need to be exposed on public IP addresses) and performance (due to use of private, rather than public, networks and IP addresses).

Learn more about VPC Network Peering.

How to connect to ZooKeeper from the same server?

You can connect to ZooKeeper from the same server where it is installed with the ZooKeeper client tool. Execute the following command:

$ zkCli.sh

You should see an output similar to this:

You can run the following example to see how to add new data to ZooKeeper, recover it, and delete the created registries:

Use the list command to see the data stored in the instance:
```
$ ls /
[zookeeper]
```
Create a new directory (in this example, test directory which includes the string mydata) using the create / command. This creates a new znode and associates the string "my_data" with the node. You should see:
```
$ create /test mydata
Created /test
```
Use the list command again to see how the directory looks like now:
```
$ ls /
[zookeeper, test]
```

Check the information associated to the new node by using the get / command:

$ get /test
mydata
cZxid = 0x23
ctime = Fri Jun 01 14:43:23 UTC 2018
mZxid = 0x23
mtime = Fri Jun 01 14:43:23 UTC 2018
pZxid = 0x23
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 6
numChildren = 0

Change the data associated to the test node using the set / command:

$ set /test apache
cZxid = 0x23
ctime = Fri Jun 01 14:43:23 UTC 2018
mZxid = 0x24
mtime = Fri Jun 01 14:51:07 UTC 2018
pZxid = 0x23
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 6
numChildren = 0

Run again the get / command to check that the data has been changed from mydata to apache:

$ get /test
apache
cZxid = 0x23
ctime = Fri Jun 01 14:43:23 UTC 2018
mZxid = 0x24
mtime = Fri Jun 01 14:51:07 UTC 2018
pZxid = 0x23
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 6
numChildren = 0

Delete the node with the delete / command:
```
$ delete /test
```
If you execute again the ls / command you should see only the [zookeeper] node.

How to connect to ZooKeeper from a different machine?

For security reasons, the ZooKeeper ports in this solution cannot be accessed over a public IP address. To connect to ZooKeeper from a different machine, you must open port Bitnami does not open any port by default in this solution for remote access. Refer to the FAQ for more information on this.

IMPORTANT: Making this application's network ports public is a significant security risk. You are strongly advised to only allow access to those ports from trusted networks. If, for development purposes, you need to access from outside of a trusted network, please do not allow access to those ports via a public IP address. Instead, use a secure channel such as a VPN or an SSH tunnel. Follow these instructions to remotely connect safely and reliably.

How to create a ZooKeeper ensemble?

This section describes the creation of a ZooKeeper ensemble with servers located on different hosts. The following example shows an ensemble comprised of three instances, follow these instructions:

Launch as much ZooKeeper instances as nodes you want to have in the ensemble. (In this example, three instances).
Connect their Virtual Private Clouds (VPCs) by configuring VPC Network Peering.

Configuring the first ZooKeeper instance

Check that the value of the /opt/bitnami/zookeeper/data/myid file is equal to 1.
Edit the /opt/bitnami/zookeeper/conf/zoo.cfg file to add the following content. Remember to replace IP_ADDRESS with the corresponding IP address of each instance.
```
server.1=IP_ADDRESS_SERVER1:2888:3888
server.2=IP_ADDRESS_SERVER2:2888:3888
server.3=IP_ADDRESS_SERVER3:2888:3888
```

Restart ZooKeeper services:

$ sudo /opt/bitnami/ctlscript.sh restart zookeeper

Configuring the second ZooKeeper instance

Edit the /opt/bitnami/zookeeper/data/myid file and change the value 1 to 2.
Edit the /opt/bitnami/zookeeper/conf/zoo.cfg file to add the following content. Remember to replace IP_ADDRESS with the corresponding IP address of each instance.
```
server.1=IP_ADDRESS_SERVER1:2888:3888
server.2=IP_ADDRESS_SERVER2:2888:3888
server.3=IP_ADDRESS_SERVER3:2888:3888
```

Restart ZooKeeper services:

$ sudo /opt/bitnami/ctlscript.sh restart zookeeper

Configuring the third ZooKeeper instance

Edit the /opt/bitnami/zookeeper/data/myid file and change the value 1 to 3.
Edit the /opt/bitnami/zookeeper/conf/zoo.cfg file to add the following content. Remember to replace IP_ADDRESS with the corresponding IP address of each instance.
```
server.1=IP_ADDRESS_SERVER1:2888:3888
server.2=IP_ADDRESS_SERVER2:2888:3888
server.3=IP_ADDRESS_SERVER3:2888:3888
```

Restart ZooKeeper services:

$ sudo /opt/bitnami/ctlscript.sh restart zookeeper

Testing the ensemble

To check if the replication is working correctly in the cluster, let's try the following:

In the server 1, connect to ZooKeeper by executing the following:
```
$ zkCli.sh
```
Create a new directory named ensemble with a string named test using the create / command. This creates a new znode and associates the string test with the node. You should see:
```
$ create /ensemble test
Created /ensemble
```
Use the list command again to see how the directory looks like now:
```
$ ls /
[zookeeper, ensemble]
```
Go to the server 2 and connect to ZooKeeper using the zkCli.sh command.
Use the ls / command to check that you have the same data in both instances. You should see the same output you have obtained in the server 1 when executing that command:
```
$ ls /
[zookeeper, ensemble]
```

Congratulations! You now have a ZooKeeper ensemble running and ready to replicate the data through all the instances/nodes.

How can I run a command in the Bitnami ZooKeeper Stack?

Log in to the server console as the bitnami user and run the command as usual. The required environment is automatically loaded for the bitnami user.

How to create a full backup of ZooKeeper?

Backup

The Bitnami ZooKeeper Stack is self-contained and the simplest option for performing a backup is to copy or compress the Bitnami stack installation directory. To do so in a safe manner, you will need to stop all servers, so this method may not be appropriate if you have people accessing the application continuously.

Follow these steps:

Change to the directory in which you wish to save your backup:
```
  $ cd /your/directory
```

Stop all servers:

  $ sudo /opt/bitnami/ctlscript.sh stop

Create a compressed file with the stack contents:

  $ sudo tar -pczvf application-backup.tar.gz /opt/bitnami

Restart all servers:

  $ sudo /opt/bitnami/ctlscript.sh start

You should now download or transfer the application-backup.tar.gz file to a safe location.

Restore

Follow these steps:

Change to the directory containing your backup:
```
  $ cd /your/directory
```

Stop all servers:

  $ sudo /opt/bitnami/ctlscript.sh stop

Move the current stack to a different location:

  $ sudo mv /opt/bitnami /tmp/bitnami-backup

Uncompress the backup file to the original directoryv

  $ sudo tar -pxzvf application-backup.tar.gz -C /

Start all servers:

  $ sudo /opt/bitnami/ctlscript.sh start

If you want to create only a database backup, refer to these instructions for MySQL and PostgreSQL.

How to debug Zookeeper errors?

The main Zookeeper log file is created at /opt/bitnami/zookeeper/logs/zookeeper.out.