Bitnami Elasticsearch for AWS Cloud

Description

Elasticsearch is a popular open source, distributed, enterprise-quality search and analytics engine. Accessible through a REST API, it sifts through large amounts of data extremely quickly.

First steps with the Bitnami Elasticsearch Stack

Welcome to your new Bitnami application running on Amazon Web Services! Here are a few questions (and answers!) you might need when first starting with your application.

What credentials do I need?

You need two sets of credentials:

The application credentials that allow you to log in to your new Bitnami application. These credentials consist of a username and password.
The server credentials that allow you to log in to your AWS Cloud server using an SSH client and execute commands on the server using the command line. These credentials consist of an SSH username and key.

Watch the following video to learn quickly how to obtain the application credentials of those applications deployed using the AWS Console:

What is the administrator username set for me to log in to the application for the first time?

Username: user

How do I get my SSH key or password?

SSH username: bitnami

What are the default ports?

A port is an endpoint of communication in an operating system that identifies a specific process or a type of service. Bitnami stacks include several services or servers that require a port.

IMPORTANT: Making this application's network ports public is a significant security risk. You are strongly advised to only allow access to those ports from trusted networks. If, for development purposes, you need to access from outside of a trusted network, please do not allow access to those ports via a public IP address. Instead, use a secure channel such as a VPN or an SSH tunnel. Follow these instructions to remotely connect safely and reliably.

Port 22 is the default port for SSH connections.

Bitnami opens some ports for the main servers. These are the ports opened by default: 80, 443.

What is the default configuration?

Elasticsearch configuration file

The main configuration file for Elasticsearch is /opt/bitnami/elasticsearch/config/elasticsearch.yml.

Elasticsearch ports

By default, Elasticsearch will use port 9200 for requests and port 9300 for communication between nodes within the cluster. If these ports are in use when the server starts, it will attempt to use the next available port, such as 9201 or 9301.

You can set custom ports using the configuration file, together with details such as the cluster name (elasticsearch by default), node name, address binding and discovery settings. All these settings are needed to add more nodes to your Elasticsearch cluster.

Elasticsearch doesn't include built-in user/password authentication. This can be added using external plugins.

Elasticsearch log file

The Elasticsearch log file is created at /opt/bitnami/elasticsearch/logs/elasticsearch.log.

How to change the Elasticsearch password?

Follow the steps below to change the Elasticsearch password:

Execute the following command. You will be prompted to enter a new password for the user user.

$ sudo /opt/bitnami/apache2/bin/htpasswd -c /opt/bitnami/elasticsearch/apache-conf/password user

Restart the Apache server:

$ sudo /opt/bitnami/ctlscript.sh restart apache

Now, you can access Elasticsearch using the new password.

How to connect to Elasticsearch?

You can connect to Elasticsearch from the same server where it is installed. Run the curl client pointing to the 9200 port like this:

curl -XGET 'localhost:9200/'

How to connect remotely?

How to connect remotely to Elasticsearch?

IMPORTANT: Making this application's network ports public is a significant security risk. You are strongly advised to only allow access to those ports from trusted networks. If, for development purposes, you need to access from outside of a trusted network, please do not allow access to those ports via a public IP address. Instead, use a secure channel such as a VPN or an SSH tunnel. Follow these instructions to remotely connect safely and reliably.

To access the Elasticsearch server from another computer or application, make the following changes to the node's /opt/bitnami/elasticsearch/config/elasticsearch.yml file:

network.host: Specify the hostname or IP address where the server will be accesible. Set it to 0.0.0.0 to listen on every interface.
network.publish_host: Specify the host name that the node publishes to other nodes for communication.

NOTE: Remember to configure the firewall on your server to allow traffic on the ports used by Elasticsearch. Refer to the FAQ for more information.

How to start or stop the services?

Each Bitnami stack includes a control script that lets you easily stop, start and restart services. The script is located at /opt/bitnami/ctlscript.sh. Call it without any service name arguments to start all services:

$ sudo /opt/bitnami/ctlscript.sh start

Or use it to restart a single service, such as Apache only, by passing the service name as argument:

$ sudo /opt/bitnami/ctlscript.sh restart apache

Use this script to stop all services:

$ sudo /opt/bitnami/ctlscript.sh stop

Restart the services by running the script without any arguments:

$ sudo /opt/bitnami/ctlscript.sh restart

Obtain a list of available services and operations by running the script without any arguments:

$ sudo /opt/bitnami/ctlscript.sh

How to install a plugin?

How to install a plugin on Elasticsearch?

Install plugins with the plugin tool provided by Elasticsearch. For example, the command below will install the ICU plugin plugin:

$ cd /opt/bitnami/elasticsearch
$ sudo bin/elasticsearch-plugin install analysis-icu

Once the plugin has been installed, change the user and group ownership of the plugin directory to the elasticsearch user. For example:

$ sudo chown -R elasticsearch:elasticsearch /opt/bitnami/elasticsearch/plugins/analysis-icu/

How to create a full backup of Elasticsearch data?

Backup

Elasticsearch provides a snapshot function that you can use to back up your data. Follow these steps:

Register a repository where the snapshot will be stored. This may be a local directory or cloud storage (which requires additional plugins). In this example, we will use a local repository, which can be initialized via the Elasticsearch REST API with the following commands:
```
 $ cd /home/bitnami
 $ mkdir backups
 $ chown elasticsearch:bitnami /home/bitnami/backups/
 $ chmod u+rwx /home/bitnami/backups/
```
Update the /opt/bitnami/elasticsearch/config/elasticsearch.yml file and add the path.repo variable to it as shown below, pointing to the above repository location:
```
 path.repo: ["/home/bitnami/backups"]
```
Initialize the repository via the Elasticsearch REST API with the following commands:
```
 $ curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
     "type":"fs",
     "settings":{
         "location":"/home/bitnami/backups/my_backup",
         "compress":true
     }
 }'
```
The location property has to be set to the absolute path to the backup files. In this example, my_backup is the name of the backup repository.

See registered repositories with this command:
```
 $ curl -XGET 'http://localhost:9200/_snapshot?pretty'
```
Once the repository is registered, launch the backup with the following command:
```
 $ curl -XPUT 'localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty'
```
In this example, my_backup is the name of the repository created previously and snapshot_1 is the name for the backup. The wait_for_completion option will block the command line until the snapshot is complete. To create the snapshot in the background, simply omit this option, as shown below:
```
 $ curl -XPUT 'localhost:9200/_snapshot/my_backup/snapshot_1'
```

Restore

To restore a backup over existing data, follow these steps:

Close the specific indices that will be overwritten with this command:
```
 $ curl -XPOST 'localhost:9200/my_index/_close'
```
Optionally, close all indices:
```
 $ curl -XPOST 'localhost:9200/_all/_close'
```
Restore the backup with the following command. This command will also reopen the indices closed before.
```
 $ curl -XPOST 'localhost:9200/_snapshot/my_backup/snapshot_1/_restore'
```

For more information, refer to the official documentation.

How to upgrade Elasticsearch?

NOTE: It's highly recommended to perform a backup before any upgrade.

Since version 0.90.7, Elasticsearch supports rolling upgrades. As a result, it's not necessary to stop the entire cluster during the upgrade process. Instead, it is possible to upgrade one node at a time and keep the rest of the cluster operating normally.

To upgrade a node, follow the steps below:

Disable shard reallocation using the command below:

 $ curl -XPUT localhost:9200/_cluster/settings -d '{
     "transient" : {
         "cluster.routing.allocation.enable" : "none"
     }
 }'

Stop the node:

 $ sudo /opt/bitnami/ctlscript.sh stop elasticsearch

Download the latest version from [elastic.co] (https://www.elastic.co/)
Extract to a new directory (not overwriting the current installation) - for example, /tmp/new_elasticsearch.

Rename old files:

 $ cd /opt/bitnami
 $ sudo mv elasticsearch/bin elasticsearch/old_bin
 $ sudo mv elasticsearch/lib elasticsearch/old_lib

Copy files from new installation directory:

 $ sudo cp -r /tmp/new_elasticsearch/bin elasticsearch/bin
 $ sudo cp -r /tmp/new_elasticsearch/lib elasticsearch/lib

Start the node again:

 $ sudo /opt/bitnami/ctlscript.sh start elasticsearch

Re-enable shard reallocation:

 $ curl -XPUT localhost:9200/_cluster/settings -d '{
     "transient" : {
         "cluster.routing.allocation.enable" : "all"
     }
 }'

Repeat the process for all remaining nodes of your cluster.

How to create an SSL certificate?

OpenSSL is required to create an SSL certificate. A certificate request can then be sent to a certificate authority (CA) to get it signed into a certificate, or if you have your own certificate authority, you may sign it yourself, or you can use a self-signed certificate (because you just want a test certificate or because you are setting up your own CA).

Follow the steps below:

Generate a new private key:

 $ sudo openssl genrsa -out /opt/bitnami/apache2/conf/server.key 2048

Create a certificate:
```
 $ sudo openssl req -new -key /opt/bitnami/apache2/conf/server.key -out /opt/bitnami/apache2/conf/cert.csr
```
IMPORTANT: Enter the server domain name when the above command asks for the "Common Name".
Send cert.csr to the certificate authority. When the certificate authority completes their checks (and probably received payment from you), they will hand over your new certificate to you.

Until the certificate is received, create a temporary self-signed certificate:

 $ sudo openssl x509 -in /opt/bitnami/apache2/conf/cert.csr -out /opt/bitnami/apache2/conf/server.crt -req -signkey /opt/bitnami/apache2/conf/server.key -days 365

Back up your private key in a safe location after generating a password-protected version as follows:
```
 $ sudo openssl rsa -des3 -in /opt/bitnami/apache2/conf/server.key -out privkey.pem
```
Note that if you use this encrypted key in the Apache configuration file, it will be necessary to enter the password manually every time Apache starts. Regenerate the key without password protection from this file as follows:
```
 $ sudo openssl rsa -in privkey.pem -out /opt/bitnami/apache2/conf/server.key
```

Find more information about certificates at http://www.openssl.org.

How to enable HTTPS support with SSL certificates?

TIP: If you wish to use a Let's Encrypt certificate, you will find specific instructions for enabling HTTPS support with Let's Encrypt SSL certificates in our Let's Encrypt guide.

NOTE: The steps below assume that you are using a custom domain name and that you have already configured the custom domain name to point to your cloud server.

Bitnami images come with SSL support already pre-configured and with a dummy certificate in place. Although this dummy certificate is fine for testing and development purposes, you will usually want to use a valid SSL certificate for production use. You can either generate this on your own (explained here) or you can purchase one from a commercial certificate authority.

Once you obtain the certificate and certificate key files, you will need to update your server to use them. Follow these steps to activate SSL support:

Use the table below to identify the correct locations for your certificate and configuration files.

Variable	Value
Current application URL	https://[custom-domain]/
	Example: https://my-domain.com/ or https://my-domain.com/appname
Apache configuration file	/opt/bitnami/apache2/conf/bitnami/bitnami.conf
Certificate file	/opt/bitnami/apache2/conf/server.crt
Certificate key file	/opt/bitnami/apache2/conf/server.key
CA certificate bundle file (if present)	/opt/bitnami/apache2/conf/server-ca.crt

Copy your SSL certificate and certificate key file to the specified locations.

NOTE: If you use different names for your certificate and key files, you should reconfigure the SSLCertificateFile and SSLCertificateKeyFile directives in the corresponding Apache configuration file to reflect the correct file names.

If your certificate authority has also provided you with a PEM-encoded Certificate Authority (CA) bundle, you must copy it to the correct location in the previous table. Then, modify the Apache configuration file to include the following line below the SSLCertificateKeyFile directive. Choose the correct directive based on your scenario and Apache version:

Variable	Value
Apache configuration file	/opt/bitnami/apache2/conf/bitnami/bitnami.conf
Directive to include (Apache v2.4.8+)	SSLCACertificateFile "/opt/bitnami/apache2/conf/server-ca.crt"
Directive to include (Apache < v2.4.8)	SSLCertificateChainFile "/opt/bitnami/apache2/conf/server-ca.crt"

NOTE: If you use a different name for your CA certificate bundle, you should reconfigure the SSLCertificateChainFile or SSLCACertificateFile directives in the corresponding Apache configuration file to reflect the correct file name.

Once you have copied all the server certificate files, you may make them readable by the root user only with the following commands:
```
 $ sudo chown root:root /opt/bitnami/apache2/conf/server*

 $ sudo chmod 600 /opt/bitnami/apache2/conf/server*
```
Open port 443 in the server firewall. Refer to the FAQ for more information.
Restart the Apache server.

You should now be able to access your application using an HTTPS URL.

How to force HTTPS redirection with Apache?

Add the following lines in the default Apache virtual host configuration file at /opt/bitnami/apache2/conf/bitnami/bitnami.conf, inside the default VirtualHost directive, so that it looks like this:

<VirtualHost _default_:80>
  DocumentRoot "/opt/bitnami/apache2/htdocs"
  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]
  ...
</VirtualHost>

After modifying the Apache configuration files:

Open port 443 in the server firewall. Refer to the FAQ for more information.
Restart Apache to apply the changes.

How to debug Apache errors?

Once Apache starts, it will create two log files at /opt/bitnami/apache2/logs/access_log and /opt/bitnami/apache2/logs/error_log respectively.

The access_log file is used to track client requests. When a client requests a document from the server, Apache records several parameters associated with the request in this file, such as: the IP address of the client, the document requested, the HTTP status code, and the current time.
The error_log file is used to record important events. This file includes error messages, startup messages, and any other significant events in the life cycle of the server. This is the first place to look when you run into a problem when using Apache.

If no error is found, you will see a message similar to:

Syntax OK

Updating the IP address or hostname

Elasticsearch requires updating the IP address/domain name if the machine IP address/domain name changes. The bnconfig tool also has an option which updates the IP address, called --machine_hostname (use --help to check if that option is available for your application). Note that this tool changes the URL to http://NEW_DOMAIN/elasticsearch.

$ sudo /opt/bitnami/apps/elasticsearch/bnconfig --machine_hostname NEW_DOMAIN

If you have configured your machine to use a static domain name or IP address, you should rename or remove the /opt/bitnami/apps/elasticsearch/bnconfig file.

$ sudo mv /opt/bitnami/apps/elasticsearch/bnconfig /opt/bitnami/apps/elasticsearch/bnconfig.disabled

NOTE: Be sure that your domain is propagated. Otherwise, this will not work. You can verify the new DNS record by using the Global DNS Propagation Checker and entering your domain name into the search field.

You can also change your hostname by modifying it in your hosts file. Enter the new hostname using your preferred editor.

$ sudo nano /etc/hosts

Add a new line with the IP address and the new hostname. Here's an example. Remember to replace the IP-ADDRESS and DOMAIN placeholders with the correct IP address and domain name.

IP-ADDRESS DOMAIN

How to upload files to the server with SFTP?

Although you can use any SFTP/SCP client to transfer files to your server, the link below explains how to configure FileZilla (Windows, Linux and Mac OS X), WinSCP (Windows) and Cyberduck (Mac OS X). It is required to use your server's private SSH key to configure the SFTP client properly. Choose your preferred application and follow the steps in the link below to connect to the server through SFTP.

How to upload files to the server

How to add nodes to an Elasticsearch cluster?

To add additional nodes to a cluster, update the following configuration parameters in the node's /opt/bitnami/elasticsearch/config/elasticsearch.yml file:

cluster.name: All the nodes should have the same cluster name to work properly.
node.name: The name of each node should be unique. Set meaningful names to your nodes according to their functions so it will be easier to identify them.
network.publish_host: The host name that a node publishes to other nodes for communication. This host should be accessible at least from the master node.
discovery.zen.ping.unicast.hosts: When nodes are in the same sub-network, they will auto-configure themselves into a cluster. In other cases, specify a list with your nodes in this parameter.

Refer to the Elasticsearch official documentation for more information.

How to debug Elasticsearch errors?

The Elasticsearch log files are created at /opt/bitnami/elasticsearch/logs/.

How to install elasticsearch-head?

Elasticsearch-head is a Web front-end for an Elasticsearch cluster. For Elasticsearch 5.x, site plugins are not supported, so it needs to run as a standalone server. Follow these steps:

Install Node.js and npm. For example, the commands below will install them on Debian:
```
 $ sudo apt install nodejs-legacy npm
```

Download the elasticsearch-head ZIP file and decompress it:

 $ wget https://github.com/mobz/elasticsearch-head/archive/master.zip
 $ unzip master.zip

Install the modules and run the service:

 $ cd elasticsearch-head-master
 $ npm install
 $ ./node_modules/grunt/bin/grunt server &

Update the /opt/bitnami/elasticsearch/config/elasticsearch.yml file and enable CORS by setting http.cors.enabled to true:
```
 http.cors.enabled: true
```
In the same file, set the http.cors.allow-origin variable to the domains that are allowed to send cross-origin requests. If you prepend and append a "/" to the value, this will be treated as a regular expression. For example:
```
 http.cors.allow-origin: /https?:\/\/localhost(:[0-9]+)?/
```

NOTE: You can set the value of http.cors.allow-origin to "*" to allow CORS requests from anywhere if you wish. However, this is not recommended as it is a security risk.

Add Apache configuration for elasticsearch-head to /opt/bitnami/elasticsearch/apache-conf/elasticsearch.conf:

 ProxyPass        /elasticsearch-head http://127.0.0.1:9100
 ProxyPassReverse /elasticsearch-head http://127.0.0.1:9100

Restart the services:

 $ sudo /opt/bitnami/ctlscript.sh restart apache

Browse to http://SERVER-IP/elasticsearch-head/?base_uri=http://SERVER-IP/elasticsearch and insert your Elasticsearch credentials. You should see something like the screenshot below:

Which components are installed with the Bitnami Elasticsearch Stack?

The Bitnami Elasticsearch Stack ships the components listed below. If you want to know which specific version of each component is bundled in the stack you are downloading, check the README.txt file on the download page or in the stack installation directory. You can also find more information about each component using the links below.

Main components

Elasticsearch
Apache Web server