Bitnami MongoDB for Microsoft Azure Multi-Tier Solutions

Description

MongoDB is a scalable, high-performance, open source NoSQL database written in C++.

How to start or stop the servers?

Each Bitnami server includes a control script that lets you easily stop, start and restart all the servers installed in the system.

Obtain the status of a service with the service bitnami status command:

$ sudo service bitnami status

Use the service bitnami command to start, stop or restart all the servers in a similar manner:

  • Start all the servers.

    $ sudo service bitnami start
    
  • Stop all the servers.

    $ sudo service bitnami stop
    
  • Restart all the servers.

    $ sudo service bitnami restart
    
NOTE: The steps below require to execute the commands in the remote server. Please check how to connect to your server through SSH for more information on this.

How to upload files with SFTP?

NOTE: Bitnami applications can be found in /opt/bitnami.

When uploading files to the application server via SFTP, you will need the key pair associated with the servers during the deployment process.

Azure server credentials

You will also need the public IP address of the server hosting the application. This may be obtained from the corresponding server detail page in the Azure management console, as shown below:

Azure server IP address

Although you can use any SFTP/SCP client to transfer files to your server, this guide documents FileZilla (Windows, Linux and Mac OS X), WinSCP (Windows) and Cyberduck (Mac OS X).

Using an SSH Key

Once you have your server's SSH key, choose your preferred application and follow the steps below to connect to the server using SFTP.

FileZilla
IMPORTANT: To use FileZilla, your server private key should be in PPK format.

Follow these steps:

  • Download and install FileZilla.
  • Launch FileZilla and use the "Edit -> Settings" command to bring up FileZilla's configuration settings.
  • Within the "Connection -> SFTP" section, use the "Add keyfile" command to select the private key file for the server. FileZilla will use this private key to log in to the server. FileZilla configuration
  • Use the "File -> Site Manager -> New Site" command to bring up the FileZilla Site Manager, where you can set up a connection to your server.
  • Enter your server host name and specify bitnami as the user name.
  • Select "SFTP" as the protocol and "Ask for password" as the logon type. FileZilla configuration
  • Use the "Connect" button to connect to the server and begin an SFTP session. You might need to accept the server key, by clicking "Yes" or "OK" to proceed.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

If you have problems accessing your server, get extra information by use the "Edit -> Settings -> Debug" menu to activate FileZilla's debug log.

FileZilla debug log

WinSCP
IMPORTANT: To use WinSCP, your server private key should be in PPK format.

Follow these steps:

  • Download and install WinSCP.
  • Launch WinSCP and in the "Session" panel, select "SFTP" as the file protocol.
  • Enter your server host name and specify bitnami as the user name. WinSCP configuration
  • Click the "Advanced…" button and within the "SSH -> Authentication -> Authentication parameters" section, select the private key file for the server. WinSCP will use this private key to log in to the server. WinSCP configuration
  • From the "Session" panel, use the "Login" button to connect to the server and begin an SCP session.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

If you need to upload files to a location where the bitnami user doesn't have write permissions, you have two options:

  • Once you have configured WinSCP as described above, click the "Advanced…" button and within the "Environment -> Shell" panel, select sudo su - as your shell. This will allow you to upload files using the administrator account. WinSCP configuration
  • Upload the files to the /home/bitnami directory as usual. Then, connect via SSH and move the files to the desired location with the sudo command, as shown below:

       $ sudo mv /home/bitnami/uploaded-file /path/to/desired/location/
    
Cyberduck
IMPORTANT: To use Cyberduck, your server private key should be in PEM format.

Follow these steps:

  • Select the "Open Connection" command and specify "SFTP" as the connection protocol. Cyberduck configuration

  • In the connection details panel, under the "More Options" section, enable the "Use Public Key Authentication" option and specify the path to the private key file for the server. Cyberduck configuration

  • Use the "Connect" button to connect to the server and begin an SFTP session.

You should now be logged into the /home/bitnami directory on the server. You can now transfer files by dragging and dropping them from the local server window to the remote server window.

What is the default configuration?

The MongoDB admin user for all databases is created during the Bitnami Stack installation process. The default configuration consists of:

  • The default data directory in Bitnami is located at /opt/bitnami/mongodb/data/db.
  • A privileged account with a username of root. The root user has remote access to the database.

MongoDB version

In order to see which MongoDB version your system is running, execute the following command:

$ mongod --version

MongoDB configuration file

The MongoDB configuration file is located at /opt/bitnami/mongodb/conf/mongodb.conf.

The official MongoDB documentation has more details about how configure the MongoDB database.

MongoDB socket

On Unix, the MongoDB clients can connect to the server using a Unix socket file at /opt/bitnami/mongodb/tmp/mongodb-27017.sock. The name of the file includes the port number: 27017.

Usually, when you use the MongoDB client tool included in the Stack, you will not need to specify the socket for the connection.

MongoDB port

The default port in which MongoDB listens is 27017.

MongoDB Process Identification Number

The MongoDB .pid file allows other programs to find out the PID (Process Identification Number) of a running script. Find it at /opt/bitnami/mongodb/tmp/mongodb.pid.

MongoDB log file

The main MongoDB log file is at /opt/bitnami/mongodb/logs/mongodb.log.

MongoDB key file

Find the security options in the key file located at /opt/bitnami/mongodb/conf/keyfile.

How is the cluster configured?

The Bitnami Multi-Tier Solution for MongoDB uses multiple VMs, consisting of:

  • Three nodes: one primary and two secondary nodes. You also may add an arbiter node which is used whereas the primary node may step down and become a secondary and a secondary may become the primary during an election). This configuration provides a horizontally scalable and fault-tolerant deployment.

    • The minimum requirement for a MongoDB cluster is to have at least, two nodes: one primary and one, as secondary node. A replica set can have up to 50 nodes, but only 7 could be voting members. Learn more about MongoDB Replica Set Elections.
  • Data doesn't automatically replicates from the primary node to all secondary nodes. You have to authorize the secondary node to read the primary.

  • Binary logging is enabled and the MongoDB server is configured to listen for connections on network address 0.0.0.0.

To understand how it works, consider the example below of a three-node cluster/ replica set (one primary, one secondary and one arbiter):

  • On the primary node, see the databases you have:

     replicaset:PRIMARY> show dbs
     admin    0.000GB
     local    0.000GB
     bitnamidb  0.000GB
     replicaset:PRIMARY>
    
  • Create a database and add a collection in the primary node:

     replicaset:PRIMARY> use bitnamidb
     switched to db bitnamidb
     replicaset:PRIMARY> db.createCollection('bitnamicol')
     { "ok" : 1 }
    
  • Check the databases on your secondary node. As you can see below, there is not replication in it:

    replicaset:SECONDARY> show dbs
    2016-12-22T10:23:28.789+0000 E QUERY    [main] Error:
    listDatabases failed:{
    "ok" : 0,
    "errmsg" : "not master and slaveOk=false",
    "code" : 13435,
    "codeName" : "NotMasterNoSlaveOk"
    } :
    _getErrorWithCode@src/mongo/shell/utils.js:25:13
    Mongo.prototype.getDBs@src/mongo/shell/mongo.js:62:1
    shellHelper.show@src/mongo/shell/utils.js:755:19
    shellHelper@src/mongo/shell/utils.js:645:15
    @(shellhelp2):1:1
    
  • Allow the replica into the secondary node and check the databases again:

    replicaset:SECONDARY> rs.slaveOk()
    replicaset:SECONDARY> show dbs
    admin    0.000GB
    local    0.000GB
    bitnamidb  0.000GB
    replicaset:SECONDARY>
    

This shows that the databases added on the primary node have been replicated to the secondary node(s).

  • Check the databases on the arbiter node. It is not allowed to replicate data or accept write operations even when you try to authorize it to read the primary node.

    replicaset:ARBITER> show dbs
    2016-12-22T10:26:04.186+0000 E QUERY    [main] Error: listDatabases failed:{
    "ok" : 0,
    "errmsg" : "not master and slaveOk=false",
    "code" : 13435,
    "codeName" : "NotMasterNoSlaveOk"
    } :
    _getErrorWithCode@src/mongo/shell/utils.js:25:13
    Mongo.prototype.getDBs@src/mongo/shell/mongo.js:62:1
    shellHelper.show@src/mongo/shell/utils.js:755:19
    shellHelper@src/mongo/shell/utils.js:645:15
    @(shellhelp2):1:1
    replicaset:ARBITER> rs.slaveOk()
    replicaset:ARBITER> show dbs
    admin  0.000GB
    local  0.000GB
    replicaset:ARBITER>
    

For more information, refer to the MongoDB documentation on replication.

How to connect to the MongoDB database?

You can connect to the MongoDB database from the same server where it is installed. Run the mongo client. You need to authenticate for accessing the database:

$ mongo admin --username root -p
MongoDB shell version v3.4.1
Enter password:
connecting to: mongodb://127.0.0.1:27017/admin
MongoDB server version: 3.4.1
replicaset:PRIMARY>

The default username for Multi-Tier Solutions is root. Check how to find your application credentials to get your password.

How to connect to MongoDB from a different machine?

NOTE: You will only able to connect to the MongoDB instances from machines that are running in the same network.

You can now connect to each MongoDB instances using a command like the one below:

$ mongo admin --username root -p -host SERVER-IP --port 27017

How to create a database for a custom application?

If you want to install an application manually, it may require the database settings during the installation. These are the basic steps to create a database for your application.

$ mongo admin --u root -p
MongoDB shell version: 2.4.8
connecting to: 127.0.0.1:27017/admin
> db = db.getSiblingDB('database_name')
database_name
> db.createUser( { user: "database_user", pwd: "database_password", roles: [ "readWrite", "dbAdmin" ]} )
{
 "user" : "database_user",
 "pwd" : "...",
 "roles" : [
  "readWrite",
  "dbAdmin"
 ],
 "_id" : ObjectId("...")
}
> exit

Some applications may require specific privileges in the database. Consult the official MongoDB documentation to find more information.

How to create an user with restricted privileges in an existing database?

In case you already have a database created, you can create a new user with restricted privileges. Find an example below of how to create a new user with reading only privileges:

$ mongo database_name --u root -p
> db.createUser( { user: "database_user", pwd: "database_password", roles: [ "readWrite"]} )
{
   "user" : "database_user",
    "pwd" : "...",
     "roles" : [
       "readWrite",
          ],
           "_id" : ObjectId("...")
}
> exit

How to create a database backup?

To back up the data contained in your database, create a dump file using the mongodump tool.

$ mongodump --authenticationDatabase admin -d DATABASE_NAME

This operation could take some time depending on the amount of data that you have stored in the database.

How to restore a database backup?

To restore data backed up using the previous command, restore a dump file using the mongorestore tool.

$ mongorestore --authenticationDatabase admin PATH_TO_BACKUP_FILE

Note that the steps previously described will only back up the data contained inside your database. There may be other files that you should take into account when performing a full backup, such as files that may have been uploaded to the application. These files are stored in the application folder itself, so copy this folder to have a backup of your uploaded files.

How to recover your database from MongoDB Oplog (operation log)?

How does the MongoDB Oplog works?

The Oplog (operations log) is a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases. MongoDB applies database operations on the primary and then records the operations on the primary's oplog. The secondary members then copy and apply these operations in an asynchronous process. All replica set members contain a copy of the oplog, in the local.oplog.rs collection, which allows them to maintain the current state of the database.

The MongoDB Oplog is created after starting a replica set member for the first time with a default size. After that, MongoDB Oplog size will grow up to 5% of free disk space. In the Bitnami Multi-Tier Solution for MongoDB, the replica sets are deployed with 10GB of data disk by default.

For more information, refer to this section of the MongoDB official documentation.

How to check the MongoDB Oplog status?

In order to check if you can use the MongoDB Oplog to recover the database, you can follow the steps below:

  • Create a database backup following this guide.

  • Connect to the primary server via ssh (refer to the FAQ for more information).

  • Connect to the MongoDB database and run the commands below:

     $ mongo admin -u root -p
     replicaset:PRIMARY> use local
     replicaset:PRIMARY> db.oplog.rs.find();
    

    Please note that you get "PRIMARY" when accessing the MongoDB Shell

  • The previous commands should print out the local MongoDB Oplog. If any of those instructions fail, or nothing is displayed, you'll not be able to use this method to recover the database. If you created a snapshot of your servers you can try to recover the data deploying a new server using that backup.

How to recover your database?

Once you check the MongoDB Oplog exists and it's available, you can proceed to dump and restore your database from it. Follow the following steps to do so:

  • Dump the MongoDB Oplog to a folder (e.g. ~/oplog):

     $ mongodump -u root --authenticationDatabase admin -d local -c oplog.rs -o ~/oplog
    
  • Create a copy of the dump file to ensure you can manipulate it safely:

     $ mkdir ~/oplog2 && cp ~/oplog/local/oplog.rs.bson ~/oplog2/oplog.bson
    
  • Use the command bsondump to inspect the .bson files and find the restoring limit:

     $ bsondump ~/oplog2/oplog.bson
    
  • You should obtain an output similar to the one below:

     {"ts":{"$timestamp":{"t":1484002910,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"7714671726931207910"},"v":2,"op":"c","ns":"test.$cmd","o":{"drop":"testData"}}
    
  • Note the "t" and "i" attributes of the restore point (Note that this point won't be included in the restore), as they are the ones used to restore the log (In this example: "t":1484002910,"i":1).

  • You can also use that tool to find offending commands (for example you can look for "drop" commands, but also "remove", "deleteOne", and "deleteMany" would be suspicious). If the dump is too big, you can use grep or any other tool to find the commands:

     $ bsondump ~/oplog2/oplog.bson | grep drop
    
  • If running mongorestore with –oplogReplay, the restore role is insufficient to replay the oplog. To replay the oplog, create a user-defined role that has anyAction on anyResource and grant only to users who must run mongorestore with –oplogReplay.

     $ mongo admin -u root -p
     replicaset:PRIMARY> db.createRole( { role: "executeFunctions", privileges: [ { resource: { anyResource: true }, actions: [ "anyAction" ] } ], roles: [] } )
     replicaset:PRIMARY> db.grantRolesToUser("root", [ { role: "executeFunctions", db: "admin" } ])
    
  • Restore your database using that point as the restore limit:

     $ mongorestore -u root --authenticationDatabase=admin --oplogReplay --oplogLimit 1484002910:1 ~/oplog
    
  • This should restore your database to the state previous to data loss.

  • Revoke the previous created role from the user root.

     $ mongo admin -u root -p
     replicaset:PRIMARY> db.revokeRolesFromUser("root", [ { role: "executeFunctions", db: "admin" } ])
    

This last step is optional and providing such access is not recommended, but don't revoke it if your organization requires an user to run eval.

Troubleshooting
  • Copy the application folder to have a backup of your upload files. Files that you have been uploaded to your database may not be included in the full backup described above.

  • Check the users in the admin database to make sure it does not exist any unknown user in case your server has been attacked.

     $ mongo admin -u root -p --eval "db.getUsers()"