google-templatesmongodb

Recover a MongoDB database using the MongoDB Oplog

Understand the MongoDB Oplog

The Oplog (operations log) is a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases. MongoDB applies database operations on the primary and then records the operations on the primary’s oplog. The secondary members then copy and apply these operations in an asynchronous process. All replica set members contain a copy of the oplog, in the local.oplog.rs collection, which allows them to maintain the current state of the database.

The MongoDB Oplog is created after starting a replica set member for the first time with a default size. After that, MongoDB Oplog size will grow up to 5% of free disk space. In the Bitnami Multi-Tier Solution for MongoDB, the replica sets are deployed with 50GB of data disk by default.

For more information, refer to this section of the MongoDB official documentation.

Check MongoDB Oplog status

In order to check if you can use the MongoDB Oplog to recover the database, you can follow the steps below:

  • Create a database backup following this guide.
  • Connect to the primary server via ssh (refer to the FAQ for more information).
  • Connect to the MongoDB database and run the commands below:

    $ mongo admin -u root -p
    replicaset:PRIMARY> use local
    replicaset:PRIMARY> db.oplog.rs.find();
    

    Please note that you get “PRIMARY” when accessing the MongoDB Shell

  • The previous commands should print out the local MongoDB Oplog. If any of those instructions fail, or nothing is displayed, you’ll not be able to use this method to recover the database. If you created a snapshot of your servers you can try to recover the data deploying a new server using that backup.

Recover a database

Once you check the MongoDB Oplog exists and it’s available, you can proceed to dump and restore your database from it. Follow the following steps to do so:

  • Dump the MongoDB Oplog to a folder (e.g. ~/oplog):

    $ mongodump -u root --authenticationDatabase admin -d local -c oplog.rs -o ~/oplog
    
  • Create a copy of the dump file to ensure you can manipulate it safely:

    $ mkdir ~/oplog2 && cp ~/oplog/local/oplog.rs.bson ~/oplog2/oplog.bson
    
  • Use the command bsondump to inspect the .bson files and find the restoring limit:

    $ bsondump ~/oplog2/oplog.bson
    
  • You should obtain an output similar to the one below:

    {"ts":{"$timestamp":{"t":1484002910,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"7714671726931207910"},"v":2,"op":"c","ns":"test.$cmd","o":{"drop":"testData"}}
    
  • Note the “t” and “i” attributes of the restore point (Note that this point won’t be included in the restore), as they are the ones used to restore the log (In this example: “t”:1484002910,“i”:1).

  • You can also use that tool to find offending commands (for example you can look for “drop” commands, but also “remove”, “deleteOne”, and “deleteMany” would be suspicious). If the dump is too big, you can use grep or any other tool to find the commands:

    $ bsondump ~/oplog2/oplog.bson | grep drop
    
  • If running mongorestore with --oplogReplay, the restore role is insufficient to replay the oplog. To replay the oplog, create a user-defined role that has anyAction on anyResource and grant only to users who must run mongorestore with --oplogReplay.

    $ mongo admin -u root -p
    replicaset:PRIMARY> db.createRole( { role: "executeFunctions", privileges: [ { resource: { anyResource: true }, actions: [ "anyAction" ] } ], roles: [] } )
    replicaset:PRIMARY> db.grantRolesToUser("root", [ { role: "executeFunctions", db: "admin" } ])
    
  • Restore your database using that point as the restore limit:

    $ mongorestore -u root --authenticationDatabase=admin --oplogReplay --oplogLimit 1484002910:1 ~/oplog
    
  • This should restore your database to the state previous to data loss.

  • Revoke the previous created role from the user root.

    $ mongo admin -u root -p
    replicaset:PRIMARY> db.revokeRolesFromUser("root", [ { role: "executeFunctions", db: "admin" } ])
    

This last step is optional and providing such access is not recommended, but don’t revoke it if your organization requires an user to run eval.

Troubleshooting

  • Copy the application folder to have a backup of your upload files. Files that you have been uploaded to your database may not be included in the full backup described above.

  • Check the users in the admin database to make sure it does not exist any unknown user in case your server has been attacked.

    $ mongo admin -u root -p --eval "db.getUsers()"
    
Last modification January 18, 2019