MongoDB Point-in-Time Backups Made Easy

In this blog post, we’ll look at MongoDB point-in-time backups — and work with them.

Mongodump is the base logical backup tool included with MongoDB. It takes a full BSON copy of database/collections, and optionally includes a log of changes during the backup used to make it consistent to a point in time. Mongorestore is the tool used to restore logical backups created by Mongodump. I’ll use these tools in the steps in this article to restore backed-up data. This article assumes a mongodump-based backup that was taken consistently with oplog changes (by using the command flag “–oplog”), and the backup is being restored to a MongoDB instance.

In this example, a mongodump backup is gathered and restored for the base collection data, and separately the oplogs/changes necessary to restore the data to a particular point-in-time are collected and applied to this data.

Note: Percona developed a backup tool named mongodb_consistent_backup, which is a wrapper for ‘mongodump’ with added cluster-wide backup consistency. The backups created by mongodb_consistent_backup (in Dump/Mongodump mode) can be restored using the same steps as a regular “mongodump” backup.

Stage 1: Get a Mongodump Backup

Mongodump Command Flags

–host/–port (and –user/–password)

This is required, even if you’re using the default host/port (localhost:27017).  If authorization is enabled, add –user/–password flags also.

–oplog 

Required for any replset member! Causes “mongodump” to capture the oplog change log during the backup for consistent to one point in time.

–gzip

Optional. For mongodump >= 3.2, this enables inline compression on the backup files.

$ mongodump --host localhost --port 27017 --oplog --gzip

2016-08-15T12:32:28.930+0200    writing wikipedia.pages to
2016-08-15T12:32:31.932+0200    [#########...............]  wikipedia.pages  674/1700   (39.6%)
2016-08-15T12:32:34.931+0200    [####################....]  wikipedia.pages  1436/1700  (84.5%)
2016-08-15T12:32:37.509+0200    [########################]  wikipedia.pages  2119/1700  (124.6%)
2016-08-15T12:32:37.510+0200    done dumping wikipedia.pages (2119 documents)
2016-08-15T12:32:37.521+0200    writing captured oplog to
2016-08-15T12:32:37.931+0200    [##......................]  .oplog  44/492   (8.9%)
2016-08-15T12:32:39.648+0200    [########################]  .oplog  504/492  (102.4%)
2016-08-15T12:32:39.648+0200    dumped 504 oplog entries

Stage 2: Restore the Backup Data

Stage 3: Get Oplogs for Point-in-Time-Recovery

In this stage, we will gather the changes needed to roll the data forward from the time of backup to the time/oplog-position to which we would like to restore.

In this example below, let’s pretend someone accidentally deleted an entire collection at oplog timestamp: “Timestamp(1470923942, 3)” and we want to fix it. If we decrement the Timestamp increment (2nd number) of “Timestamp(1470923942, 3)” we will have the last change before the accidental command, which in this case is: “Timestamp(1470923942, 2)“. Using the timestamp, we can capture and replay the oplogs from when the backup occurred to just before the issue/error.

A start and end timestamp are required to get the oplog data. In all cases, this will need to be gathered manually, case-by-case.

#!/bin/bash
#
# This tool will dump out a BSON file of MongoDB oplog changes based on a range of Timestamp() objects.
# The captured oplog changes can be applied to a host using 'mongorestore --oplogReplay --dir /path/to/dump'.
set -e
TS_START=$1
TS_END=$2
MONGODUMP_EXTRA=$3
function usage_exit() {
    echo "Usage $0: [Start-BSON-Timestamp] [End-BSON-Timestamp] [Extra-Mongodump-Flags (in quotes for multiple)]"
    exit 1
}
function check_bson_timestamp() {
    local TS=$1
    echo "$TS" | grep -qP "^Timestamp(d+,sd+)$"
    if [ $? -gt 0 ]; then
        echo "ERROR: Both timestamp fields must be in BSON Timestamp format, eg: 'Timestamp(########, #)'!"
        usage_exit
    fi
}
if [ -z "$TS_START" ] || [ -z "$TS_END" ]; then
    usage_exit
else
    check_bson_timestamp "$TS_START"
    check_bson_timestamp "$TS_END"
fi
MONGODUMP_QUERY='{ "ts" : { "$gte" : '$TS_START' }, "ts" : { "$lte" : '$TS_END' } }'
MONGODUMP_FLAGS='--db=local --collection=oplog.rs'
[ ! -z "$MONGODUMP_EXTRA" ] && MONGODUMP_FLAGS="$MONGODUMP_FLAGS $MONGODUMP_EXTRA"
if [ -d dump ]; then
    echo "'dump' subdirectory already exists! Exiting!"
    exit 1
fi
echo "# Dumping oplogs from '$TS_START' to '$TS_END'..."
mkdir dump
mongodump $MONGODUMP_FLAGS --query "$MONGODUMP_QUERY" --out - >dump/oplog.bson
if [ -f dump/oplog.bson ]; then
    echo "# Done!"
else
    echo "ERROR: Cannot find oplog.bson file! Exiting!"
    exit 1
fi

$ ./dump_oplog_range.sh
Usage ./dump_oplog_range.sh: [Start-BSON-Timestamp] [End-BSON-Timestamp] [Extra-Mongodump-Flags (in quotes for multiple)]

Steps

1. Find the PRIMARY member that contains the oplogs needed for the PITR restore.
2. Determine the “end” Timestamp() needed to restore to. This oplog time should be before the problem occurred.
3. Determine the “start” Timestamp() from right before the backup was taken.
1. This timestamp doesn’t need to be exact, so something like a Timestamp() object equal-to “a few min before the backup started” is fine, but the more accurate you are, the fewer changes you’ll need to re-apply (which saves on restore time).
4. Use the MongoToolsAndSnippets script: “get_oplog_range.sh” (above in “Helper Script”) to dump the oplog time-ranges you need to restore to your chosen point-in-time. In this example I am gathering the oplog between two point-in-times (also passing in –username/–password flags in quotes the 3rd parameter):
1. The starting timestamp: the BSON timestamp from before the mongodump backup in “Stage 2: Restore Collection Data” was taken, in this example. “Timestamp(1470923918, 0)” is a time a few seconds before my mongodump was taken (does not need to be exact).
2. The end timestamp: the end BSON Timestamp to restore to, in this example. “Timestamp(1470923942, 2)” is the last oplog-change BEFORE the problem occurred.
   
   
Example:


$ wget -q https://raw.githubusercontent.com/percona/MongoToolsAndSnippets/master/rdba/dump_oplog_range.sh
$ bash ./dump_oplog_range.sh 'Timestamp(1470923918, 0)' 'Timestamp(1470923942, 2)' '--username=secret --password=secret --host=mongo01.example.com --port=27024'
# Dumping oplogs from 'Timestamp(1470923918, 0)' to 'Timestamp(1470923942, 2)'...
2016-08-12T13:11:17.676+0200    writing local.oplog.rs to stdout
2016-08-12T13:11:18.120+0200    dumped 22 documents
# Done!




Note: all additional mongodump flags (optional third field) must be in quotes!


5. Double check that it worked by looking for the ‘oplog.bson‘ file and checking that the file has some data in it (168mb in the below example):

$ ls -alh dump/oplog.bson
-rw-rw-r--. 1 tim tim 168M Aug 12 13:11 dump/oplog.bson

Stage 4: Apply Oplogs for Point-in-Time Recovery (PITR)

In this stage, we apply the time-range-based oplogs gathered in Stage 3 to the restored data set to bring it from the time of the backup to a particular point in time before a problem occurred.

Mongorestore Command Flags

–host/–port (and –user/–password)

Required, even if you’re using the default host/port (localhost:27017).  If authorization is enabled, add –user/–password flags also.

–oplogReplay

Required. This is needed to replay the oplogs in this step.

–dir

Required. The path to the mongodump data.

Steps

1. Copy the “dump” directory containing only the “oplog.bson”. file (captured in Stage 3) to the host that needs the oplog changes applied (the restore host).
2. Run “mongorestore” on the “dump” directory to replay the oplogs into the instance. Make sure the “dump” dir contains only “oplog.bson”!
   

$ mongorestore --host localhost --port 27017 --oplogReplay --dir ./dump
2016-08-12T13:12:28.105+0200    building a list of dbs and collections to restore from dump dir
2016-08-12T13:12:28.106+0200    replaying oplog
2016-08-12T13:12:31.109+0200    oplog   80.0 MB
2016-08-12T13:12:34.109+0200    oplog   143.8 MB
2016-08-12T13:12:35.501+0200    oplog   167.8 MB
2016-08-12T13:12:35.501+0200    done

3. Validate the data was restored with the customer or using any means possible (examples: .count() queries, some random .find() queries, etc.).