From 8491bb9f33a42fc4393bb22191741baa905a3c2a Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Mon, 2 Dec 2013 15:44:02 -0500 Subject: [PATCH 01/10] DOCS-1862: Refactor data recovery tutorial take 2 --- ...oc-administration-backup-and-recovery.yaml | 7 +- ...spec-administration-tutorials-landing.yaml | 2 +- source/tutorial.txt | 6 +- source/tutorial/detect-invalid-data-files.txt | 80 +++++++ source/tutorial/maintain-valid-data-files.txt | 56 +++++ source/tutorial/manage-journaling.txt | 23 +- ...ver-data-following-unexpected-shutdown.txt | 196 ------------------ source/tutorial/recover-data.txt | 130 ++++++++++++ 8 files changed, 285 insertions(+), 215 deletions(-) create mode 100644 source/tutorial/detect-invalid-data-files.txt create mode 100644 source/tutorial/maintain-valid-data-files.txt delete mode 100644 source/tutorial/recover-data-following-unexpected-shutdown.txt create mode 100644 source/tutorial/recover-data.txt diff --git a/source/includes/toc-administration-backup-and-recovery.yaml b/source/includes/toc-administration-backup-and-recovery.yaml index 3fe4f78a95d..3171009be2f 100644 --- a/source/includes/toc-administration-backup-and-recovery.yaml +++ b/source/includes/toc-administration-backup-and-recovery.yaml @@ -20,7 +20,12 @@ description: | Detailed procedures and considerations for backing up sharded clusters and single shards. --- -file: /tutorial/recover-data-following-unexpected-shutdown +file: /tutorial/copy-databases-between-instances +description: | + Copy databases between :program:`mongod` instances or + within a single :program:`mongod` instance or deployment. +--- +file: /tutorial/detect-invalid-data-files description: | Recover data from MongoDB data files that were not properly closed or have an invalid state. diff --git a/source/includes/toc-spec-administration-tutorials-landing.yaml b/source/includes/toc-spec-administration-tutorials-landing.yaml index 5e7fb535b81..a5363186cda 100644 --- a/source/includes/toc-spec-administration-tutorials-landing.yaml +++ b/source/includes/toc-spec-administration-tutorials-landing.yaml @@ -18,7 +18,7 @@ files: level: 2 - file: /administration/backup-sharded-clusters level: 2 - - file: /tutorial/recover-data-following-unexpected-shutdown + - file: /tutorial/detect-invalid-data-files level: 2 - text: "Continue reading from :doc:`/administration/backup` for additional tutorials of MongoDB backup and recovery procedures." level: 2 diff --git a/source/tutorial.txt b/source/tutorial.txt index 8596751c80d..bc76f246629 100644 --- a/source/tutorial.txt +++ b/source/tutorial.txt @@ -57,7 +57,7 @@ Replica Sets - :doc:`/tutorial/configure-replica-set-tag-sets` - :doc:`/tutorial/manage-chained-replication` - :doc:`/tutorial/reconfigure-replica-set-with-unavailable-members` -- :doc:`/tutorial/recover-data-following-unexpected-shutdown` +- :doc:`/tutorial/detect-invalid-data-files` - :doc:`/tutorial/troubleshoot-replica-sets` Sharding @@ -85,7 +85,8 @@ Basic Operations ~~~~~~~~~~~~~~~~ - :doc:`/tutorial/use-database-commands` -- :doc:`/tutorial/recover-data-following-unexpected-shutdown` +- :doc:`/tutorial/detect-invalid-data-files` +- :doc:`/tutorial/copy-databases-between-instances` - :doc:`/tutorial/expire-data` - :doc:`/tutorial/manage-the-database-profiler` - :doc:`/tutorial/rotate-log-files` @@ -103,7 +104,6 @@ Security - :doc:`/tutorial/add-user-administrator` - :doc:`/tutorial/add-user-to-database` - :doc:`/tutorial/define-roles` -- :doc:`/tutorial/change-user-privileges` - :doc:`/tutorial/view-roles` - :doc:`/tutorial/generate-key-file` - :doc:`/tutorial/control-access-to-mongodb-with-kerberos-authentication` diff --git a/source/tutorial/detect-invalid-data-files.txt b/source/tutorial/detect-invalid-data-files.txt new file mode 100644 index 00000000000..c2b6cb61b94 --- /dev/null +++ b/source/tutorial/detect-invalid-data-files.txt @@ -0,0 +1,80 @@ +========================== +Detect Invalid Data Files +========================== + +.. default-domain:: mongodb + +.. contents:: + :backlinks: none + :local: + +Overview +-------- + +Confirm data file validity after any hardware failure, improper +shutdown, or if you suspect that data files have become invalid. Use +one of the procedures described in this tutorial before starting +any recovery procedure. If any procedure detects invalid data files, +follow the instructions in :doc:`/tutorial/recover-data`. + +Considerations +-------------- + +Data recovery on a single unjournaled :program:`mongod` instance +is more difficult than data recovery on a journaled replica set, +and may recover less data. + +To protect data, and to ensure the easiest possible recovery, use +the best practices described in :doc:`maintain-valid-data-files` +for all deployments. + +Procedure +--------- + +Running Without Journal +~~~~~~~~~~~~~~~~~~~~~~~ + +When a :program:`mongod` instance does not run with journaling enabled +and shuts down uncleanly, you must assume the data files are in an +invalid state. + +To confirm that a :program:`mongod` instance shut down uncleanly, look for the +following indicators: + +- a ``mongod.lock`` non-zero-length file in the data directory. + +- the following line in the :program:`mongod` log output: + + .. code-block:: none + + Unclean shutdown detected. + +Running With a Journal +~~~~~~~~~~~~~~~~~~~~~~ + +When :program:`mongod` instance running with journaling enabled shuts down uncleanly, +or if you suspect invalid data files, test the integrity of a single +collection with the :method:`db.collection.validate()` method. + +.. example:: + + Test the integrity of the ``people`` collection using the following + command from the :program:`mongo` shell: + + .. code-block:: javascript + + db.test.validate(true) + + A portion of the output shows that the ``test`` collection is valid: + + .. code-block:: javascript + + { + ... + + "valid" : true, + "errors" : [ ], + "ok" : 1 + } + + diff --git a/source/tutorial/maintain-valid-data-files.txt b/source/tutorial/maintain-valid-data-files.txt new file mode 100644 index 00000000000..e8f813e18c9 --- /dev/null +++ b/source/tutorial/maintain-valid-data-files.txt @@ -0,0 +1,56 @@ +========================= +Maintain Valid Data Files +========================= + +.. default-domain:: mongodb + +.. contents:: + :backlinks: none + :local: + +Overview +-------- + +Follow the recommendations in this tutorial to ensure the best possible +recovery from future hardware failure or improper shutdown. See +also :doc:`/core/backups` and :doc:`/administration/backup` for more +information on preventing data loss. Also see :doc:`/faq/replica-sets` +and :doc:`/tutorial/manage-journaling` for information on how to set +up a robust deployment. + +Recommendations +--------------- + +Use Journaling +~~~~~~~~~~~~~~ + +Always use :ref:`durability journaling `. The journal +stores recent data changes, with the primary aim of recovering from +database invalidity. By default, MongoDB updates its journal ten +times per second. In the worst case, with journaling enabled, only +``1/10`` of a second of data may be lost. + +If a :program:`mongod` instance without journaling shuts down +unexpectedly for *any* reason, always assume that your database is +in an invalid state. + +Run all Deployments as Replica Sets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Certain recovery options are much simpler if the :program:`mongod` +instance runs as a member of a replica set. The primary goal of +replica sets in MongoDB is to prevent data loss and ensure availability. In +the event of database invalidity, recovery may be as simple as syncing +from a fellow replica set member. + +Shut down Cleanly +~~~~~~~~~~~~~~~~~ + +A clean shutdown means that all ongoing MongoDB operations are +complete, and :program:`mongod` has flushed and closed all data files. + +An unclean shutdown can leave the database in an invalid state. + +To ensure a clean shutdown, use one of the shutdown procedures +described in :doc:`/tutorial/manage-mongodb-processes`. + diff --git a/source/tutorial/manage-journaling.txt b/source/tutorial/manage-journaling.txt index b900d291b7b..d24a603578e 100644 --- a/source/tutorial/manage-journaling.txt +++ b/source/tutorial/manage-journaling.txt @@ -10,20 +10,15 @@ and to provide crash resiliency. Before applying a change to the data files, MongoDB writes the change operation to the journal. If MongoDB should terminate or encounter an error before it can write the changes from the journal to the data files, MongoDB can re-apply the write -operation and maintain a consistent state. +operation and maintain a valid state. -*Without* a journal, if :program:`mongod` exits unexpectedly, you must -assume your data is in an inconsistent state, and you must run either -:doc:`repair ` -or, preferably, :doc:`resync ` -from a clean member of the replica set. +If :program:`mongod` exits unexpectedly, you must +assume your data is in an invalid state, and follow the recommendations +in :doc:`/tutorial/detect-invalid-data-files`. -With journaling enabled, if :program:`mongod` stops unexpectedly, -the program can recover everything written to the journal, and the -data remains in a consistent state. By default, the greatest extent of lost -writes, i.e., those not made to the journal, are those made in the last -100 milliseconds. See :setting:`journalCommitInterval` for more -information on the default. +By default, the greatest extent of lost writes, i.e., those not made +to the journal, are those made in the last 100 milliseconds. See +:setting:`journalCommitInterval` for more information on the default. With journaling, if you want a data set to reside entirely in RAM, you need enough RAM to hold the data set plus the "write working set." The @@ -63,10 +58,10 @@ Disable Journaling Do not disable journaling on production systems. If your :program:`mongod` instance stops without shutting down cleanly - unexpectedly for any reason, (e.g. power failure) and you are + for any reason, (e.g. power failure) and you are not running with journaling, then you must recover from an unaffected :term:`replica set` member or backup, as described in - :doc:`repair `. + :doc:`/tutorial/resync-replica-set-member`. To disable journaling, start :program:`mongod` with the :option:`--nojournal ` command line option. diff --git a/source/tutorial/recover-data-following-unexpected-shutdown.txt b/source/tutorial/recover-data-following-unexpected-shutdown.txt deleted file mode 100644 index 1ef48007d2b..00000000000 --- a/source/tutorial/recover-data-following-unexpected-shutdown.txt +++ /dev/null @@ -1,196 +0,0 @@ -========================================= -Recover Data after an Unexpected Shutdown -========================================= - -.. default-domain:: mongodb - -If MongoDB does not shutdown cleanly [#clean-shutdown]_ the on-disk -representation of the data files will likely reflect an inconsistent -state which could lead to data corruption. [#validation]_ - -To prevent data inconsistency and corruption, always shut down the -database cleanly and use the :ref:`durability journaling -`. MongoDB writes data to the journal, by default, -every 100 milliseconds, such that MongoDB can always recover to a -consistent state even in the case of an unclean shutdown due to power -loss or other system failure. - -If you are *not* running as part of a :term:`replica set` **and** do -*not* have journaling enabled, use the following procedure to recover -data that may be in an inconsistent state. If you are running as part -of a replica set, you should *always* restore from a backup or restart -the :program:`mongod` instance with an empty :setting:`dbpath` and -allow MongoDB to perform an initial sync to restore the data. - -.. seealso:: The :doc:`/administration` documents, including - :ref:`Replica Set Syncing `, and the - documentation on the :setting:`repair`, :setting:`repairpath`, and - :setting:`journal` settings. - -.. [#clean-shutdown] To ensure a clean shut down, use the - :method:`db.shutdownServer()` from the :program:`mongo` shell, your - control script, the :option:`mongod --shutdown` option on Linux - systems, "Control-C" when running :program:`mongod` in interactive - mode, or ``kill $(pidof mongod)`` or ``kill -2 $(pidof mongod)``. - -.. [#validation] You can also use the :method:`db.collection.validate()` - method to test the integrity of a single collection. However, this - process is time consuming, and without journaling you can safely - assume that the data is in an invalid state and you should either - run the repair operation or resync from an intact member of the - replica set. - -Process -------- - -Indications -~~~~~~~~~~~ - -When you are aware of a :program:`mongod` instance running without -journaling that stops unexpectedly **and** you're not running with -replication, you should always run the repair operation before -starting MongoDB again. If you're using replication, then restore from -a backup and allow replication to perform an initial :ref:`sync ` to restore data. - -If the ``mongod.lock`` file in the data directory specified by -:setting:`dbpath`, ``/data/db`` by default, is *not* a zero-byte file, -then :program:`mongod` will refuse to start, and you will find a -message that contains the following line in your MongoDB log our -output: - -.. code-block:: none - - Unclean shutdown detected. - -This indicates that you need to run :program:`mongod` with the -:option:`--repair ` option. If you run repair when -the ``mongodb.lock`` file exists in your :setting:`dbpath`, or the -optional :option:`--repairpath `, you will see a -message that contains the following line: - -.. code-block:: none - - old lock file: /data/db/mongod.lock. probably means unclean shutdown - -If you see this message, as a last resort you may remove the lockfile -**and** run the repair operation before starting the database -normally, as in the following procedure: - -Overview -~~~~~~~~ - -.. warning:: Recovering a member of a replica set. - - Do not use this procedure to recover a member of a - :term:`replica set`. Instead you should either restore from - a :doc:`backup ` or perform an initial sync using - data from an intact member of the set, as described in - :doc:`/tutorial/resync-replica-set-member`. - -There are two processes to repair data files that result from an -unexpected shutdown: - -#. Use the :option:`--repair ` option in - conjunction with the :option:`--repairpath ` - option. :program:`mongod` will read the existing data files, and - write the existing data to new data files. This does not modify or - alter the existing data files. - - You do not need to remove the ``mongod.lock`` file before using - this procedure. - -#. Use the :option:`--repair ` option. - :program:`mongod` will read the existing data files, write the - existing data to new files and replace the existing, possibly - corrupt, files with new files. - - You must remove the ``mongod.lock`` file before using this - procedure. - -.. note:: - - :option:`--repair ` functionality is also - available in the shell with the :method:`db.repairDatabase()` - helper for the :dbcommand:`repairDatabase` command. - -.. _tutorial-repair-procedures: - -Procedures -~~~~~~~~~~ - -To repair your data files using the :option:`--repairpath ` -option to preserve the original data files unmodified: - -#. Start :program:`mongod` using :option:`--repair ` - to read the existing data files. - - .. code-block:: sh - - mongod --dbpath /data/db --repair --repairpath /data/db0 - - When this completes, the new repaired data files will be in the - ``/data/db0`` directory. - -#. Start :program:`mongod` using the following invocation to point the - :setting:`dbpath` at ``/data/db0``: - - .. code-block:: sh - - mongod --dbpath /data/db0 - - Once you confirm that the data files are operational you may delete - or archive the data files in the ``/data/db`` directory. - -To repair your data files without preserving the original files, do -not use the :option:`--repairpath ` option, as in -the following procedure: - -#. Remove the stale lock file: - - .. code-block:: sh - - rm /data/db/mongod.lock - - Replace ``/data/db`` with your :setting:`dbpath` where your MongoDB - instance's data files reside. - - .. warning:: - - After you remove the ``mongod.lock`` file you *must* run the - :option:`--repair ` process before using your - database. - -#. Start :program:`mongod` using :option:`--repair ` - to read the existing data files. - - .. code-block:: sh - - mongod --dbpath /data/db --repair - - When this completes, the repaired data files will replace the - original data files in the ``/data/db`` directory. - -#. Start :program:`mongod` using the following invocation to point the - :setting:`dbpath` at ``/data/db``: - - .. code-block:: sh - - mongod --dbpath /data/db - -``mongod.lock`` ---------------- - -In normal operation, you should **never** remove the ``mongod.lock`` -file and start :program:`mongod`. Instead consider the one of the above methods -to recover the database and remove the lock files. In dire -situations you can remove the lockfile, and start the database using the -possibly corrupt files, and attempt to recover data from the database; -however, it's impossible to predict the state of the database in these -situations. - -If you are not running with journaling, and your database shuts down -unexpectedly for *any* reason, you should always proceed *as if* your database -is in an inconsistent and likely corrupt state. If at all possible restore -from :doc:`backup ` or, if running as a :term:`replica -set`, restore by performing an initial sync using data from an intact -member of the set, as described in :doc:`/tutorial/resync-replica-set-member`. diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt new file mode 100644 index 00000000000..993d8a52095 --- /dev/null +++ b/source/tutorial/recover-data.txt @@ -0,0 +1,130 @@ +================== +Recover Data Files +================== + +.. default-domain:: mongodb + +Overview +-------- + +MongoDB provides mechanisms to recover some data from invalid data +files. The most common cause of invalid data files are unclean +shutdowns without :doc:`/core/journaling`. You can inspect data file +validity using :doc:`/tutorial/detect-invalid-data-files`. + +If you detect or suspect invalid data files, you can use this procedure +to recover data files. + +The procedures in this tutorial apply only to :program:`mongod` +instances that are not part of a replica set. To recover data files +on a replica set, :doc:`/tutorial/resync-replica-set-member`. + +Considerations +-------------- + +.. warning:: + + This procedure removes the invalid parts of data files. *You can + lose data as part of the recovery process.* + + Using this procedure on a capped collection truncates the collection + down to two documents. Do not use :program:`mongodump` to recover + capped collections. + +- Use a recovery path with enough space to hold the recovered data files. + +- Do not remove the ``mongod.lock`` file from the data directory. + +- Do not perform this procedure on files that are currently opened + by a running :program:`mongod` instance. The :program:`mongodump` + program refuses to run on files that are already open. + +Procedure +--------- + +This procedure assumes a database with files located +in ``/data/db``. The directory for recovered files is +``/data/recovery``. The name of the database to recover is +``users``. When giving the following commands, replace these with +the relevant directory and database names. + +#. Run :program:`mongodump` once for each database to recover. If the + database used the :option:`--directoryperdb` option, run the + following command from the Linux shell prompt: + + .. code-block:: sh + + mongodump --journal --dbpath /data/db --directoryperdb --repair -d users -o /data/recovery > /data/recovery/users.log + + Otherwise give the command without :option:`--directoryperdb`: + + .. code-block:: sh + + mongodump --journal --dbpath /data/db --repair -d users -o /data/recovery > /data/recovery/users.log + + .. warning:: + + Run :program:`mongodump` on each database to recover. + +#. Verify the recovered files exist in ``/data/recovery``. Examine + ``/data/recovery/users.log`` to determine how many + documents :program:`mongodump` recovered. + +#. Use :program:`mongorestore` on the newly recovered files to create a new + MongoDB node. In this example the new data directory is ``/data/db2``. + + .. code-block:: sh + + mongorestore --dbpath /data/db2 /data/recovery + +#. Test the recovered :program:`mongod` node using your own test + procedures. If :program:`mongodump` successfully recovered the + data files, then *do not proceed with any further recovery efforts*. + +#. If :program:`mongodump` failed to recover the data files, use + :program:`mongod` with the :option:`--repair ` + and :option:`--repairpath ` options to create + a new data directory with a repaired set of data files. + + Specify a new directory to receive the repaired data files: + + .. code-block:: sh + + mongod --dbpath /data/db --repair --repairpath /data/recovery + + When the ``--repair`` operation ends, the new repaired data files + are in the new directory. + + .. warning:: + + :option:`--repair ` removes the invalid parts of + data files. *You can lose data as part of the recovery process.* + + Under some circumstances, :option:`--repair ` + may remove the majority of data in the data file. + + Without the :option:`--repairpath ` + option, the new data files permanently overwrite the old. + +#. Test the data files by bringing up a standalone :program:`mongod` + instance, and have :option:`--dbpath ` point + to the repaired data files in the new directory. If the repair has + removed data, the number of documents in the collection will be + lower than it had been previously. + + From the :program:`mongo` shell, test the number of documents: + + .. code-block:: sh + + db.users.find().count() + +#. You may use the recovered files normally. Start :program:`mongod` + with :setting:`dbpath` pointing to the new directory: + + .. code-block:: sh + + mongod --dbpath + +#. Once you confirm that the data files are correct, delete + or archive the data files in the new directory. + From 0d2339de5c5e0f5da53bcfa05b8a690f48b1d186 Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Wed, 11 Dec 2013 13:29:04 -0500 Subject: [PATCH 02/10] DOCS-1862: Refactor Data Recovery --- source/tutorial/detect-invalid-data-files.txt | 8 ++-- source/tutorial/maintain-valid-data-files.txt | 14 +++--- source/tutorial/manage-journaling.txt | 2 +- source/tutorial/recover-data.txt | 44 ++++++++++--------- 4 files changed, 37 insertions(+), 31 deletions(-) diff --git a/source/tutorial/detect-invalid-data-files.txt b/source/tutorial/detect-invalid-data-files.txt index c2b6cb61b94..eb3476363ac 100644 --- a/source/tutorial/detect-invalid-data-files.txt +++ b/source/tutorial/detect-invalid-data-files.txt @@ -25,13 +25,13 @@ is more difficult than data recovery on a journaled replica set, and may recover less data. To protect data, and to ensure the easiest possible recovery, use -the best practices described in :doc:`maintain-valid-data-files` -for all deployments. +the recommendations from :doc:`maintain-valid-data-files` for all +deployments. Procedure --------- -Running Without Journal +With No Journal Enabled ~~~~~~~~~~~~~~~~~~~~~~~ When a :program:`mongod` instance does not run with journaling enabled @@ -49,7 +49,7 @@ following indicators: Unclean shutdown detected. -Running With a Journal +With a Journal Enabled ~~~~~~~~~~~~~~~~~~~~~~ When :program:`mongod` instance running with journaling enabled shuts down uncleanly, diff --git a/source/tutorial/maintain-valid-data-files.txt b/source/tutorial/maintain-valid-data-files.txt index e8f813e18c9..4922dd208e0 100644 --- a/source/tutorial/maintain-valid-data-files.txt +++ b/source/tutorial/maintain-valid-data-files.txt @@ -11,12 +11,14 @@ Maintain Valid Data Files Overview -------- -Follow the recommendations in this tutorial to ensure the best possible -recovery from future hardware failure or improper shutdown. See -also :doc:`/core/backups` and :doc:`/administration/backup` for more -information on preventing data loss. Also see :doc:`/faq/replica-sets` -and :doc:`/tutorial/manage-journaling` for information on how to set -up a robust deployment. +Use these recommendations to help ensure data file validity, especially +after hardware failure or improper shutdown. + +See also :doc:`/core/backups` and :doc:`/administration/backup` +for more information on preventing data loss. Also see +:doc:`/faq/replica-sets`, :doc:`/core/journaling`, and +:doc:`/tutorial/manage-journaling` for information on how to set up +a robust deployment. Recommendations --------------- diff --git a/source/tutorial/manage-journaling.txt b/source/tutorial/manage-journaling.txt index d24a603578e..afaa78e6cae 100644 --- a/source/tutorial/manage-journaling.txt +++ b/source/tutorial/manage-journaling.txt @@ -12,7 +12,7 @@ should terminate or encounter an error before it can write the changes from the journal to the data files, MongoDB can re-apply the write operation and maintain a valid state. -If :program:`mongod` exits unexpectedly, you must +Without a journal, if :program:`mongod` exits unexpectedly, you must assume your data is in an invalid state, and follow the recommendations in :doc:`/tutorial/detect-invalid-data-files`. diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt index 993d8a52095..5ca690c8984 100644 --- a/source/tutorial/recover-data.txt +++ b/source/tutorial/recover-data.txt @@ -27,9 +27,9 @@ Considerations This procedure removes the invalid parts of data files. *You can lose data as part of the recovery process.* - Using this procedure on a capped collection truncates the collection - down to two documents. Do not use :program:`mongodump` to recover - capped collections. + Using this procedure on a capped collection truncates the + collection to two documents. Do not use :program:`mongodump` with + the :option:`--repair ` to recover capped collections. - Use a recovery path with enough space to hold the recovered data files. @@ -42,21 +42,20 @@ Considerations Procedure --------- -This procedure assumes a database with files located -in ``/data/db``. The directory for recovered files is -``/data/recovery``. The name of the database to recover is -``users``. When giving the following commands, replace these with +This procedure assumes a ``/data/db`` directory that contains data +files for a ``users`` database, and a ``/data/recovery`` directory +for recovered files. Use the following sequence of operations with the relevant directory and database names. #. Run :program:`mongodump` once for each database to recover. If the database used the :option:`--directoryperdb` option, run the - following command from the Linux shell prompt: + following command from the system shell prompt: .. code-block:: sh mongodump --journal --dbpath /data/db --directoryperdb --repair -d users -o /data/recovery > /data/recovery/users.log - Otherwise give the command without :option:`--directoryperdb`: + Otherwise omit the :option:`--directoryperdb` option: .. code-block:: sh @@ -77,9 +76,10 @@ the relevant directory and database names. mongorestore --dbpath /data/db2 /data/recovery -#. Test the recovered :program:`mongod` node using your own test - procedures. If :program:`mongodump` successfully recovered the - data files, then *do not proceed with any further recovery efforts*. +#. Validate the recovered data set using a staging environment or other + application-specific tests. If :program:`mongodump` successfully + recovered the data files, then *do not proceed with any further + recovery efforts*. #. If :program:`mongodump` failed to recover the data files, use :program:`mongod` with the :option:`--repair ` @@ -92,8 +92,8 @@ the relevant directory and database names. mongod --dbpath /data/db --repair --repairpath /data/recovery - When the ``--repair`` operation ends, the new repaired data files - are in the new directory. + When the ``--repair`` operation completes successfully, the new + repaired data files are in the new directory. .. warning:: @@ -106,17 +106,24 @@ the relevant directory and database names. Without the :option:`--repairpath ` option, the new data files permanently overwrite the old. +.. _test-data-file-recovery: + #. Test the data files by bringing up a standalone :program:`mongod` instance, and have :option:`--dbpath ` point to the repaired data files in the new directory. If the repair has removed data, the number of documents in the collection will be lower than it had been previously. - From the :program:`mongo` shell, test the number of documents: + From the :program:`mongo` shell, verify the number of documents + in each collection: - .. code-block:: sh + .. code-block:: javascript - db.users.find().count() + use users + db.collection.find().count() + + Perform other tests as needed. When satisfied that the data files + are correct, delete or archive the ``/data/recovery`` directory. #. You may use the recovered files normally. Start :program:`mongod` with :setting:`dbpath` pointing to the new directory: @@ -125,6 +132,3 @@ the relevant directory and database names. mongod --dbpath -#. Once you confirm that the data files are correct, delete - or archive the data files in the new directory. - From 61c05f07550a8285de50768bee21e62551a8f5fc Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Mon, 16 Dec 2013 11:02:56 -0500 Subject: [PATCH 03/10] DOCS-1862: reorder data test procedure; make overviews friendlier --- source/tutorial/detect-invalid-data-files.txt | 19 +++-- source/tutorial/maintain-valid-data-files.txt | 9 ++- source/tutorial/recover-data.txt | 77 ++++++++++--------- 3 files changed, 60 insertions(+), 45 deletions(-) diff --git a/source/tutorial/detect-invalid-data-files.txt b/source/tutorial/detect-invalid-data-files.txt index eb3476363ac..fbd43595647 100644 --- a/source/tutorial/detect-invalid-data-files.txt +++ b/source/tutorial/detect-invalid-data-files.txt @@ -11,11 +11,11 @@ Detect Invalid Data Files Overview -------- -Confirm data file validity after any hardware failure, improper -shutdown, or if you suspect that data files have become invalid. Use -one of the procedures described in this tutorial before starting -any recovery procedure. If any procedure detects invalid data files, -follow the instructions in :doc:`/tutorial/recover-data`. +The most common causes of invalid data files are malfunctioning +network storage, and improper shutdowns without journaling enabled. If +you suspect that some of your data files might be invalid, use the +procedures described here and in :doc:`/tutorial/recover-data` to +help recover some of your data. Considerations -------------- @@ -24,9 +24,12 @@ Data recovery on a single unjournaled :program:`mongod` instance is more difficult than data recovery on a journaled replica set, and may recover less data. -To protect data, and to ensure the easiest possible recovery, use -the recommendations from :doc:`maintain-valid-data-files` for all -deployments. +The best way to avoid data loss and ensure the most +robust deployments is to follow the recommendations in +:doc:`maintain-valid-data-files`. See also :doc:`/core/backups`, +:doc:`/administration/backup`, :doc:`/faq/replica-sets`, +:doc:`/core/journaling`, and :doc:`/tutorial/manage-journaling` +for information on how to set up and safeguard a robust deployment. Procedure --------- diff --git a/source/tutorial/maintain-valid-data-files.txt b/source/tutorial/maintain-valid-data-files.txt index 4922dd208e0..e7f8d4821d6 100644 --- a/source/tutorial/maintain-valid-data-files.txt +++ b/source/tutorial/maintain-valid-data-files.txt @@ -11,8 +11,13 @@ Maintain Valid Data Files Overview -------- -Use these recommendations to help ensure data file validity, especially -after hardware failure or improper shutdown. +MongoDB supports a variety of ways to avoid data loss. The most +common causes of invalid data files are malfunctioning network storage, +and improper shutdowns without journaling enabled. It's important to protect +your data to enable recovery from any unforseen event. + +The following recommendations help ensure that data is routinely copied +to multiple servers, and that damaged servers may recover quickly. See also :doc:`/core/backups` and :doc:`/administration/backup` for more information on preventing data loss. Also see diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt index 5ca690c8984..8fdfe4bac9b 100644 --- a/source/tutorial/recover-data.txt +++ b/source/tutorial/recover-data.txt @@ -7,17 +7,16 @@ Recover Data Files Overview -------- -MongoDB provides mechanisms to recover some data from invalid data -files. The most common cause of invalid data files are unclean -shutdowns without :doc:`/core/journaling`. You can inspect data file -validity using :doc:`/tutorial/detect-invalid-data-files`. +If you suspect that some of your data files might be invalid, it may +be possible to recover some of the data. -If you detect or suspect invalid data files, you can use this procedure -to recover data files. +In the best-case scenario, your deployment uses :term:`replica sets +`. In that case a complete recovery may be possible, using +the procedures described in :doc:`/tutorial/resync-replica-set-member`. -The procedures in this tutorial apply only to :program:`mongod` -instances that are not part of a replica set. To recover data files -on a replica set, :doc:`/tutorial/resync-replica-set-member`. +If your deployment does not use replica sets, use the following +procedures to remove the invalid portions of your data files, and to +make the remaining portions usable. Considerations -------------- @@ -36,8 +35,15 @@ Considerations - Do not remove the ``mongod.lock`` file from the data directory. - Do not perform this procedure on files that are currently opened - by a running :program:`mongod` instance. The :program:`mongodump` - program refuses to run on files that are already open. + by a running :program:`mongod` instance. This procedure will not + work on files that are already open. + +The best way to avoid data loss and ensure the most +robust deployments is to follow the recommendations in +:doc:`maintain-valid-data-files`. See also :doc:`/core/backups`, +:doc:`/administration/backup`, :doc:`/faq/replica-sets`, +:doc:`/core/journaling`, and :doc:`/tutorial/manage-journaling` +for information on how to set up and safeguard a robust deployment. Procedure --------- @@ -76,10 +82,26 @@ the relevant directory and database names. mongorestore --dbpath /data/db2 /data/recovery -#. Validate the recovered data set using a staging environment or other - application-specific tests. If :program:`mongodump` successfully - recovered the data files, then *do not proceed with any further - recovery efforts*. +.. _test-data-file-recovery: + +#. Test the data files by bringing up a standalone :program:`mongod` + instance, and have :option:`--dbpath ` point + to the repaired data files in the new directory. If the repair has + removed data, the number of documents in the collection will be + lower than it had been previously. + + From the :program:`mongo` shell, verify the number of documents + in each collection: + + .. code-block:: javascript + + use users + db.collection.find().count() + + Perform other application-specific tests in a staging environment + as needed. If the data files are correct, delete or archive the + ``/data/recovery`` directory, and *do not proceed with any further + recovery efforts* #. If :program:`mongodump` failed to recover the data files, use :program:`mongod` with the :option:`--repair ` @@ -92,8 +114,9 @@ the relevant directory and database names. mongod --dbpath /data/db --repair --repairpath /data/recovery - When the ``--repair`` operation completes successfully, the new - repaired data files are in the new directory. + When the :option:`--repair ` operation completes + successfully, the newly-repaired data files are in the new + directory. .. warning:: @@ -106,24 +129,8 @@ the relevant directory and database names. Without the :option:`--repairpath ` option, the new data files permanently overwrite the old. -.. _test-data-file-recovery: - -#. Test the data files by bringing up a standalone :program:`mongod` - instance, and have :option:`--dbpath ` point - to the repaired data files in the new directory. If the repair has - removed data, the number of documents in the collection will be - lower than it had been previously. - - From the :program:`mongo` shell, verify the number of documents - in each collection: - - .. code-block:: javascript - - use users - db.collection.find().count() - - Perform other tests as needed. When satisfied that the data files - are correct, delete or archive the ``/data/recovery`` directory. +#. Test the data files using the procedure :ref:`outlined above + `. #. You may use the recovered files normally. Start :program:`mongod` with :setting:`dbpath` pointing to the new directory: From 6990af3a3ea15578cb7589dc7e12f0b30643d1c5 Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Tue, 17 Dec 2013 14:49:16 -0500 Subject: [PATCH 04/10] DOCS-1862: Continue fixing up the overviews; other fixes --- source/tutorial/detect-invalid-data-files.txt | 50 ++++++++++++------- source/tutorial/maintain-valid-data-files.txt | 27 ++++++---- source/tutorial/recover-data.txt | 33 ++++++++---- 3 files changed, 71 insertions(+), 39 deletions(-) diff --git a/source/tutorial/detect-invalid-data-files.txt b/source/tutorial/detect-invalid-data-files.txt index fbd43595647..1ab46535082 100644 --- a/source/tutorial/detect-invalid-data-files.txt +++ b/source/tutorial/detect-invalid-data-files.txt @@ -11,11 +11,31 @@ Detect Invalid Data Files Overview -------- -The most common causes of invalid data files are malfunctioning -network storage, and improper shutdowns without journaling enabled. If -you suspect that some of your data files might be invalid, use the -procedures described here and in :doc:`/tutorial/recover-data` to -help recover some of your data. +Any deployment may suffer hardware failure, power failure, networking +failure, or some other interruption that may damage data files. MongoDB +provides a range of features, including :term:`replica sets ` and :term:`journaling`, to make recovery from those events +quick and complete. + +If you are *not* running a replica set, it may not be possible to +recover all the data stored in damaged data files. But even in this +case it is possible to remove the damaged portions of your data files, +and make them able to support application queries. + +If you are *not* running a replica set, and suspect that some of your +data files might be invalid, use the procedures described here and +in :doc:`/tutorial/recover-data` to help recover some of your data. + +The best way to avoid data loss and ensure the most +robust deployments is to follow the recommendations in +:doc:`maintain-valid-data-files`. + +See also +:doc:`/core/backups` and +:doc:`/administration/backup` for more information on preventing data loss. Also see +:doc:`/core/replication`, +:doc:`/core/journaling`, and +:doc:`/tutorial/manage-journaling`. Considerations -------------- @@ -24,17 +44,13 @@ Data recovery on a single unjournaled :program:`mongod` instance is more difficult than data recovery on a journaled replica set, and may recover less data. -The best way to avoid data loss and ensure the most -robust deployments is to follow the recommendations in -:doc:`maintain-valid-data-files`. See also :doc:`/core/backups`, -:doc:`/administration/backup`, :doc:`/faq/replica-sets`, -:doc:`/core/journaling`, and :doc:`/tutorial/manage-journaling` -for information on how to set up and safeguard a robust deployment. - Procedure --------- -With No Journal Enabled +Select the procedure that matches the :program:`mongod` configuration +that used the data files you want to recover: + +With no Journal Enabled ~~~~~~~~~~~~~~~~~~~~~~~ When a :program:`mongod` instance does not run with journaling enabled @@ -55,9 +71,10 @@ following indicators: With a Journal Enabled ~~~~~~~~~~~~~~~~~~~~~~ -When :program:`mongod` instance running with journaling enabled shuts down uncleanly, -or if you suspect invalid data files, test the integrity of a single -collection with the :method:`db.collection.validate()` method. +When a :program:`mongod` instance runs with journaling enabled +and shuts down uncleanly, or if you suspect invalid data +files, test the integrity of any single collection with the +:method:`db.collection.validate()` method. .. example:: @@ -80,4 +97,3 @@ collection with the :method:`db.collection.validate()` method. "ok" : 1 } - diff --git a/source/tutorial/maintain-valid-data-files.txt b/source/tutorial/maintain-valid-data-files.txt index e7f8d4821d6..e935204b3d4 100644 --- a/source/tutorial/maintain-valid-data-files.txt +++ b/source/tutorial/maintain-valid-data-files.txt @@ -11,17 +11,22 @@ Maintain Valid Data Files Overview -------- -MongoDB supports a variety of ways to avoid data loss. The most -common causes of invalid data files are malfunctioning network storage, -and improper shutdowns without journaling enabled. It's important to protect -your data to enable recovery from any unforseen event. - -The following recommendations help ensure that data is routinely copied -to multiple servers, and that damaged servers may recover quickly. - -See also :doc:`/core/backups` and :doc:`/administration/backup` -for more information on preventing data loss. Also see -:doc:`/faq/replica-sets`, :doc:`/core/journaling`, and +Any deployment may suffer hardware failure, power failure, networking +failure, or some other interruption that may damage data files. MongoDB +provides a range of features, including :term:`replica sets ` and :term:`journaling`, to make recovery from those events +quick and complete. + +Use the following recommendations to ensure that data is routinely +copied to multiple servers, and that damaged servers may recover +quickly. It's important to protect your data to enable recovery from +any unforseen event. + +See also +:doc:`/core/backups` and +:doc:`/administration/backup` for more information on preventing data loss. Also see +:doc:`/core/replication`, +:doc:`/core/journaling`, and :doc:`/tutorial/manage-journaling` for information on how to set up a robust deployment. diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt index 8fdfe4bac9b..20dbeb30fc5 100644 --- a/source/tutorial/recover-data.txt +++ b/source/tutorial/recover-data.txt @@ -7,17 +7,35 @@ Recover Data Files Overview -------- +Any deployment may suffer hardware failure, power failure, networking +failure, or some other interruption that may damage data files. MongoDB +provides a range of features, including :term:`replica sets ` and :term:`journaling`, to make recovery from those events +quick and complete. + If you suspect that some of your data files might be invalid, it may be possible to recover some of the data. -In the best-case scenario, your deployment uses :term:`replica sets -`. In that case a complete recovery may be possible, using -the procedures described in :doc:`/tutorial/resync-replica-set-member`. +If your deployment *does* use :term:`replica sets `, use +the procedures described in :doc:`/tutorial/resync-replica-set-member` +to resync the invalid data files from valid files on one of the other +members of the replica set. -If your deployment does not use replica sets, use the following +If your deployment does *not* use replica sets, use the following procedures to remove the invalid portions of your data files, and to make the remaining portions usable. +The best way to avoid data loss and ensure the most +robust deployments is to follow the recommendations in +:doc:`maintain-valid-data-files`. + +See also +:doc:`/core/backups` and +:doc:`/administration/backup` for more information on preventing data loss. Also see +:doc:`/core/replication`, +:doc:`/core/journaling`, and +:doc:`/tutorial/manage-journaling`. + Considerations -------------- @@ -38,13 +56,6 @@ Considerations by a running :program:`mongod` instance. This procedure will not work on files that are already open. -The best way to avoid data loss and ensure the most -robust deployments is to follow the recommendations in -:doc:`maintain-valid-data-files`. See also :doc:`/core/backups`, -:doc:`/administration/backup`, :doc:`/faq/replica-sets`, -:doc:`/core/journaling`, and :doc:`/tutorial/manage-journaling` -for information on how to set up and safeguard a robust deployment. - Procedure --------- From c6cfc54346d187ccaaf7a0f6d462842d13845579 Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Mon, 23 Dec 2013 15:30:50 -0500 Subject: [PATCH 05/10] DOCS-1862: Added validate() output of invalid collection --- source/tutorial/detect-invalid-data-files.txt | 40 +++++++++++++------ 1 file changed, 27 insertions(+), 13 deletions(-) diff --git a/source/tutorial/detect-invalid-data-files.txt b/source/tutorial/detect-invalid-data-files.txt index 1ab46535082..efaef381753 100644 --- a/source/tutorial/detect-invalid-data-files.txt +++ b/source/tutorial/detect-invalid-data-files.txt @@ -76,24 +76,38 @@ and shuts down uncleanly, or if you suspect invalid data files, test the integrity of any single collection with the :method:`db.collection.validate()` method. -.. example:: +Test the integrity of the ``people`` collection using the following +command from the :program:`mongo` shell: - Test the integrity of the ``people`` collection using the following - command from the :program:`mongo` shell: +.. code-block:: javascript - .. code-block:: javascript + db.test.validate(true) - db.test.validate(true) +A portion of the output shows that the ``test`` collection is valid: - A portion of the output shows that the ``test`` collection is valid: +.. code-block:: javascript - .. code-block:: javascript + { + ... - { - ... + "valid" : true, + "errors" : [ ], + "ok" : 1 + } - "valid" : true, - "errors" : [ ], - "ok" : 1 - } +If the collection is invalid, the output of +:method:`db.collection.validate()` shows that as well: + +.. code-block:: javascript + + { + ... + "valid" : false, + "errors" : [ + "invalid bson object detected (see logs for more info)", + "exception during validate" + ], + "advice" : "ns corrupt, requires repair", + "ok" : 1 + } From fa42e4d35420bb5c65b08081929cf03c2851bd3a Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Thu, 26 Dec 2013 10:29:38 -0500 Subject: [PATCH 06/10] DOCS-1862: Fixed code snippet --- source/tutorial/recover-data.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt index 20dbeb30fc5..d3b62135dc6 100644 --- a/source/tutorial/recover-data.txt +++ b/source/tutorial/recover-data.txt @@ -107,7 +107,7 @@ the relevant directory and database names. .. code-block:: javascript use users - db.collection.find().count() + db.collection.count() Perform other application-specific tests in a staging environment as needed. If the data files are correct, delete or archive the From e9c7651b5539e035a2c64c2c685fd4bec0d9fa57 Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Thu, 2 Jan 2014 11:08:23 -0500 Subject: [PATCH 07/10] DOCS-1862: convert steps to new format --- source/includes/steps-recover-data-files.yaml | 98 +++++++++++++++++++ source/tutorial/recover-data.txt | 86 +--------------- 2 files changed, 99 insertions(+), 85 deletions(-) create mode 100644 source/includes/steps-recover-data-files.yaml diff --git a/source/includes/steps-recover-data-files.yaml b/source/includes/steps-recover-data-files.yaml new file mode 100644 index 00000000000..6e97e0d90eb --- /dev/null +++ b/source/includes/steps-recover-data-files.yaml @@ -0,0 +1,98 @@ +ref: recover-data-files-run-mongodump-once-per-db +stepnum: 1 +title: Run ``mongodump`` once for each database to recover +action: + - pre: | + If the database used the :option:`--directoryperdb` option, run the + following command from the system shell prompt: + language: sh + code: | + mongodump --journal --dbpath /data/db --directoryperdb --repair -d users -o /data/recovery > /data/recovery/users.log + - pre: | + Otherwise omit the :option:`--directoryperdb` option: + language: sh + code: | + mongodump --journal --dbpath /data/db --repair -d users -o /data/recovery > /data/recovery/users.log +--- +ref: recover-data-files-verify-recovered-files-exist +stepnum: 2 +title: Verify the new files contain recovered documents +pre: | + Examine ``/data/recovery/users.log`` to determine how many documents + :program:`mongodump` recovered. +--- +ref: recover-data-files-create-new-mongodb-node +stepnum: 3 +title: Create MongoDB node with ``mongorestore`` +pre: | + In this example the new data directory is ``/data/db2``. +action: + language: sh + code: | + mongorestore --dbpath /data/db2 /data/recovery +--- +ref: recover-data-files-test-data-files +stepnum: 4 +title: Test the data files on a standalone ``mongod`` +action: + - pre: + language: sh + code: | + mongod --dbpath /data/recovery/ + - pre: | + If the repair has removed data, the number of documents in the + collection will be lower than it had been previously. From the + :program:`mongo` shell, verify the number of documents in each collection: + language: javascript + code: | + use users + db.collection.count() +post: | + Perform other application-specific tests in a staging environment as + needed. If the data files are correct, delete or archive the + ``/data/recovery`` directory, and *do not proceed with any further + recovery efforts*. +--- +ref: recover-data-files-use-repair-option-and-repairpath +stepnum: 5 +title: Use ``--repair`` and ``--repairpath`` +pre: | + If :program:`mongodump` failed to recover the data files, use + :program:`mongod` with the :option:`--repair ` and + :option:`--repairpath ` options to create a new + data directory with a repaired set of data files. Specify a new + directory to receive the repaired data files: +action: + language: sh + code: | + mongod --dbpath /data/db --repair --repairpath /data/recovery +post: | + When the :option:`--repair ` operation completes + successfully, the newly-repaired data files are in the new directory. + + .. warning:: + + :option:`--repair ` removes the invalid parts of + data files. *You can lose data as part of the recovery process.* + Under some circumstances, :option:`--repair ` + may remove the majority of data in the data file. Without the + :option:`--repairpath ` option, the new + data files permanently overwrite the old. +--- +ref: recover-data-files-test-data-files +stepnum: 6 +title: Test the data files +pre: | + Test the data files using the procedure outlined above. +--- +ref: recover-data-files-use-files-normally +stepnum: 7 +title: Use the recovered files normally +pre: | + Start :program:`mongod` with :setting:`dbpath` pointing to the new directory: +action: + language: sh + code: | + mongod --dbpath +... + diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt index d3b62135dc6..c12fc0699dd 100644 --- a/source/tutorial/recover-data.txt +++ b/source/tutorial/recover-data.txt @@ -64,89 +64,5 @@ files for a ``users`` database, and a ``/data/recovery`` directory for recovered files. Use the following sequence of operations with the relevant directory and database names. -#. Run :program:`mongodump` once for each database to recover. If the - database used the :option:`--directoryperdb` option, run the - following command from the system shell prompt: - - .. code-block:: sh - - mongodump --journal --dbpath /data/db --directoryperdb --repair -d users -o /data/recovery > /data/recovery/users.log - - Otherwise omit the :option:`--directoryperdb` option: - - .. code-block:: sh - - mongodump --journal --dbpath /data/db --repair -d users -o /data/recovery > /data/recovery/users.log - - .. warning:: - - Run :program:`mongodump` on each database to recover. - -#. Verify the recovered files exist in ``/data/recovery``. Examine - ``/data/recovery/users.log`` to determine how many - documents :program:`mongodump` recovered. - -#. Use :program:`mongorestore` on the newly recovered files to create a new - MongoDB node. In this example the new data directory is ``/data/db2``. - - .. code-block:: sh - - mongorestore --dbpath /data/db2 /data/recovery - -.. _test-data-file-recovery: - -#. Test the data files by bringing up a standalone :program:`mongod` - instance, and have :option:`--dbpath ` point - to the repaired data files in the new directory. If the repair has - removed data, the number of documents in the collection will be - lower than it had been previously. - - From the :program:`mongo` shell, verify the number of documents - in each collection: - - .. code-block:: javascript - - use users - db.collection.count() - - Perform other application-specific tests in a staging environment - as needed. If the data files are correct, delete or archive the - ``/data/recovery`` directory, and *do not proceed with any further - recovery efforts* - -#. If :program:`mongodump` failed to recover the data files, use - :program:`mongod` with the :option:`--repair ` - and :option:`--repairpath ` options to create - a new data directory with a repaired set of data files. - - Specify a new directory to receive the repaired data files: - - .. code-block:: sh - - mongod --dbpath /data/db --repair --repairpath /data/recovery - - When the :option:`--repair ` operation completes - successfully, the newly-repaired data files are in the new - directory. - - .. warning:: - - :option:`--repair ` removes the invalid parts of - data files. *You can lose data as part of the recovery process.* - - Under some circumstances, :option:`--repair ` - may remove the majority of data in the data file. - - Without the :option:`--repairpath ` - option, the new data files permanently overwrite the old. - -#. Test the data files using the procedure :ref:`outlined above - `. - -#. You may use the recovered files normally. Start :program:`mongod` - with :setting:`dbpath` pointing to the new directory: - - .. code-block:: sh - - mongod --dbpath +.. include:: /includes/steps/recover-data-files.rst From 013cdbfe1108fd44cbddab411a057386d54361ef Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Thu, 2 Jan 2014 11:29:26 -0500 Subject: [PATCH 08/10] DOCS-1862: fix journaling link --- source/tutorial/maintain-valid-data-files.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorial/maintain-valid-data-files.txt b/source/tutorial/maintain-valid-data-files.txt index e935204b3d4..35cbd0340ef 100644 --- a/source/tutorial/maintain-valid-data-files.txt +++ b/source/tutorial/maintain-valid-data-files.txt @@ -14,7 +14,7 @@ Overview Any deployment may suffer hardware failure, power failure, networking failure, or some other interruption that may damage data files. MongoDB provides a range of features, including :term:`replica sets ` and :term:`journaling`, to make recovery from those events +set>` and :term:`journaling `, to make recovery from those events quick and complete. Use the following recommendations to ensure that data is routinely From 273eb4666b1c6bde2ea53cb57439515a071379a8 Mon Sep 17 00:00:00 2001 From: Zack Brown Date: Thu, 2 Jan 2014 11:34:09 -0500 Subject: [PATCH 09/10] DOCS-1862: fix journaling link --- source/tutorial/detect-invalid-data-files.txt | 2 +- source/tutorial/recover-data.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/source/tutorial/detect-invalid-data-files.txt b/source/tutorial/detect-invalid-data-files.txt index efaef381753..958ce3ba784 100644 --- a/source/tutorial/detect-invalid-data-files.txt +++ b/source/tutorial/detect-invalid-data-files.txt @@ -14,7 +14,7 @@ Overview Any deployment may suffer hardware failure, power failure, networking failure, or some other interruption that may damage data files. MongoDB provides a range of features, including :term:`replica sets ` and :term:`journaling`, to make recovery from those events +set>` and :term:`journaling `, to make recovery from those events quick and complete. If you are *not* running a replica set, it may not be possible to diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt index c12fc0699dd..ec4e5716440 100644 --- a/source/tutorial/recover-data.txt +++ b/source/tutorial/recover-data.txt @@ -10,7 +10,7 @@ Overview Any deployment may suffer hardware failure, power failure, networking failure, or some other interruption that may damage data files. MongoDB provides a range of features, including :term:`replica sets ` and :term:`journaling`, to make recovery from those events +set>` and :term:`journaling `, to make recovery from those events quick and complete. If you suspect that some of your data files might be invalid, it may From 0a1f9f4b15e469224d3e0dc226a83f1645d46017 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Mon, 3 Mar 2014 16:01:13 -0500 Subject: [PATCH 10/10] DOCS-1862 data recovery --- source/includes/steps-recover-data-files.yaml | 45 +++++----- ...oc-administration-backup-and-recovery.yaml | 10 ++- source/tutorial/detect-invalid-data-files.txt | 89 ++++++++----------- source/tutorial/maintain-valid-data-files.txt | 66 +++++++------- source/tutorial/manage-journaling.txt | 66 +++++++------- source/tutorial/recover-data.txt | 75 ++++++++-------- 6 files changed, 170 insertions(+), 181 deletions(-) diff --git a/source/includes/steps-recover-data-files.yaml b/source/includes/steps-recover-data-files.yaml index 6e97e0d90eb..a4b2772a845 100644 --- a/source/includes/steps-recover-data-files.yaml +++ b/source/includes/steps-recover-data-files.yaml @@ -1,6 +1,6 @@ -ref: recover-data-files-run-mongodump-once-per-db +ref: run-mongodump stepnum: 1 -title: Run ``mongodump`` once for each database to recover +title: Run :program:`mongodump` for each database to recover. action: - pre: | If the database used the :option:`--directoryperdb` option, run the @@ -14,34 +14,38 @@ action: code: | mongodump --journal --dbpath /data/db --repair -d users -o /data/recovery > /data/recovery/users.log --- -ref: recover-data-files-verify-recovered-files-exist +ref: verify stepnum: 2 -title: Verify the new files contain recovered documents +title: Verify the new files contain recovered documents. pre: | Examine ``/data/recovery/users.log`` to determine how many documents :program:`mongodump` recovered. --- -ref: recover-data-files-create-new-mongodb-node +ref: create-new-mongodb-node stepnum: 3 -title: Create MongoDB node with ``mongorestore`` +title: Create a new MongoDB node. pre: | - In this example the new data directory is ``/data/db2``. + Use :program:`mongorestore` to create a new data directory. For + example, the following command restores data to the new data directory + ``/data/db2``. action: language: sh code: | mongorestore --dbpath /data/db2 /data/recovery --- -ref: recover-data-files-test-data-files +ref: test stepnum: 4 -title: Test the data files on a standalone ``mongod`` +title: Test the data files on a standalone ``mongod``. action: - - pre: + - pre: | + Start the :program:`mongod` with with a ``dbpath`` pointing to the + recovered data. For example: language: sh code: | mongod --dbpath /data/recovery/ - pre: | If the repair has removed data, the number of documents in the - collection will be lower than it had been previously. From the + collection will be lower than it had been previously. From the :program:`mongo` shell, verify the number of documents in each collection: language: javascript code: | @@ -50,17 +54,17 @@ action: post: | Perform other application-specific tests in a staging environment as needed. If the data files are correct, delete or archive the - ``/data/recovery`` directory, and *do not proceed with any further + ``/data/recovery`` directory and *do not proceed with any further recovery efforts*. --- -ref: recover-data-files-use-repair-option-and-repairpath +ref: repair stepnum: 5 -title: Use ``--repair`` and ``--repairpath`` +title: Repair the data. pre: | If :program:`mongodump` failed to recover the data files, use :program:`mongod` with the :option:`--repair ` and :option:`--repairpath ` options to create a new - data directory with a repaired set of data files. Specify a new + data directory with a repaired set of data files. Specify a new directory to receive the repaired data files: action: language: sh @@ -73,21 +77,21 @@ post: | .. warning:: :option:`--repair ` removes the invalid parts of - data files. *You can lose data as part of the recovery process.* + data files. You can lose data as part of the recovery process. Under some circumstances, :option:`--repair ` may remove the majority of data in the data file. Without the :option:`--repairpath ` option, the new data files permanently overwrite the old. --- -ref: recover-data-files-test-data-files +ref: test-data-files stepnum: 6 -title: Test the data files +title: Test the data files. pre: | Test the data files using the procedure outlined above. --- -ref: recover-data-files-use-files-normally +ref: use-files-normally stepnum: 7 -title: Use the recovered files normally +title: Use the recovered files normally. pre: | Start :program:`mongod` with :setting:`dbpath` pointing to the new directory: action: @@ -95,4 +99,3 @@ action: code: | mongod --dbpath ... - diff --git a/source/includes/toc-administration-backup-and-recovery.yaml b/source/includes/toc-administration-backup-and-recovery.yaml index 3171009be2f..b5d40b790a1 100644 --- a/source/includes/toc-administration-backup-and-recovery.yaml +++ b/source/includes/toc-administration-backup-and-recovery.yaml @@ -20,12 +20,16 @@ description: | Detailed procedures and considerations for backing up sharded clusters and single shards. --- -file: /tutorial/copy-databases-between-instances +file: /tutorial/maintain-valid-data-files description: | - Copy databases between :program:`mongod` instances or - within a single :program:`mongod` instance or deployment. + Ensure valid data through journaling and replica sets. --- file: /tutorial/detect-invalid-data-files +description: | + Recover data from MongoDB data files that were not properly closed + or have an invalid state. +--- +file: /tutorial/recover-data description: | Recover data from MongoDB data files that were not properly closed or have an invalid state. diff --git a/source/tutorial/detect-invalid-data-files.txt b/source/tutorial/detect-invalid-data-files.txt index 958ce3ba784..a4dcd45d603 100644 --- a/source/tutorial/detect-invalid-data-files.txt +++ b/source/tutorial/detect-invalid-data-files.txt @@ -11,54 +11,26 @@ Detect Invalid Data Files Overview -------- -Any deployment may suffer hardware failure, power failure, networking -failure, or some other interruption that may damage data files. MongoDB -provides a range of features, including :term:`replica sets ` and :term:`journaling `, to make recovery from those events -quick and complete. +If you have an interruption, such as a power failure, you must assume your +data files are in an invalid state. This section describes how to check +whether a :program:`mongod` instance shut down cleanly and how to test +data integrity. -If you are *not* running a replica set, it may not be possible to -recover all the data stored in damaged data files. But even in this -case it is possible to remove the damaged portions of your data files, -and make them able to support application queries. +To recover data that has been corrupted, see +:doc:`/tutorial/recover-data`. -If you are *not* running a replica set, and suspect that some of your -data files might be invalid, use the procedures described here and -in :doc:`/tutorial/recover-data` to help recover some of your data. +.. note:: The best way to avoid data loss and ensure the most robust + deployments is to follow the recommendations in + :doc:`maintain-valid-data-files`. -The best way to avoid data loss and ensure the most -robust deployments is to follow the recommendations in -:doc:`maintain-valid-data-files`. +Procedures +---------- -See also -:doc:`/core/backups` and -:doc:`/administration/backup` for more information on preventing data loss. Also see -:doc:`/core/replication`, -:doc:`/core/journaling`, and -:doc:`/tutorial/manage-journaling`. +Detect an Unclean Shutdown +~~~~~~~~~~~~~~~~~~~~~~~~~~ -Considerations --------------- - -Data recovery on a single unjournaled :program:`mongod` instance -is more difficult than data recovery on a journaled replica set, -and may recover less data. - -Procedure ---------- - -Select the procedure that matches the :program:`mongod` configuration -that used the data files you want to recover: - -With no Journal Enabled -~~~~~~~~~~~~~~~~~~~~~~~ - -When a :program:`mongod` instance does not run with journaling enabled -and shuts down uncleanly, you must assume the data files are in an -invalid state. - -To confirm that a :program:`mongod` instance shut down uncleanly, look for the -following indicators: +To detect whether a :program:`mongod` instance shutdown cleanly, look for +the following indicators: - a ``mongod.lock`` non-zero-length file in the data directory. @@ -68,16 +40,19 @@ following indicators: Unclean shutdown detected. -With a Journal Enabled -~~~~~~~~~~~~~~~~~~~~~~ +These indicate an unclean shutdown, in which case you must assume data +files are invalid. -When a :program:`mongod` instance runs with journaling enabled -and shuts down uncleanly, or if you suspect invalid data -files, test the integrity of any single collection with the -:method:`db.collection.validate()` method. +Test Data Integrity +~~~~~~~~~~~~~~~~~~~ -Test the integrity of the ``people`` collection using the following -command from the :program:`mongo` shell: +This procedure applies only if the :program:`mongod` instance runs with +:term:`journaling ` enabled. To test data integrity, use either +the :method:`db.collection.validate()` method or :dbcommand:`validate` +command. + +For example, to test the integrity of the ``people`` collection, use the +following command from the :program:`mongo` shell: .. code-block:: javascript @@ -111,3 +86,15 @@ If the collection is invalid, the output of "ok" : 1 } +Related Documents +----------------- + +- :doc:`maintain-valid-data-files` + +- :doc:`/tutorial/recover-data` + +- :doc:`/tutorial/manage-journaling` + +- :doc:`/core/backups` + +- :doc:`/administration/backup` diff --git a/source/tutorial/maintain-valid-data-files.txt b/source/tutorial/maintain-valid-data-files.txt index 35cbd0340ef..afa9c8b92d1 100644 --- a/source/tutorial/maintain-valid-data-files.txt +++ b/source/tutorial/maintain-valid-data-files.txt @@ -11,58 +11,56 @@ Maintain Valid Data Files Overview -------- -Any deployment may suffer hardware failure, power failure, networking -failure, or some other interruption that may damage data files. MongoDB -provides a range of features, including :term:`replica sets ` and :term:`journaling `, to make recovery from those events -quick and complete. - -Use the following recommendations to ensure that data is routinely -copied to multiple servers, and that damaged servers may recover -quickly. It's important to protect your data to enable recovery from -any unforseen event. - -See also -:doc:`/core/backups` and -:doc:`/administration/backup` for more information on preventing data loss. Also see -:doc:`/core/replication`, -:doc:`/core/journaling`, and -:doc:`/tutorial/manage-journaling` for information on how to set up -a robust deployment. - -Recommendations ---------------- - -Use Journaling -~~~~~~~~~~~~~~ +MongoDB provides features to protect your data in the event of hardware +failure, power failure, network failure, or other unforeseen events that +affect data. Use the features described here to ensure that data is +routinely copied to multiple servers and that damaged servers recover +quickly. + +Enable Journaling +----------------- Always use :ref:`durability journaling `. The journal stores recent data changes, with the primary aim of recovering from database invalidity. By default, MongoDB updates its journal ten -times per second. In the worst case, with journaling enabled, only +times per second. In the worst case, with journaling enabled, ``1/10`` of a second of data may be lost. If a :program:`mongod` instance without journaling shuts down unexpectedly for *any* reason, always assume that your database is in an invalid state. -Run all Deployments as Replica Sets -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +For 64-bit builds of :program:`mongod`, MongoDB enables journaling by +default. For more information see :doc:`/core/journaling` and +:doc:`/tutorial/manage-journaling`. + +Run All Deployments as Replica Sets +----------------------------------- Certain recovery options are much simpler if the :program:`mongod` instance runs as a member of a replica set. The primary goal of -replica sets in MongoDB is to prevent data loss and ensure availability. In +replica sets in MongoDB is to ensure availability and prevent data loss. In the event of database invalidity, recovery may be as simple as syncing from a fellow replica set member. -Shut down Cleanly -~~~~~~~~~~~~~~~~~ +For more information see :doc:`/core/replication`. -A clean shutdown means that all ongoing MongoDB operations are -complete, and :program:`mongod` has flushed and closed all data files. - -An unclean shutdown can leave the database in an invalid state. +Shut Down Cleanly +----------------- +A clean shutdown means that all ongoing MongoDB operations are +complete and :program:`mongod` has flushed and closed all data files. +An unclean shutdown, however, can leave the database in an invalid state. To ensure a clean shutdown, use one of the shutdown procedures described in :doc:`/tutorial/manage-mongodb-processes`. +Related Documents +----------------- + +- :doc:`detect-invalid-data-files` + +- :doc:`/tutorial/recover-data` + +- :doc:`/core/backups` + +- :doc:`/administration/backup` diff --git a/source/tutorial/manage-journaling.txt b/source/tutorial/manage-journaling.txt index afaa78e6cae..dfc7ce4b000 100644 --- a/source/tutorial/manage-journaling.txt +++ b/source/tutorial/manage-journaling.txt @@ -4,33 +4,36 @@ Manage Journaling .. default-domain:: mongodb +Overview +-------- + MongoDB uses *write ahead logging* to an on-disk :term:`journal` to guarantee :doc:`write operation ` durability and to provide crash resiliency. Before applying a change to the data -files, MongoDB writes the change operation to the journal. If MongoDB -should terminate or encounter an error before it can write the changes -from the journal to the data files, MongoDB can re-apply the write -operation and maintain a valid state. +files, MongoDB writes the change operation to the journal. If a +:program:`mongod` should terminate or encounter an error before it can +write the changes from the journal to the data files, MongoDB can re-apply +the write operation and maintain a valid state. -Without a journal, if :program:`mongod` exits unexpectedly, you must -assume your data is in an invalid state, and follow the recommendations +Without a journal, if a :program:`mongod` exits unexpectedly, you must +assume your data is in an invalid state and follow a detection procedure in :doc:`/tutorial/detect-invalid-data-files`. By default, the greatest extent of lost writes, i.e., those not made to the journal, are those made in the last 100 milliseconds. See :setting:`journalCommitInterval` for more information on the default. +Considerations +-------------- + With journaling, if you want a data set to reside entirely in RAM, you -need enough RAM to hold the data set plus the "write working set." The -"write working set" is the amount of unique data you expect to see +must have enough RAM to hold the data set plus the *write working set*, +which is the amount of unique data you expect to see written between re-mappings of the private view. For information on views, see :ref:`journaling-storage-views`. -.. important:: - - .. versionchanged:: 2.0 - For 64-bit builds of :program:`mongod`, journaling is enabled by - default. For other platforms, see :setting:`journal`. +For 64-bit builds of :program:`mongod`, journaling is enabled by default. +For other platforms, see :setting:`journal`. Procedures ---------- @@ -38,9 +41,6 @@ Procedures Enable Journaling ~~~~~~~~~~~~~~~~~ -.. versionchanged:: 2.0 - For 64-bit builds of :program:`mongod`, journaling is enabled by default. - To enable journaling, start :program:`mongod` with the :option:`--journal ` command line option. @@ -56,20 +56,21 @@ Disable Journaling .. warning:: - Do not disable journaling on production systems. If your - :program:`mongod` instance stops without shutting down cleanly - for any reason, (e.g. power failure) and you are - not running with journaling, then you must recover from an - unaffected :term:`replica set` member or backup, as described in - :doc:`/tutorial/resync-replica-set-member`. + Do not disable journaling on production systems. + +If you disable journaling and your :program:`mongod` instance stops +without shutting down cleanly, for example from a power failure, then you +must recover from an unaffected :term:`replica set` member or backup, as +described in :doc:`/tutorial/resync-replica-set-member`. -To disable journaling, start :program:`mongod` with the -:option:`--nojournal ` command line option. +Disable journaling only on non-production systems. To do so, start +:program:`mongod` with the :option:`--nojournal ` +command line option. Get Commit Acknowledgment ~~~~~~~~~~~~~~~~~~~~~~~~~ -You can get commit acknowledgment with the +To get commit acknowledgment, use the :dbcommand:`getLastError` command and the ``j`` option. For details, see :ref:`write-concern-operation`. @@ -78,18 +79,17 @@ You can get commit acknowledgment with the Avoid Preallocation Lag ~~~~~~~~~~~~~~~~~~~~~~~ -To avoid :ref:`preallocation lag `, you can +To avoid :ref:`preallocation lag `, preallocate files in the journal directory by copying them from another instance of :program:`mongod`. -Preallocated files do not contain data. It is safe to later remove them. -But if you restart :program:`mongod` with journaling, :program:`mongod` -will create them again. +Preallocated files do not contain data, and it is safe to later remove them. +However, if you restart :program:`mongod` with journaling, MongoDB +will again create them. .. example:: The following sequence preallocates journal files for an instance of :program:`mongod` running on port ``27017`` with a database path of ``/data/db``. - For demonstration purposes, the sequence starts by creating a set of journal files in the usual way. @@ -100,14 +100,14 @@ will create them again. mkdir ~/tmpDbpath - #. Create a set of journal files by staring a :program:`mongod` + #. Create a set of journal files by starting a :program:`mongod` instance that uses the temporary directory: .. code-block:: sh mongod --port 10000 --dbpath ~/tmpDbpath --journal - #. When you see the following log output, indicating + #. When you see the following log output, which indicates :program:`mongod` has the files, press CONTROL+C to stop the :program:`mongod` instance: @@ -162,7 +162,7 @@ Change the Group Commit Interval .. versionchanged:: 2.0 -You can set the group commit interval using the +To set the group commit interval, use the :option:`--journalCommitInterval ` command line option. The allowed range is ``2`` to ``300`` milliseconds. diff --git a/source/tutorial/recover-data.txt b/source/tutorial/recover-data.txt index ec4e5716440..d834e8297f8 100644 --- a/source/tutorial/recover-data.txt +++ b/source/tutorial/recover-data.txt @@ -7,62 +7,59 @@ Recover Data Files Overview -------- -Any deployment may suffer hardware failure, power failure, networking -failure, or some other interruption that may damage data files. MongoDB -provides a range of features, including :term:`replica sets ` and :term:`journaling `, to make recovery from those events -quick and complete. - -If you suspect that some of your data files might be invalid, it may -be possible to recover some of the data. - -If your deployment *does* use :term:`replica sets `, use -the procedures described in :doc:`/tutorial/resync-replica-set-member` -to resync the invalid data files from valid files on one of the other -members of the replica set. +If you suspect that some of your data files are invalid and if your +deployment uses :term:`replica sets `, use the procedures in +:doc:`/tutorial/resync-replica-set-member` to resync the invalid data +files from valid files on one of the other members of the replica set. If your deployment does *not* use replica sets, use the following -procedures to remove the invalid portions of your data files, and to -make the remaining portions usable. +procedure to remove the invalid portions of your data files and to make +the remaining portions usable. -The best way to avoid data loss and ensure the most -robust deployments is to follow the recommendations in -:doc:`maintain-valid-data-files`. - -See also -:doc:`/core/backups` and -:doc:`/administration/backup` for more information on preventing data loss. Also see -:doc:`/core/replication`, -:doc:`/core/journaling`, and -:doc:`/tutorial/manage-journaling`. +The best way to avoid data loss and ensure the most robust deployments is +to follow the recommendations in :doc:`maintain-valid-data-files`. Considerations -------------- .. warning:: - This procedure removes the invalid parts of data files. *You can - lose data as part of the recovery process.* + You can lose data as part of the recovery process. - Using this procedure on a capped collection truncates the - collection to two documents. Do not use :program:`mongodump` with - the :option:`--repair ` to recover capped collections. +This procedure removes the invalid parts of data files. On a capped +collection, this procedure truncates the collection to two documents. Do +not use :program:`mongodump` with the :option:`--repair ` to +recover capped collections. -- Use a recovery path with enough space to hold the recovered data files. +Do not perform this procedure on files that are currently opened by a +running :program:`mongod` instance. This procedure will not work on files +that are already open. -- Do not remove the ``mongod.lock`` file from the data directory. +Prerequisite +------------ -- Do not perform this procedure on files that are currently opened - by a running :program:`mongod` instance. This procedure will not - work on files that are already open. +Use a recovery path with enough space to hold the recovered data files. Procedure --------- -This procedure assumes a ``/data/db`` directory that contains data -files for a ``users`` database, and a ``/data/recovery`` directory -for recovered files. Use the following sequence of operations with -the relevant directory and database names. +This procedure assumes a ``/data/db`` directory that contains data files +for a ``users`` database and a ``/data/recovery`` directory for recovered +files. Use the following sequence of operations with the relevant +directory and database names. Never remove the ``mongod.lock`` file from +the data directory. .. include:: /includes/steps/recover-data-files.rst +Related Documents +----------------- + +- :doc:`maintain-valid-data-files` + +- :doc:`detect-invalid-data-files` + +- :doc:`/tutorial/manage-journaling` + +- :doc:`/core/backups` + +- :doc:`/administration/backup`