From f84b8ab25d196ff3363f8f4e0a841317cf513dcf Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Thu, 4 Sep 2014 16:01:21 -0400 Subject: [PATCH 1/3] DOCS-3350 migrate config servers --- ...ate-config-servers-with-same-hostname.yaml | 41 +++++++++++++++++++ ...rate-config-servers-with-same-hostname.txt | 32 +-------------- ...igrate-sharded-cluster-to-new-hardware.txt | 4 +- 3 files changed, 43 insertions(+), 34 deletions(-) create mode 100644 source/includes/steps-migrate-config-servers-with-same-hostname.yaml diff --git a/source/includes/steps-migrate-config-servers-with-same-hostname.yaml b/source/includes/steps-migrate-config-servers-with-same-hostname.yaml new file mode 100644 index 00000000000..773a6f5c970 --- /dev/null +++ b/source/includes/steps-migrate-config-servers-with-same-hostname.yaml @@ -0,0 +1,41 @@ +title: Shut down the config server. +stepnum: 1 +ref: shutdown +content: | + This renders all config data for the sharded cluster "read only." +--- +title: Change the DNS entry. +stepnum: 2 +ref: dns +content: | + Change the DNS entry that points to the system that provided the old + config server, so that the *same* hostname points to the new + system. + How you do this depends on how you organize your DNS and + hostname resolution services. +--- +title: Copy the the data. +stepnum: 3 +ref: db +pre: | + Copy the contents of :setting:`~storage.dbPath` from the old config server to + the new config server. +action: + pre: | + For example, to copy the contents of :setting:`~storage.dbPath` to a machine + named ``mongodb.config2.example.net``, you might issue a command + similar to the following: + language: sh + code: | + rsync -az /data/configdb/ mongodb.config2.example.net:/data/configdb +--- +title: Start the config server instance on the new system. +stepnum: 4 +ref: start +action: + pre: | + For example: + language: sh + code: | + mongod --configsvr +... diff --git a/source/tutorial/migrate-config-servers-with-same-hostname.txt b/source/tutorial/migrate-config-servers-with-same-hostname.txt index 39cf86c24d9..9b557f66f64 100644 --- a/source/tutorial/migrate-config-servers-with-same-hostname.txt +++ b/source/tutorial/migrate-config-servers-with-same-hostname.txt @@ -14,37 +14,7 @@ reverse order from how they are listed in the :program:`mongos` instances' :setting:`~sharding.configDB` string. Start with the last config server listed in the :setting:`~sharding.configDB` string. -.. start-migrate-config-server-with-same-hostname - -#. Shut down the config server. - - This renders all config data for the sharded cluster "read only." - -#. Change the DNS entry that points to the system that provided the old - config server, so that the *same* hostname points to the new - system. - How you do this depends on how you organize your DNS and - hostname resolution services. - -#. Copy the contents of :setting:`~storage.dbPath` from the old config server to - the new config server. - - For example, to copy the contents of :setting:`~storage.dbPath` to a machine - named ``mongodb.config2.example.net``, you might issue a command - similar to the following: - - .. code-block:: sh - - rsync -az /data/configdb/ mongodb.config2.example.net:/data/configdb - -#. Start the config server instance on the new system. The default - invocation is: - - .. code-block:: sh - - mongod --configsvr - -.. end-migrate-config-server-with-same-hostname +.. include:: /includes/steps/migrate-config-servers-with-same-hostname.rst When you start the third config server, your cluster will become writable and it will be able to create new splits and migrate chunks diff --git a/source/tutorial/migrate-sharded-cluster-to-new-hardware.txt b/source/tutorial/migrate-sharded-cluster-to-new-hardware.txt index 5e438b6090d..b48e198f817 100644 --- a/source/tutorial/migrate-sharded-cluster-to-new-hardware.txt +++ b/source/tutorial/migrate-sharded-cluster-to-new-hardware.txt @@ -67,9 +67,7 @@ Do not rename a config server during this process. .. important:: Start with the *last* config server listed in :setting:`~sharding.configDB`. -.. include:: /tutorial/migrate-config-servers-with-same-hostname.txt - :start-after: start-migrate-config-server-with-same-hostname - :end-before: end-migrate-config-server-with-same-hostname +.. include:: /includes/steps/migrate-config-servers-with-same-hostname.rst .. _migrate-to-new-hardware-restart-mongos: From d6fb685f7dcaf2f7154a30c45537d1df331fb3c9 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Thu, 4 Sep 2014 16:03:54 -0400 Subject: [PATCH 2/3] DOCS-3350 backup sharded cluster --- ...ded-cluster-with-filesystem-snapshots.yaml | 59 ++++++++++++++ .../steps-backup-sharded-clusters-dumps.yaml | 10 ++- ...rded-cluster-with-filesystem-snapshots.txt | 76 +------------------ 3 files changed, 67 insertions(+), 78 deletions(-) create mode 100644 source/includes/steps-backup-sharded-cluster-with-filesystem-snapshots.yaml diff --git a/source/includes/steps-backup-sharded-cluster-with-filesystem-snapshots.yaml b/source/includes/steps-backup-sharded-cluster-with-filesystem-snapshots.yaml new file mode 100644 index 00000000000..5d9bd5dd938 --- /dev/null +++ b/source/includes/steps-backup-sharded-cluster-with-filesystem-snapshots.yaml @@ -0,0 +1,59 @@ +stepnum: 1 +source: + file: steps-backup-sharded-clusters-dumps.yaml + ref: disable-balancer +--- +stepnum: 2 +source: + file: steps-backup-sharded-clusters-dumps.yaml + ref: lock-replica-set-members +--- +title: "Back up one of the :ref:`config servers `." +stepnum: 3 +ref: backup-config-server +pre: | + Backing up a config server backs up the sharded cluster's metadata. You + need back up only one config server, as they all hold the same data + + Do one of the following to back up one of the config servers: + + **Create a file-system snapshot of the config server.** Use the procedure in + :doc:`/tutorial/backup-with-filesystem-snapshots`. + + .. important:: This is only available if the config server has :term:`journaling ` enabled. *Never* use :method:`db.fsyncLock()` on config databases. + + **Use mongodump to backup the config server.** + Issue :program:`mongodump` against one of the config :program:`mongod` + instances or via the :program:`mongos`. + + Your config servers must run MongoDB 2.4 or later with the + :option:`--configsvr ` option and the + :program:`mongodump` option must include the :option:`--oplog + ` option to ensure that the dump includes a partial + oplog containing operations from the duration of the mongodump + operation. For example: +action: + language: sh + code: | + mongodump --oplog --db config +--- +title: Backup the replica set members of the shards that you locked. +stepnum: 4 +ref: backup-replica-set-members +pre: | + You may back up the shards in parallel. For each shard, create a + snapshot. Use the procedure in + :doc:`/tutorial/backup-with-filesystem-snapshots`. +--- +title: Unlock the locked replica set members. +stepnum: 5 +ref: restart-replica-set-members +content: | + Unlock all locked replica set members of each shard using the + :method:`db.fsyncUnlock()` method in the :program:`mongo` shell. +--- +stepnum: 6 +source: + file: steps-backup-sharded-clusters-dumps.yaml + ref: reenable-balancer-process +... diff --git a/source/includes/steps-backup-sharded-clusters-dumps.yaml b/source/includes/steps-backup-sharded-clusters-dumps.yaml index 083fa7743de..ef182292ca1 100644 --- a/source/includes/steps-backup-sharded-clusters-dumps.yaml +++ b/source/includes/steps-backup-sharded-clusters-dumps.yaml @@ -30,11 +30,15 @@ pre: | approximation of a single moment in time. Lock these :program:`mongod` instances in as short of an interval as possible. - To lock or freeze a sharded cluster, you shut down one member of each - replica set. Ensure that the :term:`oplog` has sufficient capacity to - allow these secondaries to catch up to the state of the primaries after + Before you lock a secondary, + ensure that the :term:`oplog` has sufficient capacity to + allow the secondary to catch up to the state of the primary after finishing the backup procedure. See :ref:`replica-set-oplog-sizing` for more information. + + To lock a secondary, connect through the :program:`mongo` shell to the + secondary member's :program:`mongod` instance and issue the + :method:`db.fsyncLock()` method. --- title: Backup one config server. stepnum: 3 diff --git a/source/tutorial/backup-sharded-cluster-with-filesystem-snapshots.txt b/source/tutorial/backup-sharded-cluster-with-filesystem-snapshots.txt index 51938bd899a..3c08ca15076 100644 --- a/source/tutorial/backup-sharded-cluster-with-filesystem-snapshots.txt +++ b/source/tutorial/backup-sharded-cluster-with-filesystem-snapshots.txt @@ -37,78 +37,4 @@ of the backup while minimizing impact on the cluster by taking the backup from a secondary member of the replica set that provides each shard. -1. Disable the :term:`balancer` process that equalizes the - distribution of data among the :term:`shards `. To disable - the balancer, use the :method:`sh.stopBalancer()` method in the - :program:`mongo` shell. For example: - - .. code-block:: sh - - use config - sh.stopBalancer() - - For more information, see the - :ref:`sharding-balancing-disable-temporarily` procedure. - - .. warning:: - - It is essential that you stop the balancer before creating - backups. If the balancer remains active, your resulting backups - could have duplicate data or miss some data, as :term:`chunks - ` may migrate while recording backups. - -#. Lock one secondary member of each replica set in each shard so that your - backups reflect the state of your database at the nearest possible - approximation of a single moment in time. Lock these - :program:`mongod` instances in as short of an interval as possible. - - To lock a secondary, connect through the :program:`mongo` shell to the - secondary member's :program:`mongod` instance and issue the - :method:`db.fsyncLock()` method. - -#. Back up one of the :ref:`config servers `. - Backing up a config server backs up the sharded cluster's metadata. You - need back up only one config server, as they all hold the same data - - Do one of the following to back up one of the config servers: - - - Create a file-system snapshot of the config server. Use the procedure in - :doc:`/tutorial/backup-with-filesystem-snapshots`. - - .. important:: This is only available if the config server has - :term:`journaling ` enabled. *Never* - use :method:`db.fsyncLock()` on config databases. - - - Use :program:`mongodump` to backup the config server. Issue - :program:`mongodump` against one of the config :program:`mongod` - instances or via the :program:`mongos`. - - If you are running MongoDB 2.4 or later with the - :option:`--configsvr ` option, then include the - :option:`--oplog ` option when running - :program:`mongodump` to ensure that the dump includes a partial oplog - containing operations from the duration of the mongodump operation. - For example: - - .. code-block:: sh - - mongodump --oplog --db config - -#. Back up the replica set members of the shards that you locked. You - may back up the shards in parallel. For each shard, create a - snapshot. Use the procedure in - :doc:`/tutorial/backup-with-filesystem-snapshots`. - -#. Unlock all locked replica set members of each shard using the - :method:`db.fsyncUnlock()` method in the :program:`mongo` shell. - -#. Re-enable the balancer with the :method:`sh.setBalancerState()` - method. - - Use the following command sequence when connected to the - :program:`mongos` with the :program:`mongo` shell: - - .. code-block:: javascript - - use config - sh.setBalancerState(true) +.. include:: /includes/steps/backup-sharded-cluster-with-filesystem-snapshots.rst From 62f4e799ac53fae4ffda12d8ecb7817b0844ce4f Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Thu, 4 Sep 2014 16:08:49 -0400 Subject: [PATCH 3/3] DOCS-3350 restore a sharded cluster --- ...re-sharded-cluster-from-database-dump.yaml | 8 + ...restore-sharded-cluster-from-snapshot.yaml | 138 ++++++++++++++++++ source/tutorial/restore-sharded-cluster.txt | 80 +++++----- 3 files changed, 182 insertions(+), 44 deletions(-) create mode 100644 source/includes/steps-restore-sharded-cluster-from-database-dump.yaml create mode 100644 source/includes/steps-restore-sharded-cluster-from-snapshot.yaml diff --git a/source/includes/steps-restore-sharded-cluster-from-database-dump.yaml b/source/includes/steps-restore-sharded-cluster-from-database-dump.yaml new file mode 100644 index 00000000000..2cf7aeae1ed --- /dev/null +++ b/source/includes/steps-restore-sharded-cluster-from-database-dump.yaml @@ -0,0 +1,8 @@ +title: Place +stepnum: 1 +ref: place +--- +title: holder +stepnum: 2 +ref: holder +... diff --git a/source/includes/steps-restore-sharded-cluster-from-snapshot.yaml b/source/includes/steps-restore-sharded-cluster-from-snapshot.yaml new file mode 100644 index 00000000000..6bd4a7a8854 --- /dev/null +++ b/source/includes/steps-restore-sharded-cluster-from-snapshot.yaml @@ -0,0 +1,138 @@ +title: Stop all MongoDB processes. +stepnum: 1 +ref: stop-processes +pre: | + Stop **all** :program:`mongos` and :program:`mongod` processes. For + :program:`mongod` processes, this includes the :program:`mongod` + instance for each :term:`replica set` member in each :term:`shard` and + includes all the :program:`mongod` processes running as :term:`config + servers `. +action: | + pre: | + To stop a process, you can connect to the process through the + :program:`mongo` shell and issue :method:`db.shutdownServer()`: + language: javascript + code: | + mongo + use admin + db.shutdownServer() +--- +title: "Copy the restore files to the data directory of each :program:`mongod` server." +stepnum: 2 +ref: restore +content: | + The restore files are the data files for a given point in time for each + shard and for the config servers. Depending on how you backed up the + cluster, the files might be tarred or zipped. + + Copy the restore files for a given shard to all the replica set members + on that shard. Copy the restore files for the config servers to each + config server. + + Copy the restore files to the location where the :program:`mongod` + instance will access them. This is the location you will specify as the + :setting:`dbpath` when running the :program:`mongod`. If the restore + files are zipped or tarred, unzip or untar them. +--- +title: Restart the config servers. +stepnum: 3 +ref: restart-config-servers +action: + pre: | + For example, for a config server that runs on port ``27019`` and that + stores data in ``/data/configdb``, issue: + language: sh + code: | + mongod --configsvr --port 27019 --dbpath /data/configdb +--- +title: "If the hostnames or ports for shards have changed, update cluster's shard metadata." +stepnum: 4 +ref: metadata +action: + - pre: | + Start one :program:`mongos` instance, using the updated + configuration string in the :option:`configdb ` + option. For example + language: sh + code: | + mongos --configdb + - pre: | + Connect to the :program:`mongos`, go to the :doc:`config + ` database, and query the ``shards`` + collection to display shard metadata. For example: + language: sh + code: | + mongo --port + use config + db.shards.find().pretty() + - pre: | + For a given shard, the ``host`` field displays the shard's replica + set, hostname, and port. For example: + language: javascript + code: | + { "_id" : "shard0000", "host" : "rs1/localhost:30000" } + - pre: | + Use the :method:`db.collection.update()` method to update each + shard's data to the correct hostname and port. For example, if the + above port has changed to 40000, issue: + language: javascript + code: | + db.people.update( + { "_id": "shard0000" }, + { + "_id": "shard0000", + "host": "rs1/localhost:40000" + } + ) + - pre: | + Stop the :program:`mongos`. For example: + language: javascript + code: | + db.shutdownServer() +# TODO: Before starting the primaries, do they have to drop the oplog and re-seed? +--- +title: "Restart each primary as part of its replica set." +stepnum: 5 +ref: restart-each-primary +action: + pre: | + For example, the following command issued from a system shell starts a + primary and specifies the port number, data directory, and replica set. + language: sh + code: | + mongod --port 27017 --dbpath /data --replSet rsa +# TODO: Do they have to initiate the replica set? +--- +title: Restart each secondary as part of its replica set. +stepnum: 6 +ref: restart-each-secondary +action: + pre: | + For example, the following command issued from a system shell starts a + secondary and specifies the port number, data directory, and replica set. + language: sh + code: | + mongod --port 27017 --dbpath /data --replSet rsa +# TODO: Do they have to add members to the replica set? +--- +title: "Restart the :program:`mongos` instances." +stepnum: 7 +ref: restart-mongos-instances +content: | + Restart the :program:`mongos` instances. If hostnames or ports have + changed for the config servers, making sure to use the updated + information in the :option:`--configdb` option. +--- +title: "Connect to a :program:`mongos` instance and view shard status." +stepnum: 8 +ref: connect +action: + pre: | + Connect to a :program:`mongos` instance from a :program:`mongo` shell + and use the :method:`db.printShardingStatus()` method to ensure that + the cluster is operational, as follows: + language: sh + code: | + db.printShardingStatus() + show collections +... diff --git a/source/tutorial/restore-sharded-cluster.txt b/source/tutorial/restore-sharded-cluster.txt index 4585ab28d7c..855740289b1 100644 --- a/source/tutorial/restore-sharded-cluster.txt +++ b/source/tutorial/restore-sharded-cluster.txt @@ -7,66 +7,58 @@ Restore a Sharded Cluster Overview -------- -The procedure outlined in this document addresses how to restore an -entire sharded cluster. For information on related backup procedures -consider the following tutorials which describe backup procedures in -greater detail: +You can restore a :term:`sharded cluster` onto new or existing hardware +from the artifacts captured by a :doc:`snapshot +` or +:doc:`database dump +`. -- :doc:`/tutorial/backup-sharded-cluster-with-filesystem-snapshots` -- :doc:`/tutorial/backup-sharded-cluster-with-database-dumps` +Consideration +------------- -The exact procedure used to restore a database depends on the method -used to capture the backup. See the :doc:`/core/backups` -document for an overview of backups with MongoDB and -:doc:`/administration/backup-sharded-clusters` for a complete -information on backups in MongoDB and backups of sharded clusters in -particular. +For an overview of backup in MongoDB, see :doc:`/core/backups`. -Procedure ---------- +Procedures +---------- -#. Stop all :program:`mongos` and :program:`mongod` processes, - including all shards *and* all config servers. +Select the procedure appropriate to the backed-up data: -#. Restore the following: +- :ref:`restore-sharded-cluster-from-snapshot` - - Data files for each server in each :term:`shard`. Because replica - sets provide each production shard, restore all the members of - the replica set or use the other standard approaches for - restoring a replica set from backup. See the - :ref:`backup-restore-snapshot` and :ref:`backup-restore-dump` - sections for details on these procedures. +- :ref:`restore-sharded-cluster-from-database-dump` - - Data files for each :ref:`config server `. +.. _restore-sharded-cluster-from-snapshot: -#. Restart all the :ref:`config servers ` - :program:`mongod` instances by issuing command similar to the - following, using values appropriate - to your configuration: +Restore Sharded Cluster from a Snapshot ++++++++++++++++++++++++++++++++++++++++ - .. code-block:: sh +The sequence to restore from a snapshot is to: - mongod --configsvr --dbpath /data/configdb --port 27019 +- stop all MongoDB processes, -#. If shard hostnames **have changed**: +- distribute the restore files to their new locations, - a. Start one :program:`mongos` instance, using the updated - config string with the new ``configdb`` hostnames and ports. +- start the :program:`mongod` instances, - #. Update the ``shards`` collection in the :ref:`config-database` to - reflect the new hostnames. +- configure each shard's replica set, and - #. Stop the :program:`mongos` instance. +- configure and start the cluster. -#. Restart all the shard :program:`mongod` instances. +To restore a sharded cluster from a snapshot: -#. Restart all the :program:`mongos` instances, making sure to use the updated config string. +.. include:: /includes/steps/restore-sharded-cluster-from-snapshot.rst -#. Connect to a :program:`mongos` instance from a :program:`mongo` shell - and use the :method:`db.printShardingStatus()` method to ensure - that the cluster is operational, as follows: +.. _restore-sharded-cluster-from-database-dump: - .. code-block:: javascript +Restore Sharded Cluster from a Database Dump +++++++++++++++++++++++++++++++++++++++++++++ - db.printShardingStatus() - show collections +The sequence to restore from database dump is to: + +- start the :program:`mongod` instances, + +- and restore the database dumps. + +To restore a sharded cluster from a database dump: + +.. include:: /includes/steps/restore-sharded-cluster-from-database-dump.rst