DOCSP-23262 check oplog window size v0.9.0 (#67)

Dave Cuthbert · web-flow · commit ccabb1996a5d · 2022-07-18T16:30:52.000-04:00
* DOCSP-23262 oplog size * DOCSP-23262 oplog window sizing * Staging fixes * Staging fixes * Staging fixes * Review feedback
diff --git a/source/faq.txt b/source/faq.txt
@@ -29,16 +29,10 @@ clusters.
 
 .. _c2c-faq-increase-oplog:
 
-Should I increase the size of the oplog in the source cluster?
---------------------------------------------------------------
+Should I increase the size of the ``oplog`` in the source cluster?
+------------------------------------------------------------------
 
-If you anticipate synchronizing a large data set, or if you plan to
-pause synchronization for an extended period of time, increase the size
-of the replica set :term:`oplog` in the source cluster.
-
-The ``oplog`` in the source cluster must be large enough to track
-events that happen during the time it takes to complete the initial
-sync to the destination cluster.
+.. include:: /includes/fact-oplog-background
 
 The proper ``oplog`` size depends on system hardware, network speed,
 and other factors including system workload. However, assuming network
@@ -49,6 +43,9 @@ required ``oplog`` size is:
 
    minimumRetentionHours = dataSizeInGB / 30
 
+To estimate the size of ``oplog`` needed for initial synchronization,
+see: :ref:`c2c-oplog-sizing`.
+
 To learn more about how to increase the size of the ``oplog``, see:
 :ref:`tutorial-change-oplog-size`. 
 
diff --git a/source/includes/fact-oplog-background.rst b/source/includes/fact-oplog-background.rst
@@ -0,0 +1,7 @@
+The :term:`oplog` in the source cluster must be large enough to track
+events that happen during the time it takes to complete the initial
+sync to the destination cluster.
+
+If you anticipate synchronizing a large data set, or if you plan to
+pause synchronization for an extended period of time, increase the size
+of the replica set ``oplog`` in the source cluster.
diff --git a/source/quickstart.txt b/source/quickstart.txt
@@ -251,3 +251,6 @@ Synchronization Notes
   but macOS in particular has low defaults. See :ref:`UNIX ulimit
   settings <system-resource-utilization>`.
 
+- To estimate the size of ``oplog`` needed for initial synchronization,
+  see: :ref:`c2c-oplog-sizing`.
+
diff --git a/source/reference.txt b/source/reference.txt
@@ -12,5 +12,6 @@ Reference
    /reference/api
    /reference/mongosync-states
    /reference/configuration
+   /reference/oplog-sizing
    /reference/limitations
 
diff --git a/source/reference/limitations.txt b/source/reference/limitations.txt
@@ -6,6 +6,14 @@ Limitations
 
 .. include:: /includes/preview-warning
 
+.. default-domain:: mongodb
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 1
+   :class: twocols
+
 {+c2c-full-product-name+} is under active development. This preview
 release has several limitations. Review the list for changes after the
 {+c2c-product-name+} General Availability release.
@@ -138,8 +146,3 @@ In these cases, the replication of ``dropDatabase`` removes all user-created
 collections from the database, but leaves its system collections on the
 destination cluster.
 
-.. contents:: On this page
-   :local:
-   :backlinks: none
-   :depth: 1
-   :class: singlecol
diff --git a/source/reference/mongosync-states.txt b/source/reference/mongosync-states.txt
@@ -6,13 +6,14 @@
 
 .. default-domain:: mongodb
 
+.. include:: /includes/preview-warning
+
 .. contents:: On this page
    :local:
    :backlinks: none
    :depth: 2
    :class: singlecol
 
-
 ``mongosync`` enters different states depending on the requests it
 receives. ``mongosync`` can only be in a single state at a given time.
 The current ``mongosync`` states determines which API operations you can
diff --git a/source/reference/oplog-sizing.txt b/source/reference/oplog-sizing.txt
@@ -0,0 +1,95 @@
+.. _c2c-oplog-sizing:
+
+================
+``oplog`` Sizing
+================
+
+.. include:: /includes/preview-warning
+
+.. default-domain:: mongodb
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 2
+   :class: singlecol
+
+The :ref:`mongosync <c2c-mongosync>` program uses :ref:`change streams
+<changeStreams>` to synchronize data between source and destination
+clusters. ``mongosync`` does not access the :term:`oplog` directly,
+but when a change stream returns events from the past, the events must
+be within the :term:`oplog` time range. 
+
+.. include:: /includes/fact-oplog-background.txt
+
+Estimate ``oplog`` Size Needed for Initial Sync
+-----------------------------------------------
+
+.. procedure::
+   :style: normal
+
+   .. step:: Determine the ``oplog`` Time Span
+
+      To get the difference in seconds between the first and last entry
+      in the ``oplog`` run :method:`db.getReplicationInfo()`. If you
+      are replicating a sharded cluster, run the command on each shard.
+      
+      .. code-block:: javascript
+       
+         db.getReplicationInfo().timeDiff
+         
+      The value returned is the minimum ``oplog`` window of the
+      cluster. If there are multiple shards, the smallest number is the
+      minimum ``oplog`` window.
+
+   .. step:: Estimate Copy Rate During Synching
+
+      .. _c2c-step-est-size:
+
+      To gather performance data while synching, start the sync process
+      and monitor how fast data is transferred between clusters.
+      
+      To start syncing, run the :ref:`/start <c2c-api-start>` command.
+
+      To get the ``copy_rate``:
+      
+      - run the  :ref:`/progress <c2c-api-progress>` command to get
+        ``estimatedCopiedBytes_time01``
+      - wait a second or two
+      - run the  :ref:`/progress <c2c-api-progress>` command to get
+        ``estimatedCopiedBytes_time02``
+      
+      The ``copy_rate`` is:
+
+      .. code-block:: shell
+
+         copy_rate = ( estimatedCopiedBytes_time02 - estimatedCopiedBytes_01) / time_between_requests
+
+   .. step:: Estimate Copy Time
+
+      Estimate the time needed to copy the entire collection. The
+      estimated copy time is:
+      
+      .. code-block:: shell
+
+         estimatedCopyTime = estimatedTotalBytes / copy_rate
+
+   .. step:: Validate ``oplog`` Size
+
+      If the estimated time is larger than the minimum oplog window you
+      must cancel synchronization. Before restarting, make one of the
+      following changes:
+
+      - Increase the oplog window. Use :dbcommand:`replSetResizeOplog`
+        to set ``minRetentionHours`` greater than the estimated copy
+        time.
+      - Scale up the ``mongosync`` instance. Add cpu or memory to scale
+        up the ``mongosync`` node so that it has a higher copy rate.
+
+.. note::
+
+   The copy rate may vary during synchronization. To monitor progress,
+   repeat the :ref:`steps to estimate the copy rate
+   <c2c-step-est-size>` and verify that the copy rate stays about the
+   same.
+
