DOCSP-29500 Clarify oplog Sizing (#131)

* DOCSP-29500 clarify oplog sizing

* fixes build issue

* Refactors text

* Refactors text

* Refactors text

* Fixes per Dave

* Fixes per Evgeni

* fixes per Alex

* fixes per Alex

---------

Co-authored-by: Tim Fogarty <[email protected]>

Author: kennethdyer

source/faq.txt

@@ -31,19 +31,10 @@ clusters.
 Should I increase the size of the ``oplog`` in the source cluster?
 ------------------------------------------------------------------
 
-.. include:: /includes/fact-oplog-background
-
-The proper ``oplog`` size depends on system hardware, network speed,
-and other factors including system workload. However, assuming network
-transfer speeds of 30-50GB per hour, a rough formula to estimate the
-required ``oplog`` size is:
-
-.. code-block:: shell
-
-   minimumRetentionHours = dataSizeInGB / 30
+The :term:`oplog` is a special capped collection that keeps a rolling 
+record of all operations that modify the data stored in your databases. 
 
-To estimate the size of ``oplog`` needed for initial synchronization,
-see: :ref:`c2c-oplog-sizing`.
+.. include:: /includes/fact-oplog-background
 
 To learn more about how to increase the size of the ``oplog``, see:
 :ref:`tutorial-change-oplog-size`. 

source/includes/fact-oplog-background.rst

@@ -1,7 +1,15 @@
-The :term:`oplog` in the source cluster must be large enough to track
-events that happen during the time it takes to complete the initial
-sync to the destination cluster.
+
+``mongosync`` applies operations in the ``oplog`` on the source cluster
+to the data on the destination cluster.  When older operations 
+that ``mongosync`` has not applied roll off the ``oplog`` 
+on the source cluster, the sync fails and ``mongosync`` exits.
+
+During initial sync, ``mongosync`` may apply operations at a slower
+rate. After ``mongosync`` completes the initial sync, it applies changes 
+faster and stays more current in the source cluster ``oplog``.
 
 If you anticipate syncing a large data set, or if you plan to pause
-synchronization for an extended period of time, increase the size of the
-replica set ``oplog`` in the source cluster.
+synchronization for an extended period of time, you might exceed the
+:term:`oplog window`. Use the :setting:`~replication.oplogSizeMB` setting
+to increase the size of the ``oplog`` on the source cluster.
+

source/reference/oplog-sizing.txt

@@ -16,11 +16,12 @@ The :ref:`mongosync <c2c-mongosync>` program uses :ref:`change streams
 <changeStreams>` to synchronize data between source and destination
 clusters. ``mongosync`` does not access the :term:`oplog` directly,
 but when a change stream returns events from the past, the events must
-be within the :term:`oplog` time range. 
+be within the ``oplog`` time range. 
 
-.. include:: /includes/fact-oplog-background.txt
 
-Estimate ``oplog`` Size Needed for Initial Sync
+.. include:: /includes/fact-oplog-background
+
+Monitor ``oplog`` Size Needed for Initial Sync
 -----------------------------------------------
 
 .. procedure::
@@ -40,55 +41,30 @@ Estimate ``oplog`` Size Needed for Initial Sync
       cluster. If there are multiple shards, the smallest number is the
       minimum ``oplog`` window.
 
-   .. step:: Estimate Copy Rate During Synching
-
-      .. _c2c-step-est-size:
-
-      To gather performance data while synching, start the sync process
-      and monitor how fast data is transferred between clusters.
-      
-      To start syncing, run the :ref:`/start <c2c-api-start>`
-      command.
+   .. step:: Determine mongosync replication lag
 
-      To get the ``copy_rate``:
-      
-      - run the  :ref:`/progress <c2c-api-progress>` command to get
-        ``estimatedCopiedBytes_time01``
-      - wait a second or two
-      - run the  :ref:`/progress <c2c-api-progress>` command to get
-        ``estimatedCopiedBytes_time02``
-      
-      The ``copy_rate`` is:
-
-      .. code-block:: shell
-
-         copy_rate = ( estimatedCopiedBytes_time02 - estimatedCopiedBytes_01) / time_between_requests
-
-   .. step:: Estimate Copy Time
-
-      Estimate the time needed to copy the entire collection. The
-      estimated copy time is:
-      
-      .. code-block:: shell
+      To get the ``lagTimeSeconds`` value, run the  
+      :ref:`/progress <c2c-api-progress>` command. 
+      The lag time is the time in seconds between the 
+      last event applied by ``mongosync`` and time of the current
+      latest event on the source cluster.
 
-         estimatedCopyTime = estimatedTotalBytes / copy_rate
+      It is a measure of how far behind the source cluster mongosync is.
 
    .. step:: Validate ``oplog`` Size
 
-      If the estimated time is larger than the minimum oplog window you
-      must cancel synchronization. Before restarting, make one of the
-      following changes:
+      If the lag time approaches the minimum ``oplog`` window, you
+      should make one of the following changes:
 
-      - Increase the oplog window. Use :dbcommand:`replSetResizeOplog`
-        to set ``minRetentionHours`` greater than the estimated copy
-        time.
+      - Increase the ``oplog`` window. Use :dbcommand:`replSetResizeOplog`
+        to set ``minRetentionHours`` greater than the current ``oplog`` 
+        window.
       - Scale up the ``mongosync`` instance. Add cpu or memory to scale
         up the ``mongosync`` node so that it has a higher copy rate.
 
 .. note::
 
-   The copy rate may vary during synchronization. To monitor progress,
-   repeat the :ref:`steps to estimate the copy rate
-   <c2c-step-est-size>` and verify that the copy rate stays about the
-   same.
+   The ``oplog`` window and rate of change of replication lag may vary during 
+   synchronization. Repeat these steps during a migration to monitor the 
+   progress.