Skip to content

Commit ccabb19

Browse files
author
Dave Cuthbert
authored
DOCSP-23262 check oplog window size v0.9.0 (#67)
* DOCSP-23262 oplog size * DOCSP-23262 oplog window sizing * Staging fixes * Staging fixes * Staging fixes * Review feedback
1 parent 1aa8c4e commit ccabb19

File tree

7 files changed

+122
-15
lines changed

7 files changed

+122
-15
lines changed

‎source/faq.txt‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -29,16 +29,10 @@ clusters.
2929

3030
.. _c2c-faq-increase-oplog:
3131

32-
Should I increase the size of the oplog in the source cluster?
33-
--------------------------------------------------------------
32+
Should I increase the size of the ``oplog`` in the source cluster?
33+
------------------------------------------------------------------
3434

35-
If you anticipate synchronizing a large data set, or if you plan to
36-
pause synchronization for an extended period of time, increase the size
37-
of the replica set :term:`oplog` in the source cluster.
38-
39-
The ``oplog`` in the source cluster must be large enough to track
40-
events that happen during the time it takes to complete the initial
41-
sync to the destination cluster.
35+
.. include:: /includes/fact-oplog-background
4236

4337
The proper ``oplog`` size depends on system hardware, network speed,
4438
and other factors including system workload. However, assuming network
@@ -49,6 +43,9 @@ required ``oplog`` size is:
4943

5044
minimumRetentionHours = dataSizeInGB / 30
5145

46+
To estimate the size of ``oplog`` needed for initial synchronization,
47+
see: :ref:`c2c-oplog-sizing`.
48+
5249
To learn more about how to increase the size of the ``oplog``, see:
5350
:ref:`tutorial-change-oplog-size`.
5451

‎source/includes/fact-oplog-background.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
The :term:`oplog` in the source cluster must be large enough to track
2+
events that happen during the time it takes to complete the initial
3+
sync to the destination cluster.
4+
5+
If you anticipate synchronizing a large data set, or if you plan to
6+
pause synchronization for an extended period of time, increase the size
7+
of the replica set ``oplog`` in the source cluster.

‎source/quickstart.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,3 +251,6 @@ Synchronization Notes
251251
but macOS in particular has low defaults. See :ref:`UNIX ulimit
252252
settings <system-resource-utilization>`.
253253

254+
- To estimate the size of ``oplog`` needed for initial synchronization,
255+
see: :ref:`c2c-oplog-sizing`.
256+

‎source/reference.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,6 @@ Reference
1212
/reference/api
1313
/reference/mongosync-states
1414
/reference/configuration
15+
/reference/oplog-sizing
1516
/reference/limitations
1617

‎source/reference/limitations.txt‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,14 @@ Limitations
66

77
.. include:: /includes/preview-warning
88

9+
.. default-domain:: mongodb
10+
11+
.. contents:: On this page
12+
:local:
13+
:backlinks: none
14+
:depth: 1
15+
:class: twocols
16+
917
{+c2c-full-product-name+} is under active development. This preview
1018
release has several limitations. Review the list for changes after the
1119
{+c2c-product-name+} General Availability release.
@@ -138,8 +146,3 @@ In these cases, the replication of ``dropDatabase`` removes all user-created
138146
collections from the database, but leaves its system collections on the
139147
destination cluster.
140148

141-
.. contents:: On this page
142-
:local:
143-
:backlinks: none
144-
:depth: 1
145-
:class: singlecol

‎source/reference/mongosync-states.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,14 @@
66

77
.. default-domain:: mongodb
88

9+
.. include:: /includes/preview-warning
10+
911
.. contents:: On this page
1012
:local:
1113
:backlinks: none
1214
:depth: 2
1315
:class: singlecol
1416

15-
1617
``mongosync`` enters different states depending on the requests it
1718
receives. ``mongosync`` can only be in a single state at a given time.
1819
The current ``mongosync`` states determines which API operations you can

‎source/reference/oplog-sizing.txt‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
.. _c2c-oplog-sizing:
2+
3+
================
4+
``oplog`` Sizing
5+
================
6+
7+
.. include:: /includes/preview-warning
8+
9+
.. default-domain:: mongodb
10+
11+
.. contents:: On this page
12+
:local:
13+
:backlinks: none
14+
:depth: 2
15+
:class: singlecol
16+
17+
The :ref:`mongosync <c2c-mongosync>` program uses :ref:`change streams
18+
<changeStreams>` to synchronize data between source and destination
19+
clusters. ``mongosync`` does not access the :term:`oplog` directly,
20+
but when a change stream returns events from the past, the events must
21+
be within the :term:`oplog` time range.
22+
23+
.. include:: /includes/fact-oplog-background.txt
24+
25+
Estimate ``oplog`` Size Needed for Initial Sync
26+
-----------------------------------------------
27+
28+
.. procedure::
29+
:style: normal
30+
31+
.. step:: Determine the ``oplog`` Time Span
32+
33+
To get the difference in seconds between the first and last entry
34+
in the ``oplog`` run :method:`db.getReplicationInfo()`. If you
35+
are replicating a sharded cluster, run the command on each shard.
36+
37+
.. code-block:: javascript
38+
39+
db.getReplicationInfo().timeDiff
40+
41+
The value returned is the minimum ``oplog`` window of the
42+
cluster. If there are multiple shards, the smallest number is the
43+
minimum ``oplog`` window.
44+
45+
.. step:: Estimate Copy Rate During Synching
46+
47+
.. _c2c-step-est-size:
48+
49+
To gather performance data while synching, start the sync process
50+
and monitor how fast data is transferred between clusters.
51+
52+
To start syncing, run the :ref:`/start <c2c-api-start>` command.
53+
54+
To get the ``copy_rate``:
55+
56+
- run the :ref:`/progress <c2c-api-progress>` command to get
57+
``estimatedCopiedBytes_time01``
58+
- wait a second or two
59+
- run the :ref:`/progress <c2c-api-progress>` command to get
60+
``estimatedCopiedBytes_time02``
61+
62+
The ``copy_rate`` is:
63+
64+
.. code-block:: shell
65+
66+
copy_rate = ( estimatedCopiedBytes_time02 - estimatedCopiedBytes_01) / time_between_requests
67+
68+
.. step:: Estimate Copy Time
69+
70+
Estimate the time needed to copy the entire collection. The
71+
estimated copy time is:
72+
73+
.. code-block:: shell
74+
75+
estimatedCopyTime = estimatedTotalBytes / copy_rate
76+
77+
.. step:: Validate ``oplog`` Size
78+
79+
If the estimated time is larger than the minimum oplog window you
80+
must cancel synchronization. Before restarting, make one of the
81+
following changes:
82+
83+
- Increase the oplog window. Use :dbcommand:`replSetResizeOplog`
84+
to set ``minRetentionHours`` greater than the estimated copy
85+
time.
86+
- Scale up the ``mongosync`` instance. Add cpu or memory to scale
87+
up the ``mongosync`` node so that it has a higher copy rate.
88+
89+
.. note::
90+
91+
The copy rate may vary during synchronization. To monitor progress,
92+
repeat the :ref:`steps to estimate the copy rate
93+
<c2c-step-est-size>` and verify that the copy rate stays about the
94+
same.
95+

0 commit comments

Comments
 (0)