DOCSP-21763 documents new allowDiskUseByDefault param (#963)

* documents allowDiskUseByDefault param

* internal review feedback

* additional internal review feedback

* additional internal review feedback

* clarifies that { allowDiskUse: true } can still be used to spill working data to disk when allowDiskUseByDefault is set to false

* internal feedback

* second round of review

* external review feedback

* external

* clarification

* includify

* clean-up

* more consistency between sort and sortByCount

* param desc

* includify

* more includification

* continuation

* continuation

* more includification

* update extracts-views blurb to use include

* includes

* consolidate

* consolidation

* remove empty file

* updates cursor.allowDiskUse

* parenthetical re allowDiskUseByDefault default setting

* external review

* external review

Author: jmd-mongo

source/includes/extracts-views.yaml

@@ -34,10 +34,14 @@ content: |
 
    - The view's underlying aggregation pipeline is subject to the 100
      megabyte memory limit for blocking sort and blocking group
-     operations. Starting in MongoDB 4.4, you can issue a
-     :dbcommand:`find` command with ``allowDiskUse: true`` on the view
-     to allow MongoDB to use temporary files for blocking sort and group
-     operations.
+     operations. 
+
+     .. include:: /includes/fact-allowDiskUseByDefault.rst
+
+     Starting in MongoDB 4.4, you can specify the ``allowDiskUse`` 
+     option on a :dbcommand:`find` command. This allows the application 
+     to control whether disk use is allowed when running a 
+     :dbcommand:`find` command against a view.
 
      Prior to MongoDB 4.4, only the :dbcommand:`aggregate` command
      accepted the ``allowDiskUse`` option.

source/includes/fact-agg-memory-limit.rst

@@ -1,22 +1,33 @@
 .. FYI -- 2.5.3 introduced the limit to $group and changed the limit for
    $sort from 10% to 100 MB.
 
-Each individual pipeline stage has a limit of 100 megabytes of RAM. By
-default, if a stage exceeds this limit, MongoDB produces an error. For
-some pipeline stages you can allow pipeline processing to take up more
-space by using the :ref:`allowDiskUse <aggregate-cmd-allowDiskUse>`
-option to enable aggregation pipeline stages to write data to temporary
-files.
+Starting in MongoDB 6.0, the :parameter:`allowDiskUseByDefault` 
+parameter controls whether pipeline stages that require more than 100 
+megabytes of memory to execute write temporary files to disk by 
+default.
+
+- If :parameter:`allowDiskUseByDefault` is set to ``true``, pipeline
+  stages that require more than 100 megabytes of memory to execute 
+  write temporary files to disk by default. You can disable writing 
+  temporary files to disk for specific ``find`` or ``aggregate`` 
+  commands using the ``{ allowDiskUse: false }`` option.
+
+- If :parameter:`allowDiskUseByDefault` is set to ``false``, pipeline
+  stages that require more than 100 megabytes of memory to execute 
+  raise an error by default. You can enable writing temporary files to 
+  disk for specific ``find`` or ``aggregate`` using 
+  the ``{ allowDiskUse: true }`` option.
 
 The :pipeline:`$search` aggregation stage is not restricted to 
 100 megabytes of RAM because it runs in a separate process.
 
-Examples of stages that can spill to disk when :ref:`allowDiskUse
-<aggregate-cmd-allowDiskUse>` is ``true`` are:
+Examples of stages that can write temporary files to disk when 
+:ref:`allowDiskUse <aggregate-cmd-allowDiskUse>` is ``true`` are:
 
 - :pipeline:`$bucket`
 - :pipeline:`$bucketAuto`
 - :pipeline:`$group`
+- :pipeline:`$setWindowFields`
 - :pipeline:`$sort` when the sort operation is not supported by an
   index
 - :pipeline:`$sortByCount`

source/includes/fact-allowDiskUse-option-6.0.rst

@@ -0,0 +1,15 @@
+Use this option to override :parameter:`allowDiskUseByDefault` 
+for a specific query. You can use this option to either:
+
+- Prohibit disk use on a system where disk use is allowed by 
+  default.
+- Allow disk use on a system where disk use is prohibited by 
+  default.
+
+Starting in MongoDB 6.0, if :parameter:`allowDiskUseByDefault` is set 
+to ``true`` and the server requires more than 100 megabytes of memory 
+for a pipeline execution stage, MongoDB automatically writes 
+temporary files to disk unless the query specifies 
+``{ allowDiskUse: false }``.
+
+For details, see :parameter:`allowDiskUseByDefault`.

source/includes/fact-allowDiskUseByDefault.rst

@@ -0,0 +1,14 @@
+Starting in MongoDB 6.0, pipeline stages that require more than 100 
+megabytes of memory to execute write temporary files to disk by 
+default. In earlier verisons of MongoDB, you must pass 
+``{ allowDiskUse: true }`` to individual ``find`` and ``aggregate``
+commands to enable this behavior.
+
+Individual ``find`` and ``aggregate`` commands may override the 
+:parameter:`allowDiskUseByDefault` parameter by either:
+
+- Using ``{ allowDiskUse: true }`` to allow writing temporary files out 
+  to disk when ``allowDiskUseByDefault`` is set to ``false``
+
+- Using ``{ allowDiskUse: false }`` to probibit writing temporary files
+  out to disk when ``allowDiskUseByDefault`` is set to ``true``

source/reference/command/aggregate.txt

@@ -105,11 +105,9 @@ arguments:
 
      - boolean
 
-     - Optional. Enables writing to temporary files. When set to
-       ``true``, aggregation stages can write data to the :file:`_tmp`
-       subdirectory in the :setting:`~storage.dbPath` directory.
+     - Optional. 
 
-       By default, ``allowDiskUse`` is ``false``.
+       .. include:: /includes/fact-allowDiskUse-option-6.0.rst
 
        .. include:: /includes/extracts/4.2-changes-usedDisk.rst
 
@@ -436,23 +434,10 @@ to ``true`` to return information about the aggregation operation.
    :method:`db.collection.aggregate()` method
 
 
-Aggregate Data using External Sort
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Each individual pipeline stage has :ref:`a limit of 100 megabytes of RAM
-<agg-memory-restrictions>`. By default, if a stage exceeds this limit,
-MongoDB produces an error. To allow pipeline processing to take up
-more space, set the :ref:`allowDiskUse <aggregate-cmd-allowDiskUse>`
-option to ``true`` to enable writing data to temporary files, as in the
-following example:
-
-.. code-block:: javascript
+Interaction with ``allowDiskUseByDefault``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-   db.stocks.aggregate( [
-         { $sort : { cusip : 1, date: 1 } }
-      ],
-      { allowDiskUse: true }
-   )
+.. include:: /includes/fact-allowDiskUseByDefault.rst
 
 .. include:: /includes/extracts/4.2-changes-usedDisk.rst
 

source/reference/command/find.txt

@@ -378,13 +378,7 @@ The command accepts the following fields:
 
      - Optional. 
 
-       Use ``allowDiskUse`` to allow MongoDB to use temporary files on
-       disk to store data exceeding the 100 megabyte memory limit while
-       processing a non-indexed ("blocking") sort operation. If MongoDB
-       requires using more than 100 megabytes of memory for a blocking
-       sort operation, MongoDB returns an error *unless* the query
-       specifies ``allowDiskUse``. See :ref:`sort-index-use` for more
-       information on blocking sort operations.
+       .. include:: /includes/fact-allowDiskUse-option-6.0.rst
 
        ``allowDiskUse`` has no effect if MongoDB can satisfy the
        specified :ref:`sort <find-cmd-sort>` using an index, *or* if the

source/reference/method/cursor.allowDiskUse.txt

@@ -20,12 +20,11 @@ Definition
 
    .. include:: /includes/fact-mongosh-shell-method.rst
 
-   :method:`~cursor.allowDiskUse()` allows MongoDB to use temporary
-   files on disk to store data exceeding the 100 megabyte system memory
-   limit while processing a blocking sort operation. If MongoDB requires
-   using more than 100 megabytes of system memory for the blocking sort
-   operation, MongoDB returns an error *unless* the query specifies
-   :method:`cursor.allowDiskUse()`. 
+   Use :method:`~cursor.allowDiskUse()` to either allow or prohibit 
+   writing temporary files on disk when a pipeline stage exceeds 
+   the 100 megabyte limit. Starting in MongoDB 6.0, operations that 
+   require greater than 100 megabytes of memory automatically write 
+   data to temporary files by default.
 
    :method:`~cursor.allowDiskUse()` has the following form:
 
@@ -39,6 +38,33 @@ operations.
 Behavior
 --------
 
+Interaction with ``allowDiskUseByDefault``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Starting in MongoDB 6.0, pipeline stages that require more than 100 
+megabytes of memory to execute write temporary files to disk by 
+default. 
+
+.. note:: 
+
+   The following only applies to the legacy ``mongo`` shell in MongoDB
+   v6.0. In previous versions of the legacy ``mongo`` shell, 
+   ``.allowDiskUse(false)`` and ``.allowDiskUse(true)`` have the same 
+   effect.
+
+If :parameter:`allowDiskUseByDefault` is ``true`` (this is the default):
+
+- ``.allowDiskUse()`` has no additional effect 
+- ``.allowDiskUse(true)`` has no additional effect
+- ``.allowDiskUse(false)`` prohibits the query from writing temporary
+  files to disk
+
+If :parameter:`allowDiskUseByDefault` is ``false``:
+
+- ``.allowDiskUse()`` enables writing temporary files to disk
+- ``.allowDiskUse(true)`` enables writing temporary files to disk
+- ``.allowDiskUse(false)`` has no additional effect
+
 Supports Large Non-Indexed Sorts Only
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

source/reference/method/db.collection.aggregate.txt

@@ -404,26 +404,10 @@ You can view more verbose explain output by passing the
 
 .. _example-aggregate-method-external-sort:
 
-Perform Large Sort Operation with External Sort
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Interaction with ``allowDiskUseByDefault``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Each individual pipeline stage has :ref:`a limit of 100 megabytes of RAM
-<agg-memory-restrictions>`. By default, if a stage exceeds this limit,
-MongoDB produces an error. To allow pipeline processing to take up
-more space, set the :ref:`allowDiskUse <aggregate-cmd-allowDiskUse>`
-option to ``true`` to enable writing data to temporary files, as in the
-following example:
-
-.. code-block:: javascript
-
-   var results = db.stocks.aggregate(
-     [
-       { $sort : { cusip : 1, date: 1 } }
-     ],
-     {
-       allowDiskUse: true
-     }
-   )
+.. include:: /includes/fact-allowDiskUseByDefault.rst
 
 .. include:: /includes/extracts/4.2-changes-usedDisk.rst
 

source/reference/operator/aggregation/sort.txt

@@ -209,12 +209,10 @@ documents. See :expression:`$meta` for more information.
 ``$sort`` and Memory Restrictions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The :pipeline:`$sort` stage has a limit of 100 megabytes of RAM for
-in-memory sorts. By default, if the stage exceeds this limit,
-:pipeline:`$sort` produces an error. To allow pipeline processing to
-take up more space, use the :ref:`allowDiskUse
-<aggregate-cmd-allowDiskUse>` option to enable aggregation pipeline
-stages to write data to temporary files.
+``$sort`` is subject to the 100 megabyte memory usage limit, but is 
+able to write temporary files to disk if additional space is required.
+
+.. include:: /includes/fact-allowDiskUseByDefault.rst
 
 .. seealso::
 

source/reference/operator/aggregation/sortByCount.txt

@@ -81,14 +81,14 @@ Considerations
 
 .. _sortbycount-memory-limit:
 
-``$count`` and Memory Restrictions
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The :pipeline:`$sortByCount` stage has a limit of 100 megabytes of RAM.
-By default, if the stage exceeds this limit, :pipeline:`$sortByCount`
-returns an error. To allow more space for stage processing, use the
-:ref:`allowDiskUse <aggregate-cmd-allowDiskUse>` option to enable
-aggregation pipeline stages to write data to temporary files.
+``$sortByCount`` and Memory Restrictions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``$sortByCount`` is subject to the 100 megabyte memory usage limit, but 
+is able to write temporary files to disk if additional space is 
+required.
+
+.. include:: /includes/fact-allowDiskUseByDefault.rst
 
 .. seealso::