Skip to content

DOCS-14048 mapReduce refactoring #4816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 10 additions & 25 deletions source/aggregation.txt
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ transforms the documents into an aggregated result. For example:

<video style="width:100%;" src="_images/agg-pipeline.mp4" controls> </video>

In the example,
In the example:

.. code-block:: javascript

Expand Down Expand Up @@ -82,28 +82,6 @@ internal optimization phase. See
:ref:`aggregation-pipeline-operators-and-performance` and
:doc:`/core/aggregation-pipeline-optimization` for details.

.. _aggregation-map-reduce:

Map-Reduce
----------

.. tip::

:doc:`Aggregation pipeline </core/aggregation-pipeline>`
provides better performance and a more coherent interface than
map-reduce.

For examples of aggregation alternatives to map-reduce operations,
see :doc:`/tutorial/map-reduce-examples`. See also
:doc:`/reference/map-reduce-to-aggregation-pipeline`.

MongoDB also provides :doc:`map-reduce </core/map-reduce>` operations
to perform aggregation. Map-reduce uses custom JavaScript functions to
perform the map and reduce operations, as well as the optional
*finalize* operation.

.. include:: /images/map-reduce.rst

.. _single-purpose-agg-operations:

Single Purpose Aggregation Operations
Expand All @@ -114,11 +92,18 @@ MongoDB also provides :method:`db.collection.estimatedDocumentCount()`,

All of these operations aggregate documents from a single collection.
While these operations provide simple access to common aggregation
processes, they lack the flexibility and capabilities of the
aggregation pipeline and map-reduce.
processes, they lack the flexibility and capabilities of an aggregation
pipeline.

.. include:: /images/distinct.rst

.. _aggregation-map-reduce:

Map-Reduce
----------

.. include:: /includes/fact-use-aggregation-not-map-reduce.rst

Additional Features and Behaviors
---------------------------------

Expand Down
60 changes: 24 additions & 36 deletions source/core/aggregation-pipeline.txt
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ results. For example:

<video style="width:100%;" src="../_images/agg-pipeline.mp4" controls> </video>

In the example,
In the example:

.. code-block:: javascript

Expand All @@ -53,8 +53,8 @@ Pipeline

The MongoDB aggregation pipeline consists of :ref:`stages
<aggregation-pipeline-operator-reference>`. Each stage transforms the
documents as they pass through the pipeline. Pipeline stages do not
need to produce one output document for every input document; e.g.,
documents as they pass through the pipeline. Pipeline stages do not need
to produce one output document for every input document. For example,
some stages may generate new documents or filter out documents.

Pipeline stages can appear multiple times in the pipeline with the
Expand Down Expand Up @@ -98,11 +98,11 @@ by the aggregation process with one exception: :ref:`accumulator
<aggregation-accumulator-operators>` expressions.

The accumulators, used in the :pipeline:`$group` stage, maintain their
state (e.g. totals, maximums, minimums, and related data) as documents
progress through the pipeline. Some accumulators are available in the
:pipeline:`$project` stage; however, when used in the
:pipeline:`$project` stage, the accumulators do not maintain their
state across documents.
state (for example, totals, maximums, minimums, and related data) as
documents progress through the pipeline. Some accumulators are available
in the :pipeline:`$project` stage; however, when used in the
:pipeline:`$project` stage, the accumulators do not maintain their state
across documents.

Starting in version 4.4, MongoDB provides the :group:`$accumulator` and
:expression:`$function` aggregation operators. These operators provide
Expand Down Expand Up @@ -194,41 +194,29 @@ pipeline.
Considerations
--------------

Sharded Collections
~~~~~~~~~~~~~~~~~~~~
Aggregation Pipeline Limitations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The aggregation pipeline supports operations on sharded collections.
See :ref:`aggregation-pipeline-sharded-collection`.

Aggregation Pipeline vs Map-Reduce
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An aggregation pipeline has some limitations on the value types and
the result size. See :doc:`/core/aggregation-pipeline-limits`.

The aggregation pipeline provides better performance and a more coherent
interface than :doc:`map-reduce </core/map-reduce>`.
Aggregation Pipeline Optimization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Various map-reduce operations can be rewritten using :doc:`aggregation
pipeline operators </meta/aggregation-quick-reference>`, such as
:pipeline:`$group`, :pipeline:`$merge`, etc. For map-reduce operations
that require custom functionality, MongoDB provides the
:group:`$accumulator` and :expression:`$function` aggregation operators
starting in version 4.4. These operators provide users with the ability
to define custom aggregation expressions in JavaScript.
An aggregation pipeline has an internal optimization phase that provides
improved performance for certain sequences of operators. See
:doc:`/core/aggregation-pipeline-optimization`.

See :doc:`/tutorial/map-reduce-examples` for details.

Limitations
~~~~~~~~~~~
Aggregation on Sharded Collections
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Aggregation pipeline have some limitations on value types and result
size. See :doc:`/core/aggregation-pipeline-limits` for details on
limits and restrictions on the aggregation pipeline.
An aggregation pipeline supports operations on sharded collections.
See :ref:`aggregation-pipeline-sharded-collection`.

Pipeline Optimization
~~~~~~~~~~~~~~~~~~~~~
Aggregation Pipeline as an Alternative to Map-Reduce
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The aggregation pipeline has an internal optimization phase that
provides improved performance for certain sequences of operators. For
details, see :doc:`/core/aggregation-pipeline-optimization`.
.. include:: /includes/fact-use-aggregation-not-map-reduce.rst

.. toctree::
:titlesonly:
Expand Down
5 changes: 5 additions & 0 deletions source/core/map-reduce-concurrency.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ Map-Reduce Concurrency
:depth: 1
:class: singlecol

.. admonition:: Aggregation Pipeline as an Alternative to Map-Reduce
:class: note

.. include:: /includes/fact-use-aggregation-not-map-reduce.rst

The map-reduce operation is composed of many tasks, including reads
from the input collection, executions of the ``map`` function,
executions of the ``reduce`` function, writes to a temporary collection
Expand Down
5 changes: 5 additions & 0 deletions source/core/map-reduce-sharded-collections.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ Map-Reduce and Sharded Collections
:depth: 1
:class: singlecol

.. admonition:: Aggregation Pipeline as an Alternative to Map-Reduce
:class: note

.. include:: /includes/fact-use-aggregation-not-map-reduce.rst

Map-reduce supports operations on sharded collections, both as an input
and as an output. This section describes the behaviors of
:dbcommand:`mapReduce` specific to sharded collections.
Expand Down
13 changes: 2 additions & 11 deletions source/core/map-reduce.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,10 @@ Map-Reduce
:depth: 1
:class: singlecol

.. admonition:: Aggregation Pipeline as Alternative
.. admonition:: Aggregation Pipeline as Alternative to Map-Reduce
:class: note

:doc:`Aggregation pipeline </core/aggregation-pipeline>`
provides better performance and a more coherent interface than
map-reduce, and various map-reduce operations can be rewritten
using :doc:`aggregation pipeline operators
</meta/aggregation-quick-reference>`, such as :pipeline:`$group`,
:pipeline:`$merge`, :group:`$accumulator`, etc.

For examples of aggregation alternatives to map-reduce operations,
see :doc:`/tutorial/map-reduce-examples`. See also
:doc:`/reference/map-reduce-to-aggregation-pipeline`.
.. include:: /includes/fact-use-aggregation-not-map-reduce.rst

Map-reduce is a data processing paradigm for condensing large volumes
of data into useful *aggregated* results. To perform map-reduce
Expand Down
23 changes: 4 additions & 19 deletions source/includes/examples-map-reduce.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,10 @@ Map-Reduce Examples
.. map-reduce-document-examples-begin
.. map-reduce-document-prototype-begin

.. admonition:: Aggregation Pipeline as Alternative
:class: note

:doc:`Aggregation pipeline </core/aggregation-pipeline>` provides
better performance and a simpler interface than map-reduce, and
map-reduce expressions can be rewritten using :doc:`aggregation
pipeline operators </meta/aggregation-quick-reference>` such as
:pipeline:`$group`, :pipeline:`$merge`, and others.

For map-reduce expressions that require custom functionality, MongoDB
provides the :group:`$accumulator` and :expression:`$function`
aggregation operators starting in version 4.4. These operators
provide the ability to define custom aggregation expressions in
JavaScript.

The examples in this section include aggregation pipeline
alternatives without custom aggregation expressions. For alternatives
that use custom expressions, see :ref:`Map-Reduce to Aggregation
Pipeline Translation Examples <mr-to-agg-examples>`.
The examples in this section include aggregation pipeline alternatives
without custom aggregation expressions. For alternatives that use custom
expressions, see :ref:`Map-Reduce to Aggregation Pipeline Translation
Examples <mr-to-agg-examples>`.

Create a sample collection ``orders`` with these documents:

Expand Down
16 changes: 16 additions & 0 deletions source/includes/fact-use-aggregation-not-map-reduce.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
An :doc:`aggregation pipeline </core/aggregation-pipeline>` provides
better performance and usability than a :doc:`map-reduce
</core/map-reduce>` operation.

Map-reduce operations can be rewritten using :doc:`aggregation pipeline
operators </meta/aggregation-quick-reference>`, such as
:pipeline:`$group`, :pipeline:`$merge`, and others.

For map-reduce operations that require custom functionality, MongoDB
provides the :group:`$accumulator` and :expression:`$function`
aggregation operators starting in version 4.4. Use these operators to
define custom aggregation expressions in JavaScript.

For examples of aggregation pipeline alternatives to map-reduce
operations, see :doc:`/reference/map-reduce-to-aggregation-pipeline` and
:doc:`/tutorial/map-reduce-examples`.
16 changes: 4 additions & 12 deletions source/reference/aggregation-commands-comparison.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,10 @@ Aggregation Commands Comparison
:depth: 1
:class: singlecol

.. tip::
.. admonition:: Aggregation Pipeline as Alternative to Map-Reduce
:class: note

Starting in version 4.4, MongoDB adds the :group:`$accumulator` and
:expression:`$function` aggregation operators. Using
:group:`$accumulator` and :expression:`$function` ,
:dbcommand:`mapReduce` expressions can be re-written using the
aggregation operators.

Even before version 4.4, some map-reduce expressions could also be
rewritten using :doc:`other aggregation pipeline operators
</meta/aggregation-quick-reference>`, such as :pipeline:`$group`,
:pipeline:`$merge`, etc.
.. include:: /includes/fact-use-aggregation-not-map-reduce.rst

Aggregation Commands Comparison Table
-------------------------------------
Expand All @@ -47,7 +39,7 @@ MongoDB aggregation commands.

Uses a "pipeline" approach where objects are transformed as they
pass through a series of pipeline operators such as
:pipeline:`$group`, :pipeline:`$match`, and :pipeline:`$sort`.
:pipeline:`$group`, :pipeline:`$match`, and :pipeline:`$sort`.

See :doc:`/reference/operator/aggregation` for more information
on the pipeline operators.
Expand Down
26 changes: 3 additions & 23 deletions source/reference/command/mapReduce.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ mapReduce
:depth: 1
:class: singlecol

.. admonition:: Deprecated
:class: warning
.. admonition:: Aggregation Pipeline as Alternative to Map-Reduce
:class: note

.. include:: /includes/5.0-mapReduce-deprecated.rst
.. include:: /includes/fact-use-aggregation-not-map-reduce.rst

Definition
----------
Expand All @@ -22,26 +22,6 @@ Definition

The :dbcommand:`mapReduce` command allows you to run
:term:`map-reduce` aggregation operations over a collection.

.. admonition:: Aggregation Pipeline as Alternative
:class: note

:doc:`Aggregation pipeline </core/aggregation-pipeline>`
provides better performance and a simpler interface than
map-reduce, and map-reduce expressions can be
rewritten using :doc:`aggregation pipeline operators
</meta/aggregation-quick-reference>`, such as :pipeline:`$group`,
:pipeline:`$merge`, and so on.

For map-reduce expressions that require custom functionality,
MongoDB provides the :group:`$accumulator` and
:expression:`$function` aggregation operators starting in version
4.4. These operators provide users with the ability to define custom
aggregation expressions in JavaScript.

For examples of aggregation alternatives to map-reduce operations,
see :doc:`/tutorial/map-reduce-examples`. See also
:doc:`/reference/map-reduce-to-aggregation-pipeline`.

Syntax
------
Expand Down
29 changes: 15 additions & 14 deletions source/reference/map-reduce-to-aggregation-pipeline.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,21 @@ Map-Reduce to Aggregation Pipeline

.. default-domain:: mongodb

Starting in version 4.4, MongoDB adds the :group:`$accumulator` and
:expression:`$function` aggregation operators. These operators provide
users the ability to define custom aggregation expressions. Using these
operations, the map-reduce expressions can be approximately re-written
as in the following table.
An :doc:`aggregation pipeline </core/aggregation-pipeline>` provides
better performance and usability than a :doc:`map-reduce
</core/map-reduce>` operation.

.. note::
Map-reduce operations can be rewritten using :doc:`aggregation pipeline
operators </meta/aggregation-quick-reference>`, such as
:pipeline:`$group`, :pipeline:`$merge`, and others.

Various map-reduce expressions can be rewritten using
:doc:`aggregation pipeline operators
</meta/aggregation-quick-reference>`, such as :pipeline:`$group`,
:pipeline:`$merge`, etc., without requiring custom functions.
For map-reduce operations that require custom functionality, MongoDB
provides the :group:`$accumulator` and :expression:`$function`
aggregation operators starting in version 4.4. Use these operators to
define custom aggregation expressions in JavaScript.

For examples, see :doc:`/tutorial/map-reduce-examples`.
Map-reduce expressions can be re-written as shown in the following
sections.

Map-Reduce to Aggregation Pipeline Translation Table
----------------------------------------------------
Expand Down Expand Up @@ -250,9 +251,9 @@ Examples

Various map-reduce expressions can be rewritten using :doc:`aggregation
pipeline operators </meta/aggregation-quick-reference>`, such as
:pipeline:`$group`, :pipeline:`$merge`, etc., without requiring custom
functions. However, for illustrative purposes, the following examples
provide both alternatives.
:pipeline:`$group`, :pipeline:`$merge`, and others, without requiring
custom functions. However, for illustrative purposes, the following
examples provide both alternatives.

.. _mr-to-agg-examples1:

Expand Down
Loading