From 055589168892ec4318e96e7bee77fd85b0864ec9 Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 7 Jan 2013 14:56:29 -0500 Subject: [PATCH 1/4] DOCS-956 covered indexes and dot notation --- source/applications/indexes.txt | 36 +++++++++++++++++++++++++++++---- source/core/read-operations.txt | 10 ++++++--- 2 files changed, 39 insertions(+), 7 deletions(-) diff --git a/source/applications/indexes.txt b/source/applications/indexes.txt index cba3b90c2f9..a137bd01d42 100644 --- a/source/applications/indexes.txt +++ b/source/applications/indexes.txt @@ -134,10 +134,38 @@ the result set) must explicitly exclude the ``_id`` field from the result set, unless the index includes ``_id``. -MongoDB cannot use a covered query if any of the indexed fields in any -of the documents in the collection includes an array. If an indexed field -is an array, the index becomes a :ref:`multi-key index -` index and cannot support a covered query. +MongoDB **cannot** use a covered query if: + +- any of the indexed fields in any of the documents in the collection + includes an array. If an indexed field is an array, the index becomes + a :ref:`multi-key index ` index and cannot + support a covered query. + +- any of the indexed fields uses the :term:`dot notation`. For example, + consider a collection ``users`` with documents of the following form: + + .. code-block:: javascript + + { _id: 1, user: { login: "tester" } } + + The collection has the following indexex: + + .. code-block:: javascript + + { user: 1 } + { "user.login": 1 } + + The following query is covered: + + .. code-block:: javascript + + db.users.find( { user: { login: "tester" } }, { user: 1, _id: 0 } ) + + However, the following query is **not** covered: + + .. code-block:: javascript + + db.users.find( { "user.login": "tester" }, { "user.login": 1, _id: 0 } ) To test whether MongoDB used a covered query, use :method:`explain() `. If the output displays ``true`` diff --git a/source/core/read-operations.txt b/source/core/read-operations.txt index 563f7f0ee6b..a1262c1e466 100644 --- a/source/core/read-operations.txt +++ b/source/core/read-operations.txt @@ -354,9 +354,13 @@ Result Projections ~~~~~~~~~~~~~~~~~~ The :term:`projection` specification limits the fields to return for -all matching documents. Constraining the result set by restricting the -fields to return can minimize network transit costs and the costs of -deserializing documents in the application layer. +all matching documents. Restricting the fields to return can minimize +network transit costs and the costs of deserializing documents in the +application layer. Additionally, if all the return fields are part of +an :ref:`index ` and the :ref:`query +` uses that same index, MongoDB can +fulfill the query using the :ref:`index only ` +which is faster than querying whole documents. The second argument to the :method:`find() ` method is a projection, and it takes the form of a :term:`document` with From a85fc5ba4bc90eab9e39d881963e07a568a94cdc Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 14 Jan 2013 01:42:39 -0500 Subject: [PATCH 2/4] DOCS-956 covered index and subdocument fields --- source/applications/indexes.txt | 69 ++++++++++++++++++++------------- source/core/read-operations.txt | 50 ++++++++++++++++++++---- 2 files changed, 83 insertions(+), 36 deletions(-) diff --git a/source/applications/indexes.txt b/source/applications/indexes.txt index a137bd01d42..6c4fc2ff87b 100644 --- a/source/applications/indexes.txt +++ b/source/applications/indexes.txt @@ -117,60 +117,73 @@ can support all the queries that search a "prefix" subset of those fields. Create Indexes that Support Covered Queries ------------------------------------------- -A covered index query is a query in which all the queried fields are -part of an index. They are "covered queries" because an index "covers" the query. -MongoDB can fulfill the query by using *only* the index. MongoDB need -not scan documents from the database. +A covered query is a query in which all the queried fields and +the returned fields are part of an index; i.e. -Querying *only* the index is much faster than querying documents. Index -keys are typically smaller than the documents they catalog, and indexes -are typically stored in RAM or located sequentially on disk. +- all the fields in the :ref:`query ` + are part of that index, **and** -Mongod automatically uses a covered query when possible. To ensure use -of a covered query, create an index that includes all the fields listed -in the query result. This means that the :term:`projection` document -given to a query (to specify which fields MongoDB returns from -the result set) must -explicitly exclude the ``_id`` field from the result set, unless the -index includes ``_id``. +- all the fields returned in the results are in the same index. -MongoDB **cannot** use a covered query if: +Because the index "covers" the query, MongoDB can match the :ref:`query +conditions ` and return the results +using only the index. MongoDB does not need to not look at documents +outside of the index. + +Querying *only* the index can be much faster than querying documents +outside of the index. Index keys are typically smaller than the +documents they catalog, and indexes are typically available in RAM or +located sequentially on disk. + +MongoDB automatically uses a covered query when possible. To ensure the +use of a covered query, create an index that includes all the fields +listed in the query result and the :ref:`query document +`. This means that, if the index does +**not** include the ``_id`` field, the :term:`projection` document +given to a query (to specify which fields MongoDB returns from the +result set) must explicitly exclude the ``_id`` field from the result +set. + +An index **cannot** cover a query if: - any of the indexed fields in any of the documents in the collection includes an array. If an indexed field is an array, the index becomes a :ref:`multi-key index ` index and cannot support a covered query. -- any of the indexed fields uses the :term:`dot notation`. For example, - consider a collection ``users`` with documents of the following form: +- any of the indexed fields are fields in subdocuments. To index fields + in subdocuments, use :term:`dot notation`. For example, consider + a collection ``users`` with documents of the following form: .. code-block:: javascript { _id: 1, user: { login: "tester" } } - The collection has the following indexex: + The collection has the following indexes: - .. code-block:: javascript + .. code-block:: none { user: 1 } + { "user.login": 1 } - - The following query is covered: - .. code-block:: javascript + The ``{ user: 1 }`` index covers the following query: + + .. code-block:: none db.users.find( { user: { login: "tester" } }, { user: 1, _id: 0 } ) - However, the following query is **not** covered: + However, the ``{ "user.login": 1 }`` index does **not** cover the + following query: - .. code-block:: javascript + .. code-block:: none db.users.find( { "user.login": "tester" }, { "user.login": 1, _id: 0 } ) -To test whether MongoDB used a covered query, use -:method:`explain() `. If the output displays ``true`` -for the ``indexOnly`` field, MongoDB used a covered query. For -more information see :ref:`indexes-measuring-use`. +To determine whether a query is covered, use :method:`explain() +`. If the output displays ``true`` for the +``indexOnly`` field, the query is covered and MongoDB queried only the +index. For more information see :ref:`indexes-measuring-use`. .. _index-sort: .. _sorting-with-indexes: diff --git a/source/core/read-operations.txt b/source/core/read-operations.txt index a1262c1e466..023e5729ee9 100644 --- a/source/core/read-operations.txt +++ b/source/core/read-operations.txt @@ -356,11 +356,7 @@ Result Projections The :term:`projection` specification limits the fields to return for all matching documents. Restricting the fields to return can minimize network transit costs and the costs of deserializing documents in the -application layer. Additionally, if all the return fields are part of -an :ref:`index ` and the :ref:`query -` uses that same index, MongoDB can -fulfill the query using the :ref:`index only ` -which is faster than querying whole documents. +application layer. The second argument to the :method:`find() ` method is a projection, and it takes the form of a :term:`document` with @@ -462,6 +458,42 @@ operation: ` to support sort operations in compound queries. +.. _read-operations-covered-query: + +Covering a Query +~~~~~~~~~~~~~~~~ + +An index :ref:`covers ` a query when: + +- all the fields in the :ref:`query ` + are part of that index, **and** + +- all the fields returned in the documents that match the query are in + the same index. + +Consider the following example where the collection ``inventory`` has +an index on the fields ``type`` and ``item``. + +.. code-block:: sh + + { type: 1, item: 1 } + +Then a query on the ``inventory`` collection that queries on the +``type`` field and returns only the ``item`` field is covered by the +index: + +.. code-block:: javascript + + db.inventory.find( { type: "food" }, + { item: 1, _id: 0 } ) + +Because the index "covers" the query, MongoDB can match the :ref:`query +conditions ` and return the results +using only the index. MongoDB does not need to look at documents +outside of the index. Querying the index can be faster than querying +the documents outside of the index. See :ref:`indexes-covered-queries` +for more information, including when indexes cannot cover a query. + Measuring Index Use ~~~~~~~~~~~~~~~~~~~ @@ -556,9 +588,11 @@ the query used an index. This query: - then read 5 full documents from the collection, as indicated by the :data:`nscannedObjects` field. - This indicates that the query was not "covered," or able to complete - only using the index, as reflected in the :data:`indexOnly`. See - :ref:`indexes-covered-queries` for more information. + Although the query used an index, the ``indexOnly: false`` indicates + that the index did not :ref:`cover ` + the query; i.e. MongoDB could not match the :ref:`query conditions + ` and return the results using only + that index. See :ref:`indexes-covered-queries` for more information. .. index:: query optimizer .. _read-operations-query-optimization: From 64dc9966012028f4c90b46da3aa1bf1bfd80dc35 Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 14 Jan 2013 15:20:17 -0500 Subject: [PATCH 3/4] DOCS-956 indexes that cover queries --- source/applications/indexes.txt | 64 +++++++++++++++++++++++---------- source/core/indexes.txt | 20 ++++++++--- source/core/read-operations.txt | 46 +++++++++++++++--------- source/reference/explain.txt | 20 +++++++---- 4 files changed, 104 insertions(+), 46 deletions(-) diff --git a/source/applications/indexes.txt b/source/applications/indexes.txt index 6c4fc2ff87b..32955937b5f 100644 --- a/source/applications/indexes.txt +++ b/source/applications/indexes.txt @@ -117,32 +117,52 @@ can support all the queries that search a "prefix" subset of those fields. Create Indexes that Support Covered Queries ------------------------------------------- -A covered query is a query in which all the queried fields and -the returned fields are part of an index; i.e. +A covered query is a query in which: - all the fields in the :ref:`query ` - are part of that index, **and** + are part of an index, **and** - all the fields returned in the results are in the same index. -Because the index "covers" the query, MongoDB can match the :ref:`query -conditions ` and return the results -using only the index. MongoDB does not need to not look at documents -outside of the index. +Because the index "covers" the query, MongoDB can both match the +:ref:`query conditions ` **and** return +the results using only the index; MongoDB does not need to look at +documents outside of the index. Querying *only* the index can be much faster than querying documents outside of the index. Index keys are typically smaller than the documents they catalog, and indexes are typically available in RAM or located sequentially on disk. -MongoDB automatically uses a covered query when possible. To ensure the -use of a covered query, create an index that includes all the fields -listed in the query result and the :ref:`query document -`. This means that, if the index does -**not** include the ``_id`` field, the :term:`projection` document -given to a query (to specify which fields MongoDB returns from the -result set) must explicitly exclude the ``_id`` field from the result -set. +MongoDB automatically uses an index that covers a query when possible. +To ensure that a query is covered, create an index that includes all +the fields listed in the query result and the :ref:`query document +`. This means that if the index does +**not** include the ``_id`` field, the :term:`projection` document, +which specifies the fields MongoDB returns, must explicitly exclude the +``_id`` field from the result set. + +Consider the following example where the collection ``user`` has +an index on the fields ``user`` and ``status``: + +.. code-block:: javascript + + { status: 1, user: 1 } + +Then, the following query which queries on the ``status`` field and +returns only the ``user`` field is covered: + +.. code-block:: javascript + + db.users.find( { status: "A" }, { user: 1, _id: 0 } ) + +However, the following query that uses the index to match documents is +**not** covered by the index because it returns both the ``user`` field +**and** the ``_id`` field: + +.. code-block:: javascript + + db.users.find( { status: "A" }, { user: 1 } ) An index **cannot** cover a query if: @@ -180,10 +200,16 @@ An index **cannot** cover a query if: db.users.find( { "user.login": "tester" }, { "user.login": 1, _id: 0 } ) -To determine whether a query is covered, use :method:`explain() -`. If the output displays ``true`` for the -``indexOnly`` field, the query is covered and MongoDB queried only the -index. For more information see :ref:`indexes-measuring-use`. + The query, however, does use the ``{ "user.login": 1 }`` index to + find matching documents. + +To determine whether a query is a covered query, use the +:method:`~cursor.explain()` method. If the :method:`~cursor.explain()` +output displays ``true`` for the :data:`indexOnly` field, the query is +covered by an index, and MongoDB queries only that index to match the +query **and** return the results. + +For more information see :ref:`indexes-measuring-use`. .. _index-sort: .. _sorting-with-indexes: diff --git a/source/core/indexes.txt b/source/core/indexes.txt index b994e85833c..44764d09230 100644 --- a/source/core/indexes.txt +++ b/source/core/indexes.txt @@ -46,11 +46,21 @@ MongoDB indexes have the following core features: with the best response time for each query type. You can override the query optimizer using the :method:`cursor.hint()` method. -- An index "covers" a query if the index keys store all the data that - the query must return. When an index covers a query, the database - returns results more quickly than for queries that have to scan many - individual documents. See :ref:`indexes-covered-queries` for more - information. +- An index "covers" a query if: + + - all the fields in the :ref:`query ` + are part of that index, **and** + + - all the fields returned in the documents that match the query are + in the same index. + + When an index covers a query, the database can both match the + :ref:`query conditions ` **and** + return the results using only the index; MongoDB does not need to + look at documents outside of the index. Querying the index can be + faster than querying the documents outside of the index. + + See :ref:`indexes-covered-queries` for more information. - Using queries with good index coverage reduces the number of full documents that MongoDB needs to store in memory, thus maximizing database diff --git a/source/core/read-operations.txt b/source/core/read-operations.txt index 023e5729ee9..79209d09421 100644 --- a/source/core/read-operations.txt +++ b/source/core/read-operations.txt @@ -463,7 +463,8 @@ operation: Covering a Query ~~~~~~~~~~~~~~~~ -An index :ref:`covers ` a query when: +An index :ref:`covers ` a query, or a query is +a "covered" query, when: - all the fields in the :ref:`query ` are part of that index, **and** @@ -471,6 +472,12 @@ An index :ref:`covers ` a query when: - all the fields returned in the documents that match the query are in the same index. +When an index "covers" the query, MongoDB can both match the +:ref:`query conditions ` **and** return +the results using only the index; MongoDB does not need to look at +documents outside of the index. Querying the index can be faster than +querying the documents outside of the index. + Consider the following example where the collection ``inventory`` has an index on the fields ``type`` and ``item``. @@ -478,21 +485,26 @@ an index on the fields ``type`` and ``item``. { type: 1, item: 1 } -Then a query on the ``inventory`` collection that queries on the -``type`` field and returns only the ``item`` field is covered by the -index: +Then the following query that queries on the ``type`` and ``item`` +fields and returns only the ``item`` field is covered by the ``{type: +1, item: 1}`` index: .. code-block:: javascript - db.inventory.find( { type: "food" }, + db.inventory.find( { type: "food", item:/^c/ }, { item: 1, _id: 0 } ) -Because the index "covers" the query, MongoDB can match the :ref:`query -conditions ` and return the results -using only the index. MongoDB does not need to look at documents -outside of the index. Querying the index can be faster than querying -the documents outside of the index. See :ref:`indexes-covered-queries` -for more information, including when indexes cannot cover a query. +However, the following query that uses the ``{ type: 1, item: 1 }`` index +to match documents is **not** covered by the index because the query +returns the ``item`` field **and** the ``_id`` field: + +.. code-block:: javascript + + db.inventory.find( { type: "food", item:/^c/ }, + { item: 1 } ) + +See :ref:`indexes-covered-queries` for more information, including when +indexes cannot cover a query. Measuring Index Use ~~~~~~~~~~~~~~~~~~~ @@ -588,11 +600,13 @@ the query used an index. This query: - then read 5 full documents from the collection, as indicated by the :data:`nscannedObjects` field. - Although the query used an index, the ``indexOnly: false`` indicates - that the index did not :ref:`cover ` - the query; i.e. MongoDB could not match the :ref:`query conditions - ` and return the results using only - that index. See :ref:`indexes-covered-queries` for more information. + Although the query uses an index to find the matching documents, the + :data:`indexOnly:false ` indicates that this index did + not :ref:`cover ` the query; i.e. + MongoDB could not both match the :ref:`query conditions + ` **and** return the results using + only this index. See :ref:`indexes-covered-queries` for more + information. .. index:: query optimizer .. _read-operations-query-optimization: diff --git a/source/reference/explain.txt b/source/reference/explain.txt index 817724a73a0..8d7e7bdebbd 100644 --- a/source/reference/explain.txt +++ b/source/reference/explain.txt @@ -155,7 +155,8 @@ Core Explain Output Specifies the total number of documents scanned during the query. The :data:`nscannedObjects` may be lower than :data:`nscanned`, such - as if the index is a covered index. + as if the index :ref:`covers ` a query. See + :data:`indexOnly`. .. data:: nscanned @@ -163,8 +164,8 @@ Core Explain Output during the database operation. You want :data:`n` and :data:`nscanned` to be close in value as possible. The :data:`nscanned` value may be higher than the - :data:`nscannedObjects` value, such as if the index is a covered - index. + :data:`nscannedObjects` value, such as if the index :ref:`covers + ` a query. See :data:`indexOnly`. .. data:: nscannedObjectsAllPlans @@ -194,9 +195,16 @@ Core Explain Output .. data:: indexOnly :data:`indexOnly` is a boolean value that returns ``true`` when the - query is :ref:`covered ` by the index. In - *covered* queries, the index contains all data that MongoDB needs - to fulfill the query. + the query is :ref:`covered ` by the index + indicated in the :data:`cursor` field. When an index covers a query, + MongoDB can both match the :ref:`query conditions + ` **and** return the results using + only the index because: + + - all the fields in the :ref:`query + ` are part of that index, **and** + + - all the fields returned in the results set are in the same index. .. data:: nYields From 8c04f8bdb62c1d4146562ba28e8f300774c1ab3c Mon Sep 17 00:00:00 2001 From: kay Date: Mon, 11 Feb 2013 22:52:19 -0500 Subject: [PATCH 4/4] DOCS-956 covered index --- source/applications/indexes.txt | 2 +- source/core/indexes.txt | 7 ++++--- source/reference/explain.txt | 4 +++- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/source/applications/indexes.txt b/source/applications/indexes.txt index 32955937b5f..0284f2e7f6e 100644 --- a/source/applications/indexes.txt +++ b/source/applications/indexes.txt @@ -127,7 +127,7 @@ A covered query is a query in which: Because the index "covers" the query, MongoDB can both match the :ref:`query conditions ` **and** return the results using only the index; MongoDB does not need to look at -documents outside of the index. +the documents, only the index, to fulfill the query. Querying *only* the index can be much faster than querying documents outside of the index. Index keys are typically smaller than the diff --git a/source/core/indexes.txt b/source/core/indexes.txt index 44764d09230..83da668f21a 100644 --- a/source/core/indexes.txt +++ b/source/core/indexes.txt @@ -54,11 +54,12 @@ MongoDB indexes have the following core features: - all the fields returned in the documents that match the query are in the same index. - When an index covers a query, the database can both match the + When an index covers a query, the server can both match the :ref:`query conditions ` **and** return the results using only the index; MongoDB does not need to - look at documents outside of the index. Querying the index can be - faster than querying the documents outside of the index. + look at the documents, only the index, to fulfill the query. + Querying the index can be faster than querying the documents outside + of the index. See :ref:`indexes-covered-queries` for more information. diff --git a/source/reference/explain.txt b/source/reference/explain.txt index 8d7e7bdebbd..6f2273ecc84 100644 --- a/source/reference/explain.txt +++ b/source/reference/explain.txt @@ -156,7 +156,9 @@ Core Explain Output Specifies the total number of documents scanned during the query. The :data:`nscannedObjects` may be lower than :data:`nscanned`, such as if the index :ref:`covers ` a query. See - :data:`indexOnly`. + :data:`indexOnly`. Additionally, the :data:`nscannedObjects` may be + lower than :data:`nscanned` in the case of multikey index on an + array field with duplicate documents. .. data:: nscanned