From f193d13895ec7c592c40889b122982043e6649df Mon Sep 17 00:00:00 2001 From: Dave Cuthbert Date: Fri, 23 Jul 2021 10:01:31 -0400 Subject: [PATCH] DOCSP-15956 reintroduce dotDollar validation --- source/core/crud.txt | 8 +- source/core/document.txt | 15 +- source/core/dot-dollar-considerations.txt | 228 ++++++++++++++++++ .../fact-document-field-name-restrictions.rst | 18 +- source/includes/fact-id-field-name-rules.rst | 5 + .../includes/warning-possible-data-loss.rst | 20 ++ source/reference/command/findAndModify.txt | 5 +- source/reference/limits.txt | 6 + .../operator/aggregation/getField.txt | 5 + .../operator/aggregation/setField.txt | 4 + source/release-notes/5.0.txt | 10 + 11 files changed, 300 insertions(+), 24 deletions(-) create mode 100644 source/core/dot-dollar-considerations.txt create mode 100644 source/includes/fact-id-field-name-rules.rst create mode 100644 source/includes/warning-possible-data-loss.rst diff --git a/source/core/crud.txt b/source/core/crud.txt index 8392d81cf79..c961c5f3aac 100644 --- a/source/core/crud.txt +++ b/source/core/crud.txt @@ -26,6 +26,7 @@ Query Plan, Performance, and Analysis Miscellaneous - :doc:`/core/tailable-cursors` + - :doc:`/core/dot-dollar-considerations` .. seealso:: @@ -34,10 +35,11 @@ Miscellaneous .. toctree:: :titlesonly: + /tutorial/analyze-query-plan /core/write-operations-atomicity - /core/read-isolation-consistency-recency /core/distributed-queries - /core/query-plans + /core/dot-dollar-considerations + /core/read-isolation-consistency-recency /core/query-optimization - /tutorial/analyze-query-plan + /core/query-plans /core/tailable-cursors diff --git a/source/core/document.txt b/source/core/document.txt index 5433d844725..b0761476f9c 100644 --- a/source/core/document.txt +++ b/source/core/document.txt @@ -75,9 +75,7 @@ Field names are strings. :doc:`Documents ` have the following restrictions on field names: -- The field name ``_id`` is reserved for use as a primary key; its - value must be unique in the collection, is immutable, and may be of - any type other than an array. +- .. include:: /includes/fact-id-field-name-rules.rst .. include:: /includes/fact-document-field-name-restrictions.rst @@ -95,7 +93,6 @@ existing user document. Field Value Limit ~~~~~~~~~~~~~~~~~ - MongoDB 2.6 through MongoDB versions with :ref:`featureCompatibilityVersion ` (fCV) set to ``"4.0"`` or earlier For :doc:`indexed collections `, the values for the indexed fields have a :limit:`Maximum Index Key Length `. See @@ -142,7 +139,7 @@ For examples querying arrays, see: .. seealso:: - :update:`$[]` all positional operator for update operations, - + - :update:`$[\]` filtered positional operator for update operations, - :update:`$` positional operator for update operations, @@ -188,6 +185,7 @@ For examples querying embedded documents, see: - :doc:`/tutorial/query-array-of-documents/` + Document Limitations -------------------- @@ -220,6 +218,9 @@ The ``_id`` field has the following behavior and constraints: server receives a document that does not have the ``_id`` field first, then the server will move the field to the beginning. +_ If the ``_id`` contains subfields, the subfield names cannot begin + with a (``$``) symbol. + - The ``_id`` field may contain values of any :doc:`BSON data type `, other than an array, regex, or undefined. @@ -286,7 +287,7 @@ For examples, see: - :doc:`/tutorial/query-documents` - :doc:`/tutorial/query-embedded-documents` - + - :doc:`/tutorial/query-arrays` - :doc:`/tutorial/query-array-of-documents/` @@ -327,7 +328,7 @@ type: Further Reading --------------- -For more information on the MongoDB document model, download the +For more information on the MongoDB document model, download the `MongoDB Application Modernization Guide `_. diff --git a/source/core/dot-dollar-considerations.txt b/source/core/dot-dollar-considerations.txt new file mode 100644 index 00000000000..7e841066ca9 --- /dev/null +++ b/source/core/dot-dollar-considerations.txt @@ -0,0 +1,228 @@ +==================================== +Field Names with (``.``) and (``$``) +==================================== + +.. default-domain:: mongodb + +.. contents:: On this page + :local: + :backlinks: none + :depth: 1 + :class: singlecol + +.. _crud-concepts-dot-dollar-considerations: + +Overview +-------- + +MongoDB 5.0 adds improved support for field names that are (``$``) +prefixed or that contain (``.``) characters. The validation rules for +storing data have been updated to make it easier to work with data +sources that use these characters. + +In most cases data that has been stored using field names like these +is not directly accessible. You need to use helper methods like +:expression:`$getField`, :expression:`$setField`, and +:expression:`$literal` in queries that access those fields. + +The field name validation rules are not the same for all types of +storage operations. This page summarizes how different insert and +update operations handle (``$``) prefixed field names. + +Insert operations +----------------- + +(``$``) prefixed fields are permitted as top level and nested field +names for inserts. + +.. code-block:: javascript + :emphasize-lines: 3 + + db.sales.insertOne( { + "$price": 50.00, + "quantity": 30 + } ) + +(``$``) prefixed fields are permitted on inserts using otherwise +reserved words. Operator names like :update:`$inc` can be used as +field names as well as words like ``id``, ``db``, and ``ref``. + +.. code-block:: javascript + :emphasize-lines: 2, 4-6 + + db.books.insertOne( { + "$id": "h1961-01", + "location": { + "$db": "novels", + "$ref": "2007042768", + "$inc": true + } } ) + +An update which creates a new document during an :term:`upsert` is +treated as an ``insert`` rather than an ``update`` for field name +validation. :term:`Upserts ` can accept (``$``) prefixed +fields. However, :term:`upserts ` are a special case and +similar update operations may cause an error if the ``match`` portion +of the update selects an existing document. + +This code sample has ``upsert: true`` so it will insert a new document +if the collection doesn't already contain a document that matches the +query term, ``{ "date": "2021-07-07" }``. If this sample code matches +an existing document, the update will fail since ``$hotel`` is (``$``) +prefixed. + +.. code-block:: javascript + :emphasize-lines: 5 + + db.expenses.updateOne( + { "date": "2021-07-07" }, + { $set: { + "phone": 25.17, + "$hotel": 320.10 + } }, + { upsert: true } + ) + +Document Replacing Updates +-------------------------- + +Update operators either replace existing fields with new documents +or else modify those fields. In cases where the update performs a +replacement, (``$``) prefixed fields are not permitted as top level +field names. + +Consider a document like + +.. code-block:: javascript:: + + { + "_id": "E123", + "address": { + "$number": 123, + "$street": "Elm Road" + }, + "$rooms": { + "br": 2, + "bath": 1 + } + } + +You could use an update operator that replaces an existing document to +modify the ``address.$street`` field but you could not update the +``$rooms`` field that way. + +.. code-block:: + + db.housing.updateOne( + { "_id": "E123" }, + { $set: { "address.$street": "Elm Ave" } } + ) + +Use :expression:`$setField` as part of an aggregation pipeline to +:ref:`update top level ` (``$``) prefixed +fields like ``$rooms``. + +Document Modifying Updates +-------------------------- + +When an update modifies, rather than replaces, existing document +fields, (``$``) prefixed fields can be top level field names. +Subfields can be accessed directly, but you need a helper method to +access the top level fields. + +.. seealso:: + + :expression:`$getField`, :expression:`$setField`, + :expression:`$literal`, :pipeline:`$replaceWith` + +Consider a collection with documents like this inventory record: + +.. code-block:: + :copyable: false + + { + _id: ObjectId("610023ad7d58ecda39b8d161"), + "part": "AB305", + "$bin": 200, + "quantity": 100, + "pricing": { sale: true, "$discount": 60 } + } + +The ``pricing.$discount`` subfield can be queried directly. + +.. code-block:: + + db.inventory.findAndModify( { + query: { "part": { $eq: "AB305" } }, + update: { $inc: { "pricing.$discount": 10 } } + } ) + + +Use :expression:`$getField` and :expression:`$literal` to access the +value of the top level ``$bin`` field. + +.. code-block:: + :emphasize-lines: 3 + + db.inventory.findAndModify( { + query: { $expr: { + $eq: [ { $getField: { $literal: "$bin" } }, 200 ] + } }, + update: { $inc: { "quantity": 10 } } + } ) + +.. _dotDollar-aggregate-update: + +Updates Using Aggregation Pipelines +----------------------------------- + +Use :expression:`$setField`, :expression:`$getField`, and +:expression:`$literal` in the :pipeline:`$replaceWith` stage to modify +(``$``) prefixed fields in an aggregation :term:`pipeline`. + +Consider a collection of school records like: + +.. code-block:: javascript + :copyable: false + + { + "_id": 100001, + "$term": "fall", + "registered": true, + "grade": 4 + } + +Create a new collection for the spring semester using a +:term:`pipeline` to update the (``$``) prefixed ``$term`` field. + +.. code-block:: javascript + :emphasize-lines: 3-5 + + db.school.aggregate( [ + { $match: { "registered": true } }, + { $replaceWith: { + $setField: { + field: { $literal: "$term" }, + input: "$$ROOT", + value: "spring" + } } }, + { $out: "spring2022" } + ] ) + +General Restrictions +-------------------- + +In addition to the storage validation rules above, there are some +general restrictions on using (``$``) prefixed field names. These +fields cannot: + +- Be indexed +- Be used as part of a shard key +- Be validated using :query:`$jsonSchema` +- Be be modified with an escape sequence +- Be used with + :driver:`Field Level Encryption ` +- Be used as a subfield in an ``_id`` document + +.. include:: /includes/warning-possible-data-loss.rst + diff --git a/source/includes/fact-document-field-name-restrictions.rst b/source/includes/fact-document-field-name-restrictions.rst index b493252ac14..59e0467255f 100644 --- a/source/includes/fact-document-field-name-restrictions.rst +++ b/source/includes/fact-document-field-name-restrictions.rst @@ -1,17 +1,9 @@ - Field names **cannot** contain the ``null`` character. -- Top-level field names **cannot** start with the dollar sign (``$``) character. +- The server permits storage of field names that contain dots (``.``) + and dollar signs (``$``). - Otherwise, starting in MongoDB 3.6, the server permits storage of - field names that contain dots (i.e. ``.``) and dollar signs (i.e. - ``$``). +- MongodB 5.0 adds improved support for the use of (``$``) and (``.``) + in field names. There are some restrictions. See + :ref:`Field Name Considerations ` for more details. - .. important:: - - The MongoDB Query Language cannot always meaningfully express - queries over documents whose field names contain these characters - (see :issue:`SERVER-30575`). - - Until support is added in the query language, the use of ``$`` and - ``.`` in field names is not recommended and is not supported by - the official MongoDB drivers. diff --git a/source/includes/fact-id-field-name-rules.rst b/source/includes/fact-id-field-name-rules.rst new file mode 100644 index 00000000000..81ad09fd7d3 --- /dev/null +++ b/source/includes/fact-id-field-name-rules.rst @@ -0,0 +1,5 @@ +The field name ``_id`` is reserved for use as a primary key; its value +must be unique in the collection, is immutable, and may be of any type +other than an array. If the ``_id`` contains subfields, the subfield +names cannot begin with a (``$``) symbol. + diff --git a/source/includes/warning-possible-data-loss.rst b/source/includes/warning-possible-data-loss.rst new file mode 100644 index 00000000000..bc55a9df091 --- /dev/null +++ b/source/includes/warning-possible-data-loss.rst @@ -0,0 +1,20 @@ +.. warning:: + + There is a small chance of data loss when using (``$``) prefixed + field names or field names with (``.``) characters when this type of + field name is used in conjunction with unacknowledged writes + (:doc:`write concern ` ``w=0``) on servers + that are older than MongoDB 5.0. + + When running :doc:`insert `, + :doc:`update `, and + :doc:`findAndModify ` + commands, drivers that are 5.0 compatible remove restrictions on + using documents with field names that are (``$``) prefixed or that + contain (``.``) characters. These field names generated a + client-side error in earlier driver versions. + + The restrictions are removed regardless of the server version the + driver is connected to. If a 5.0 driver sends a document to an older + server, the document will be rejected without sending an error. + diff --git a/source/reference/command/findAndModify.txt b/source/reference/command/findAndModify.txt index c031f3addf2..bc3211c29d9 100644 --- a/source/reference/command/findAndModify.txt +++ b/source/reference/command/findAndModify.txt @@ -457,6 +457,10 @@ Comparisons with the ``update`` Method .. include:: /includes/fact-findAndModify-update-comparison.rst +.. seealso :: + + :ref:`Considerations for field names ` + Transactions ~~~~~~~~~~~~ @@ -474,7 +478,6 @@ Write Concerns and Transactions .. include:: /includes/extracts/transactions-operations-write-concern.rst - Examples -------- diff --git a/source/reference/limits.txt b/source/reference/limits.txt index 200f1c37727..1e66d3202d2 100644 --- a/source/reference/limits.txt +++ b/source/reference/limits.txt @@ -93,8 +93,14 @@ Naming Restrictions .. include:: /includes/fact-document-field-name-restrictions.rst + .. include:: /includes/warning-possible-data-loss.rst + .. include:: /includes/warning-document-duplicate-key-names.rst +.. limit:: Restrictions on ``_id`` + + .. include:: /includes/fact-id-field-name-rules.rst + .. _faq-dev-namespace: Namespaces diff --git a/source/reference/operator/aggregation/getField.txt b/source/reference/operator/aggregation/getField.txt index 48ebd1df7df..1268f5b7871 100644 --- a/source/reference/operator/aggregation/getField.txt +++ b/source/reference/operator/aggregation/getField.txt @@ -107,6 +107,10 @@ Behavior ``a.b.c`` as a top-level field ``a.b.c`` instead of a nested field ``{ a: { b: { c: } } }``. +.. seealso :: + + :ref:`Considerations for field names ` + Examples -------- @@ -185,3 +189,4 @@ The operation returns the following results: { _id: 2, item: 'winter coat', qty: 200, '$price': 499.99 }, { _id: 4, item: 'leather boots', qty: 300, '$price': 249.99 } ] + diff --git a/source/reference/operator/aggregation/setField.txt b/source/reference/operator/aggregation/setField.txt index 6995e7ced04..3beddb4c3a8 100644 --- a/source/reference/operator/aggregation/setField.txt +++ b/source/reference/operator/aggregation/setField.txt @@ -95,6 +95,10 @@ Behavior ``a.b.c`` as a top-level field ``a.b.c`` instead of a nested field ``{ a: { b: { c: } } }``. +.. seealso :: + + :ref:`Considerations for field names ` + Examples -------- diff --git a/source/release-notes/5.0.txt b/source/release-notes/5.0.txt index 6b4d2cdaacb..7279d2c8c4d 100644 --- a/source/release-notes/5.0.txt +++ b/source/release-notes/5.0.txt @@ -479,6 +479,16 @@ Transactions General Improvements -------------------- +Improved Handling of (``$``) and (``.``) in Field Names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MongoDB 5.0 adds :ref:`improved support +` for field names that are +(``$``) prefixed or that contain (``.``) characters. The validation +rules for storing data have been updated to make it easier to work with +data sources that use these characters. + + Implicit Default Write Concern ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~