DOCSP-15956 reintroduce dotDollar validation

source/core/crud.txt

@@ -26,6 +26,7 @@ Query Plan, Performance, and Analysis
 
 Miscellaneous
    - :doc:`/core/tailable-cursors`
+   - :doc:`/core/dot-dollar-considerations`
 
 .. seealso::
 
@@ -34,10 +35,11 @@ Miscellaneous
 .. toctree::
    :titlesonly: 
 
+   /tutorial/analyze-query-plan
    /core/write-operations-atomicity
-   /core/read-isolation-consistency-recency
    /core/distributed-queries
-   /core/query-plans
+   /core/dot-dollar-considerations
+   /core/read-isolation-consistency-recency
    /core/query-optimization
-   /tutorial/analyze-query-plan
+   /core/query-plans
    /core/tailable-cursors

source/core/document.txt

@@ -75,9 +75,7 @@ Field names are strings.
 :doc:`Documents </core/document>` have the following restrictions on field
 names:
 
-- The field name ``_id`` is reserved for use as a primary key; its
-  value must be unique in the collection, is immutable, and may be of
-  any type other than an array.
+- .. include:: /includes/fact-id-field-name-rules.rst
 
 .. include:: /includes/fact-document-field-name-restrictions.rst
 
@@ -95,7 +93,6 @@ existing user document.
 Field Value Limit
 ~~~~~~~~~~~~~~~~~
 
-
 MongoDB 2.6 through MongoDB versions with :ref:`featureCompatibilityVersion <view-fcv>` (fCV) set to ``"4.0"`` or earlier
    For :doc:`indexed collections </indexes>`, the values for the
    indexed fields have a :limit:`Maximum Index Key Length <Index Key Limit>`. See
@@ -142,7 +139,7 @@ For examples querying arrays, see:
 .. seealso::
 
    - :update:`$[]` all positional operator for update operations,
-   
+
    - :update:`$[\<identifier\>]` filtered positional operator for update operations,
 
    - :update:`$` positional operator for update operations,
@@ -188,6 +185,7 @@ For examples querying embedded documents, see:
 
 - :doc:`/tutorial/query-array-of-documents/`
 
+
 Document Limitations
 --------------------
 
@@ -220,6 +218,9 @@ The ``_id`` field has the following behavior and constraints:
   server receives a document that does not have the ``_id`` field
   first, then the server will move the field to the beginning.
 
+_ If the ``_id`` contains subfields, the subfield names cannot begin
+  with a (``$``) symbol.
+
 - The ``_id`` field may contain values of any :doc:`BSON data type
   </reference/bson-types>`, other than an array, regex, or undefined.
 
@@ -286,7 +287,7 @@ For examples, see:
 - :doc:`/tutorial/query-documents`
 
 - :doc:`/tutorial/query-embedded-documents`
-   
+
 - :doc:`/tutorial/query-arrays`
 
 - :doc:`/tutorial/query-array-of-documents/`
@@ -327,7 +328,7 @@ type:
 Further Reading
 ---------------
 
-For more information on the MongoDB document model, download the 
+For more information on the MongoDB document model, download the
 `MongoDB Application Modernization Guide
 <https://www.mongodb.com/modernize?tck=docs_server>`_.
 

source/core/dot-dollar-considerations.txt

@@ -0,0 +1,228 @@
+====================================
+Field Names with (``.``) and (``$``) 
+====================================
+
+.. default-domain:: mongodb
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 1
+   :class: singlecol
+
+.. _crud-concepts-dot-dollar-considerations:
+
+Overview 
+--------
+
+MongoDB 5.0 adds improved support for field names that are (``$``)
+prefixed or that contain (``.``) characters. The validation rules for
+storing data have been updated to make it easier to work with data
+sources that use these characters. 
+
+In most cases data that has been stored using field names like these
+is not directly accessible. You need to use helper methods like
+:expression:`$getField`, :expression:`$setField`, and
+:expression:`$literal` in queries that access those fields. 
+
+The field name validation rules are not the same for all types of
+storage operations. This page summarizes how different insert and
+update operations handle (``$``) prefixed field names. 
+
+Insert operations
+-----------------
+
+(``$``) prefixed fields are permitted as top level and nested field
+names for inserts.
+
+.. code-block:: javascript
+   :emphasize-lines: 3
+
+   db.sales.insertOne( {
+      "$price": 50.00,
+      "quantity": 30
+   } )
+
+(``$``) prefixed fields are permitted on inserts using otherwise
+reserved words. Operator names like :update:`$inc` can be used as
+field names as well as words like ``id``, ``db``, and ``ref``.
+
+.. code-block:: javascript
+   :emphasize-lines: 2, 4-6
+
+   db.books.insertOne( {
+      "$id": "h1961-01",
+      "location": {
+         "$db": "novels",
+         "$ref": "2007042768",
+         "$inc": true
+   } } )
+
+An update which creates a new document during an :term:`upsert` is
+treated as an ``insert`` rather than an ``update`` for field name
+validation. :term:`Upserts <upsert>` can accept (``$``) prefixed
+fields. However, :term:`upserts <upsert>` are a special case and
+similar update operations may cause an error if the ``match`` portion
+of the update selects an existing document.
+
+This code sample has ``upsert: true`` so it will insert a new document
+if the collection doesn't already contain a document that matches the
+query term, ``{ "date": "2021-07-07" }``. If this sample code matches
+an existing document, the update will fail since ``$hotel`` is (``$``)
+prefixed.
+
+.. code-block:: javascript
+   :emphasize-lines: 5
+
+   db.expenses.updateOne(
+      { "date": "2021-07-07" },
+      { $set: {
+         "phone": 25.17,
+         "$hotel": 320.10
+      } },
+      { upsert: true }
+   )
+
+Document Replacing Updates
+--------------------------
+
+Update operators either replace existing fields with new documents
+or else modify those fields. In cases where the update performs a
+replacement, (``$``) prefixed fields are not permitted as top level
+field names. 
+
+Consider a document like
+
+.. code-block:: javascript::
+
+   {
+      "_id": "E123",
+      "address": {
+         "$number": 123,
+         "$street": "Elm Road"
+      },
+      "$rooms": {
+         "br": 2,
+         "bath": 1
+      }
+   }
+
+You could use an update operator that replaces an existing document to
+modify the ``address.$street`` field but you could not update the
+``$rooms`` field that way. 
+
+.. code-block::
+
+   db.housing.updateOne(
+      { "_id": "E123" },
+      { $set: { "address.$street": "Elm Ave" } }
+   )
+
+Use :expression:`$setField` as part of an aggregation pipeline to
+:ref:`update top level <dotDollar-aggregate-update>` (``$``) prefixed
+fields like ``$rooms``.
+
+Document Modifying Updates
+--------------------------
+
+When an update modifies, rather than replaces, existing document
+fields, (``$``) prefixed fields can be top level field names.
+Subfields can be accessed directly, but you need a helper method to
+access the top level fields. 
+
+.. seealso::
+
+   :expression:`$getField`, :expression:`$setField`,
+   :expression:`$literal`, :pipeline:`$replaceWith`
+
+Consider a collection with documents like this inventory record:
+
+.. code-block::
+   :copyable: false
+
+   {
+      _id: ObjectId("610023ad7d58ecda39b8d161"),
+      "part": "AB305",
+      "$bin": 200,
+      "quantity": 100,
+      "pricing": { sale: true, "$discount": 60 }
+   }
+
+The ``pricing.$discount`` subfield can be queried directly.
+
+.. code-block::
+
+   db.inventory.findAndModify( {
+       query: { "part": { $eq: "AB305" } },
+       update: { $inc: { "pricing.$discount": 10 } }
+   } )
+
+
+Use :expression:`$getField` and :expression:`$literal` to access the
+value of the top level ``$bin`` field.
+
+.. code-block::
+   :emphasize-lines: 3
+
+   db.inventory.findAndModify( {
+      query: { $expr: {
+         $eq: [ { $getField: { $literal: "$bin" } }, 200 ]
+      } }, 
+      update: { $inc: { "quantity": 10 } }
+   } )
+
+.. _dotDollar-aggregate-update:
+
+Updates Using Aggregation Pipelines
+-----------------------------------
+
+Use :expression:`$setField`, :expression:`$getField`, and
+:expression:`$literal` in the :pipeline:`$replaceWith` stage to modify
+(``$``) prefixed fields in an aggregation :term:`pipeline`.
+
+Consider a collection of school records like:
+
+.. code-block:: javascript
+   :copyable: false
+
+   {
+      "_id": 100001,
+      "$term": "fall",
+      "registered": true,
+      "grade": 4
+   }
+
+Create a new collection for the spring semester using a 
+:term:`pipeline` to update the (``$``) prefixed ``$term`` field.
+
+.. code-block:: javascript
+   :emphasize-lines: 3-5
+
+   db.school.aggregate( [
+      { $match: { "registered": true } }, 
+      { $replaceWith: {
+         $setField: {
+            field: { $literal: "$term" }, 
+            input: "$$ROOT",
+            value: "spring"
+      } } },
+      { $out: "spring2022" }
+   ] )
+
+General Restrictions
+--------------------
+
+In addition to the storage validation rules above, there are some
+general restrictions on using (``$``) prefixed field names. These
+fields cannot: 
+
+- Be indexed
+- Be used as part of a shard key
+- Be validated using :query:`$jsonSchema`
+- Be be modified with an escape sequence
+- Be used with
+  :driver:`Field Level Encryption </security/client-side-field-level-encryption-guide>`
+- Be used as a subfield in an ``_id`` document
+
+.. include:: /includes/warning-possible-data-loss.rst
+

source/includes/fact-document-field-name-restrictions.rst

@@ -1,17 +1,9 @@
 - Field names **cannot** contain the ``null`` character.
 
-- Top-level field names **cannot** start with the dollar sign (``$``) character.
+- The server permits storage of field names that contain dots (``.``)
+  and dollar signs (``$``).
 
-  Otherwise, starting in MongoDB 3.6, the server permits storage of
-  field names that contain dots (i.e. ``.``) and dollar signs (i.e.
-  ``$``).
+- MongodB 5.0 adds improved support for the use of (``$``) and (``.``)
+  in field names. There are some restrictions. See
+  :ref:`Field Name Considerations <crud-concepts-dot-dollar-considerations>` for more details.
 
-  .. important::
-
-     The MongoDB Query Language cannot always meaningfully express
-     queries over documents whose field names contain these characters
-     (see :issue:`SERVER-30575`).
-
-     Until support is added in the query language, the use of ``$`` and
-     ``.`` in field names is not recommended and is not supported by
-     the official MongoDB drivers.

source/includes/fact-id-field-name-rules.rst

@@ -0,0 +1,5 @@
+The field name ``_id`` is reserved for use as a primary key; its value
+must be unique in the collection, is immutable, and may be of any type
+other than an array. If the ``_id`` contains subfields, the subfield
+names cannot begin with a (``$``) symbol.
+

source/includes/warning-possible-data-loss.rst

@@ -0,0 +1,20 @@
+.. warning::
+
+   There is a small chance of data loss when using (``$``) prefixed
+   field names or field names with (``.``) characters when this type of
+   field name is used in conjunction with unacknowledged writes
+   (:doc:`write concern </reference/write-concern>` ``w=0``) on servers
+   that are older than MongoDB 5.0.
+
+   When running :doc:`insert </tutorial/insert-documents>`,
+   :doc:`update </tutorial/insert-documents>`, and
+   :doc:`findAndModify </reference/method/db.collection.findAndModify>`
+   commands, drivers that are 5.0 compatible remove restrictions on
+   using documents with field names that are (``$``) prefixed or that
+   contain (``.``) characters. These field names generated a
+   client-side error in earlier driver versions.
+
+   The restrictions are removed regardless of the server version the
+   driver is connected to. If a 5.0 driver sends a document to an older
+   server, the document will be rejected without sending an error.
+

source/reference/command/findAndModify.txt

@@ -457,6 +457,10 @@ Comparisons with the ``update`` Method
 
 .. include:: /includes/fact-findAndModify-update-comparison.rst
 
+.. seealso ::
+
+   :ref:`Considerations for field names <crud-concepts-dot-dollar-considerations>`
+
 Transactions
 ~~~~~~~~~~~~
 
@@ -474,7 +478,6 @@ Write Concerns and Transactions
 
 .. include:: /includes/extracts/transactions-operations-write-concern.rst
 
-
 Examples
 --------
 

source/reference/limits.txt

@@ -93,8 +93,14 @@ Naming Restrictions
 
    .. include:: /includes/fact-document-field-name-restrictions.rst
 
+   .. include:: /includes/warning-possible-data-loss.rst
+
    .. include:: /includes/warning-document-duplicate-key-names.rst
 
+.. limit:: Restrictions on ``_id``
+
+   .. include:: /includes/fact-id-field-name-rules.rst
+
 .. _faq-dev-namespace:
 
 Namespaces