From 5305ac65e92c2c91ac459046f2d9124d1d64af59 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Thu, 8 Nov 2012 21:21:50 -0500 Subject: [PATCH 1/5] DOCS-653 capped collections first draft --- draft/reference/capped-collection.txt | 205 ++++++++++++++++++++++++++ source/core/indexes.txt | 1 + source/reference/commands.txt | 2 + 3 files changed, 208 insertions(+) create mode 100644 draft/reference/capped-collection.txt diff --git a/draft/reference/capped-collection.txt b/draft/reference/capped-collection.txt new file mode 100644 index 00000000000..9152da3de17 --- /dev/null +++ b/draft/reference/capped-collection.txt @@ -0,0 +1,205 @@ +================= +Capped Collection +================= + +.. default-domain:: mongodb + +:term:`Capped collections ` are fixed-size +collections. Once they reach their fixed size, capped collections +automatically overwrite their oldest entries. The :term:`oplog.rs ` +collection use in replication is an example of a capped collection. + +Capped collections have a high performance, automatic, +first-in-first-out feature based on insertion order. Once the +collection's allocated space is fully used, newly added documents +replace the oldest documents in the collection. + +Capped collections automatically maintain insertion order for the +documents in the collection. This is very powerful for certain use +cases, such as logging. + +You cannot shard a capped collection. + +Examples of Capped Collections +------------------------------ + +The following are scenarios in which you might want to use a capped collection: + +- Logging + + Capped collections provide a high-performance means for storing + logging documents in the database. Inserting documents in an unindexed + capped collection is close to the speed of logging to a + filesystem. Additionally, with the built-in FIFO mechanism, you are + not at risk of using excessive disk space for the logging. + +- Automatic Maintaining of Insertion Order + + Capped collections keep documents in their insertion order + automatically, with no index being required for this property. The + logging example above is a good example of a case where keeping items + in order is important. + +- Caching + + If you wish to cache a small number of documents in the database, + perhaps cached computations of information, the capped tables provide + a convenient mechanism for this. Note that for this application you + typically use an index on the capped table as there are be more reads + than writes. + +- Automatic Age Out + + If you know you want data to automatically "roll out" over time as it + ages, a capped collection can be an easier way to support than writing + manual removal via cron scripts. Ejection from the capped collection + is also inexpensive compared to explicit remove operations. + +Recommendations and Restrictions +-------------------------------- + +You may insert new documents in the capped collection. + +You may update the existing documents in the collection. However, the +documents must not grow in size. If they do, the update fails. + +Note that for versions prior to 2.2, if perform updates, you likely want +to declare an appropriate index as pre-2.2 versions do not create an +``_id`` index by default. + +You cannot delete documents from a capped collection. To remove all +records from the collection, use the :method:`drop() +` method. After the drop you must explicitly +recreate the collection. + +Capped collection are not shard-able. + +.. versionchanged:: 2.2 + +All :term:`capped collections ` have an ``_id`` field +by default and have indexes on the ``_id`` field, *with the exception* +of the capped collections in the ``local`` :term:`database`. This change +affects capped collections created with 2.2 instances and does not +affect existing capped collections. + +.. warning:: + + Prior to 2.2, capped collections do not have a unique index on + ``_id``. However, replication refers to each document by its ``_id``, + so if you are lacking an index on ``_id``, replication can fall + behind or stall. Thus, if you are using a capped collections and + replication, you should create a unique index on ``_id``. Having + a duplicate ``_id`` in your capped collection also can confuse + replication, so it is important to ensure uniqueness by using the + default client-generated MongoDB ``_id`` or the ``autoIndexId`` + field, as explained in :ref:`capped-collections-options`. + +When appropriate, do not create indexes on a capped collection. If the +collection is written to much more than it is read from, it is +better to have no indexes. Note that you may create indexes on a capped +collection; however, you are then moving from "log speed" inserts to +"database speed" inserts -- that is, it still is quite fast by +database standards. + +Use natural ordering to retrieve the most recently inserted elements +from the collection efficiently. This is (somewhat) analogous to tail on +a log file. + +Create a Capped Collection +-------------------------- + +Unlike a standard collection, you must explicitly create a capped +collection, specifying a collection size in bytes. The collection's data +space is then preallocated. Note that the size specified includes +database headers. + +.. code-block:: javascript + + db.createCollection("mycoll", {capped:true, size:100000}) + +.. seealso:: :method:`db.createCollection()` + +.. _capped-collections-options: + +Options +``````` + +When you create a capped collection, you can set the following options: + +- ``size`` + + The size, in bytes, of the capped collection. This must be specified. + +- ``max`` + + This caps the number of documents in the collection. This setting is optional. + Once the limit is reached, items roll out on a least recently inserted + basis. + + When specifying a cap on the number of documents, make sure you set + the ``size`` parameter to leave room for your chosen number of + documents so that items don't roll out faster than expected. You can + use the :method:`validate() ` method to see + how much space an existing collection uses and from that estimate your + size needs. Consider the following sequence of commands: + + .. code-block:: javascript + + db.createCollection("mycoll", {capped:true, size:100000, max:100}); + db.mycoll.validate(); + +- ``autoIndexId`` + + The autoIndexId field may be set to ``true`` or ``false`` to explicitly enable + or disable automatic creation of a unique key index on the ``_id`` + field. Beginning with MongoDB version 2.2, ``autoIndexId`` is enabled by + default. Prior to 2.2 it is disabled by default. + +.. seealso:: :ref:`collection-commands` + +Query a Capped Collection +------------------------- + +If you perform a :method:`find() ` on a capped +collection with no ordering specified, the documents are returned in +insertion order. + +To retrieve documents in reverse insertion order, issue :method:`find() +` along with the :method:`sort() ` +method with the ``$natural`` parameter set to ``-1``, as shown in the +following example: + +.. code-block:: javascript + + db.cappedCollection.find().sort({$natural:-1}) + +Check if a Collection is Capped +------------------------------- + +You can check if a collection is capped by using the ``isCapped()`` method: + +.. code-block:: javascript + + db.collection.isCapped() + +Convert a Collection to Capped +------------------------------ + +You can convert a non-capped collection to a capped collection with +the :dbcommand:`convertToCapped` command: + +.. code-block:: javascript + + db.runCommand({"convertToCapped": "mycoll", size: 100000}); + +Note that the size is in bytes. + +No indexes are created when the new capped collection is. If you +want the old indexes you must recreate them after it has been +converted. + +Automatically Remove Data After a Specified Period of Time +---------------------------------------------------------- + +To automatically remove data, use MongoDB’s :term:`TTL` feature, as +described in :ref:`/tutorial/expire-data`. diff --git a/source/core/indexes.txt b/source/core/indexes.txt index dda9a8b0487..39b384e19cf 100644 --- a/source/core/indexes.txt +++ b/source/core/indexes.txt @@ -97,6 +97,7 @@ is a 12-byte unique identifiers suitable for use as the value of an for more information. .. todo:: fix the above when a full capped-collection page exists. + 2012.11.08 Note: The capped-collection page is now created and in the draft folder. .. _index-types-secondary: diff --git a/source/reference/commands.txt b/source/reference/commands.txt index 66f2237044f..451d9f49f48 100644 --- a/source/reference/commands.txt +++ b/source/reference/commands.txt @@ -172,6 +172,8 @@ Geospatial Commands .. include:: command/geoSearch.txt :start-after: mongodb +.. _collection-commands: + Collection Commands ~~~~~~~~~~~~~~~~~~~ From 60c14ceaebf518b1fbbd07dabd07a3c09453cbd8 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Mon, 12 Nov 2012 13:39:53 -0500 Subject: [PATCH 2/5] DOCS-653 review edits --- draft/core/capped-collections.txt | 165 +++++++++++++++++++++ draft/reference/capped-collection.txt | 205 -------------------------- 2 files changed, 165 insertions(+), 205 deletions(-) delete mode 100644 draft/reference/capped-collection.txt diff --git a/draft/core/capped-collections.txt b/draft/core/capped-collections.txt index 03bb605289e..e06413e0801 100644 --- a/draft/core/capped-collections.txt +++ b/draft/core/capped-collections.txt @@ -2,3 +2,168 @@ Capped Collections ================== +.. default-domain:: mongodb + +:term:`Capped collections ` are fixed-size +collections that automatically overwrite their oldest entries when the +fixed size is reached. Capped collections work in a way similar to +circular buffers. The :term:`oplog.rs ` +collection used in replication is an example of a capped collection. + +Capped collections automatically store documents in their insertion +order. Once the collection's allocated space is fully used, newly added +documents replace the oldest documents in the collection. The automatic +replacement based on insertion order make capped collections fast and +efficient. + +You cannot increase the size of a document in a capped. If you update a +document, it must not grow in size. + +You cannot shard a capped collection. + +Capped Collection Use Cases +--------------------------- + +The following are scenarios in which you might want to use a capped collection: + +- Logging + + Capped collections provide a high-performance means for storing + logging documents in the database. Inserting documents in an unindexed + capped collection is close to the speed of logging to a + filesystem. Additionally, with the built-in FIFO mechanism, you are + not at risk of using excessive disk space for the logging. + +- Automatic Maintaining of Insertion Order + + Capped collections keep documents in their insertion order + automatically, with no index being required for this property. The + logging example above is a good example of a case where keeping items + in order is important. + +- Caching + + If you wish to cache a small number of documents in the database, + perhaps cached computations of information, the capped tables provide + a convenient mechanism for this. Note that for this application you + typically use an index on the capped table as there are be more reads + than writes. + +- Automatic Age Out + + If you know you want data to automatically "roll out" over time as it + ages, a capped collection can be an easier way to support than writing + manual removal via cron scripts. Ejection from the capped collection + is also inexpensive compared to explicit remove operations. + +Recommendations and Restrictions +-------------------------------- + +You may update the existing documents in the collection. However, the +documents must not grow in size. If they do, the update fails. + +Note that for versions prior to 2.2, if perform updates, you likely want +to declare an appropriate index as pre-2.2 versions do not create an +``_id`` index by default. + +You cannot delete documents from a capped collection. To remove all +records from the collection, use the :method:`drop() +` method. After the drop you must explicitly +recreate the collection. + +Capped collection are not shard-able. + +.. versionchanged:: 2.2 + +All :term:`capped collections ` have an ``_id`` field +by default and have indexes on the ``_id`` field, *with the exception* +of the capped collections in the ``local`` :term:`database`. This change +affects capped collections created with 2.2 instances and does not +affect capped collections created with pre-2.2 instances. + +.. warning:: + + Prior to 2.2, capped collections do not have a unique index on + ``_id``. If you are using a capped collection and + replication, you should create a unique index on ``_id``. + Ensure uniqueness by using the + default client-generated MongoDB ``_id`` or the ``autoIndexId`` + field, as explained in :ref:`capped-collections-options`. + +When appropriate, do not create indexes on a capped collection. If the +collection is written to much more than it is read from, it is +better to have no indexes. Note that you may create indexes on a capped +collection; however, you are then moving from "log speed" inserts to +"database speed" inserts -- that is, it still is quite fast by +database standards. + +Use natural ordering to retrieve the most recently inserted elements +from the collection efficiently. This is (somewhat) analogous to tail on +a log file. + +Procedures +---------- + +Create a Capped Collection +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Unlike a standard collection, you must explicitly create a capped +collection using the :method:`db.createCollection()` method. When you +do, you specify the collection size in bytes. The size must include +space for database headers. MongoDB then preallocates the space. + +.. code-block:: javascript + + db.createCollection("mycoll", {capped:true, size:100000}) + +.. seealso:: :method:`db.createCollection()` + +.. _capped-collections-options: + +Query a Capped Collection +~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you perform a :method:`find() ` on a capped +collection with no ordering specified, the documents are returned in +insertion order. + +To retrieve documents in reverse insertion order, issue :method:`find() +` along with the :method:`sort() ` +method with the ``$natural`` parameter set to ``-1``, as shown in the +following example: + +.. code-block:: javascript + + db.cappedCollection.find().sort({$natural:-1}) + +Check if a Collection is Capped +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can check if a collection is capped by using the ``isCapped()`` method: + +.. code-block:: javascript + + db.collection.isCapped() + +Convert a Collection to Capped +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can convert a non-capped collection to a capped collection with +the :dbcommand:`convertToCapped` command: + +.. code-block:: javascript + + db.runCommand({"convertToCapped": "mycoll", size: 100000}); + +Note that the size is in bytes. + +No indexes are created when the new capped collection is. If you +want the old indexes you must recreate them after it has been +converted. + +Automatically Remove Data After a Specified Period of Time +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To automatically remove data, use MongoDB’s :term:`TTL` feature, as +described in :ref:`/tutorial/expire-data`. + diff --git a/draft/reference/capped-collection.txt b/draft/reference/capped-collection.txt deleted file mode 100644 index 9152da3de17..00000000000 --- a/draft/reference/capped-collection.txt +++ /dev/null @@ -1,205 +0,0 @@ -================= -Capped Collection -================= - -.. default-domain:: mongodb - -:term:`Capped collections ` are fixed-size -collections. Once they reach their fixed size, capped collections -automatically overwrite their oldest entries. The :term:`oplog.rs ` -collection use in replication is an example of a capped collection. - -Capped collections have a high performance, automatic, -first-in-first-out feature based on insertion order. Once the -collection's allocated space is fully used, newly added documents -replace the oldest documents in the collection. - -Capped collections automatically maintain insertion order for the -documents in the collection. This is very powerful for certain use -cases, such as logging. - -You cannot shard a capped collection. - -Examples of Capped Collections ------------------------------- - -The following are scenarios in which you might want to use a capped collection: - -- Logging - - Capped collections provide a high-performance means for storing - logging documents in the database. Inserting documents in an unindexed - capped collection is close to the speed of logging to a - filesystem. Additionally, with the built-in FIFO mechanism, you are - not at risk of using excessive disk space for the logging. - -- Automatic Maintaining of Insertion Order - - Capped collections keep documents in their insertion order - automatically, with no index being required for this property. The - logging example above is a good example of a case where keeping items - in order is important. - -- Caching - - If you wish to cache a small number of documents in the database, - perhaps cached computations of information, the capped tables provide - a convenient mechanism for this. Note that for this application you - typically use an index on the capped table as there are be more reads - than writes. - -- Automatic Age Out - - If you know you want data to automatically "roll out" over time as it - ages, a capped collection can be an easier way to support than writing - manual removal via cron scripts. Ejection from the capped collection - is also inexpensive compared to explicit remove operations. - -Recommendations and Restrictions --------------------------------- - -You may insert new documents in the capped collection. - -You may update the existing documents in the collection. However, the -documents must not grow in size. If they do, the update fails. - -Note that for versions prior to 2.2, if perform updates, you likely want -to declare an appropriate index as pre-2.2 versions do not create an -``_id`` index by default. - -You cannot delete documents from a capped collection. To remove all -records from the collection, use the :method:`drop() -` method. After the drop you must explicitly -recreate the collection. - -Capped collection are not shard-able. - -.. versionchanged:: 2.2 - -All :term:`capped collections ` have an ``_id`` field -by default and have indexes on the ``_id`` field, *with the exception* -of the capped collections in the ``local`` :term:`database`. This change -affects capped collections created with 2.2 instances and does not -affect existing capped collections. - -.. warning:: - - Prior to 2.2, capped collections do not have a unique index on - ``_id``. However, replication refers to each document by its ``_id``, - so if you are lacking an index on ``_id``, replication can fall - behind or stall. Thus, if you are using a capped collections and - replication, you should create a unique index on ``_id``. Having - a duplicate ``_id`` in your capped collection also can confuse - replication, so it is important to ensure uniqueness by using the - default client-generated MongoDB ``_id`` or the ``autoIndexId`` - field, as explained in :ref:`capped-collections-options`. - -When appropriate, do not create indexes on a capped collection. If the -collection is written to much more than it is read from, it is -better to have no indexes. Note that you may create indexes on a capped -collection; however, you are then moving from "log speed" inserts to -"database speed" inserts -- that is, it still is quite fast by -database standards. - -Use natural ordering to retrieve the most recently inserted elements -from the collection efficiently. This is (somewhat) analogous to tail on -a log file. - -Create a Capped Collection --------------------------- - -Unlike a standard collection, you must explicitly create a capped -collection, specifying a collection size in bytes. The collection's data -space is then preallocated. Note that the size specified includes -database headers. - -.. code-block:: javascript - - db.createCollection("mycoll", {capped:true, size:100000}) - -.. seealso:: :method:`db.createCollection()` - -.. _capped-collections-options: - -Options -``````` - -When you create a capped collection, you can set the following options: - -- ``size`` - - The size, in bytes, of the capped collection. This must be specified. - -- ``max`` - - This caps the number of documents in the collection. This setting is optional. - Once the limit is reached, items roll out on a least recently inserted - basis. - - When specifying a cap on the number of documents, make sure you set - the ``size`` parameter to leave room for your chosen number of - documents so that items don't roll out faster than expected. You can - use the :method:`validate() ` method to see - how much space an existing collection uses and from that estimate your - size needs. Consider the following sequence of commands: - - .. code-block:: javascript - - db.createCollection("mycoll", {capped:true, size:100000, max:100}); - db.mycoll.validate(); - -- ``autoIndexId`` - - The autoIndexId field may be set to ``true`` or ``false`` to explicitly enable - or disable automatic creation of a unique key index on the ``_id`` - field. Beginning with MongoDB version 2.2, ``autoIndexId`` is enabled by - default. Prior to 2.2 it is disabled by default. - -.. seealso:: :ref:`collection-commands` - -Query a Capped Collection -------------------------- - -If you perform a :method:`find() ` on a capped -collection with no ordering specified, the documents are returned in -insertion order. - -To retrieve documents in reverse insertion order, issue :method:`find() -` along with the :method:`sort() ` -method with the ``$natural`` parameter set to ``-1``, as shown in the -following example: - -.. code-block:: javascript - - db.cappedCollection.find().sort({$natural:-1}) - -Check if a Collection is Capped -------------------------------- - -You can check if a collection is capped by using the ``isCapped()`` method: - -.. code-block:: javascript - - db.collection.isCapped() - -Convert a Collection to Capped ------------------------------- - -You can convert a non-capped collection to a capped collection with -the :dbcommand:`convertToCapped` command: - -.. code-block:: javascript - - db.runCommand({"convertToCapped": "mycoll", size: 100000}); - -Note that the size is in bytes. - -No indexes are created when the new capped collection is. If you -want the old indexes you must recreate them after it has been -converted. - -Automatically Remove Data After a Specified Period of Time ----------------------------------------------------------- - -To automatically remove data, use MongoDB’s :term:`TTL` feature, as -described in :ref:`/tutorial/expire-data`. From 3a9b6de35f3dcb49473689268b5c4d1ad0a18011 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Mon, 12 Nov 2012 13:50:02 -0500 Subject: [PATCH 3/5] DOCS-653 minor --- draft/core/capped-collections.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/core/capped-collections.txt b/draft/core/capped-collections.txt index e06413e0801..d8025c98200 100644 --- a/draft/core/capped-collections.txt +++ b/draft/core/capped-collections.txt @@ -108,7 +108,7 @@ Create a Capped Collection ~~~~~~~~~~~~~~~~~~~~~~~~~~ Unlike a standard collection, you must explicitly create a capped -collection using the :method:`db.createCollection()` method. When you +collection using the :method:`createCollection() ` method. When you do, you specify the collection size in bytes. The size must include space for database headers. MongoDB then preallocates the space. From 957ed451fd99563675c8b0881a22876467bce959 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Mon, 12 Nov 2012 13:56:15 -0500 Subject: [PATCH 4/5] DOCS-653 minor --- draft/core/capped-collections.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/draft/core/capped-collections.txt b/draft/core/capped-collections.txt index d8025c98200..c6b714f3cca 100644 --- a/draft/core/capped-collections.txt +++ b/draft/core/capped-collections.txt @@ -13,11 +13,11 @@ collection used in replication is an example of a capped collection. Capped collections automatically store documents in their insertion order. Once the collection's allocated space is fully used, newly added documents replace the oldest documents in the collection. The automatic -replacement based on insertion order make capped collections fast and +replacement based on insertion order makes capped collections fast and efficient. -You cannot increase the size of a document in a capped. If you update a -document, it must not grow in size. +You cannot increase the size of a document in a capped collection. You can update a +document, but it must not grow in size. You cannot shard a capped collection. From 7900a0cddc0d6acfc66eac194df38e24d6bb17d3 Mon Sep 17 00:00:00 2001 From: Bob Grabar Date: Mon, 12 Nov 2012 16:08:13 -0500 Subject: [PATCH 5/5] DOCS-653 rewrote top section --- draft/core/capped-collections.txt | 73 +++++++++++++------------------ 1 file changed, 31 insertions(+), 42 deletions(-) diff --git a/draft/core/capped-collections.txt b/draft/core/capped-collections.txt index c6b714f3cca..e0242ba6ee4 100644 --- a/draft/core/capped-collections.txt +++ b/draft/core/capped-collections.txt @@ -5,60 +5,49 @@ Capped Collections .. default-domain:: mongodb :term:`Capped collections ` are fixed-size -collections that automatically overwrite their oldest entries when the -fixed size is reached. Capped collections work in a way similar to -circular buffers. The :term:`oplog.rs ` -collection used in replication is an example of a capped collection. +collections that automatically overwrite their oldest documents when new +documents are inserted. Capped collections work in a way similar to +circular buffers. Once a collection fills its allocated space, it makes +room for new documents by deleting the oldest documents in the +collection. -Capped collections automatically store documents in their insertion -order. Once the collection's allocated space is fully used, newly added -documents replace the oldest documents in the collection. The automatic -replacement based on insertion order makes capped collections fast and -efficient. +Capped collections are particularly fast and efficient for operations, +such as logging, that retrieve and delete documents based on insertion +order. -You cannot increase the size of a document in a capped collection. You can update a -document, but it must not grow in size. +Capped collections have the following behaviors: -You cannot shard a capped collection. - -Capped Collection Use Cases ---------------------------- - -The following are scenarios in which you might want to use a capped collection: - -- Logging +- Capped collections automatically store documents in their insertion + order. Therefore, no index is required to query on insertion order, + and capped collections incur no indexing overhead for this property. - Capped collections provide a high-performance means for storing - logging documents in the database. Inserting documents in an unindexed - capped collection is close to the speed of logging to a - filesystem. Additionally, with the built-in FIFO mechanism, you are - not at risk of using excessive disk space for the logging. +- Capped collections guarantee that insertion order is identical to the + order on disk (:term:`natural order`) and do so by prohibiting updates + that increase document size. Capped collections only allow updates + that fit the original document size, which ensures a document does not + change its location on disk. -- Automatic Maintaining of Insertion Order +- Capped collections automatically remove the oldest documents in the + collection without requiring scripts or explicit remove operations. - Capped collections keep documents in their insertion order - automatically, with no index being required for this property. The - logging example above is a good example of a case where keeping items - in order is important. +Capped collections provide a high-performance means for storing logging +information. Inserting documents in an unindexed capped collection is +close to the speed of logging to a filesystem. Additionally, the +built-in FIFO mechanism ensures logging does not use excessive disk +space. -- Caching +Capped collections also provide a convenient way to cache known small +numbers of documents, such as cached computations of information. This +scenario would likely have more reads than writes and require an index. - If you wish to cache a small number of documents in the database, - perhaps cached computations of information, the capped tables provide - a convenient mechanism for this. Note that for this application you - typically use an index on the capped table as there are be more reads - than writes. - -- Automatic Age Out - - If you know you want data to automatically "roll out" over time as it - ages, a capped collection can be an easier way to support than writing - manual removal via cron scripts. Ejection from the capped collection - is also inexpensive compared to explicit remove operations. +The :term:`oplog.rs ` collection used in replication is an +example of a capped collection. Recommendations and Restrictions -------------------------------- +You cannot shard a capped collection. + You may update the existing documents in the collection. However, the documents must not grow in size. If they do, the update fails.