DOCSP-19155: port timeseries migration usage example (#184)

* DOCSP-19155: port timeseries migration usage example

Author: Chris Cho

source/sink-connector.txt

@@ -23,17 +23,17 @@ Overview
 --------
 
 This section focuses on the **{+sink-connector+}**.
-The {+sink-connector+} is a {+kc+} connector that reads data from {+ak+} and 
+The {+sink-connector+} is a {+kc+} connector that reads data from {+ak+} and
 writes data to MongoDB.
 
 Configuration Properties
 ------------------------
 
-To learn about configuration options for your sink connector, see the 
+To learn about configuration options for your sink connector, see the
 :ref:`Configuration Properties <kafka-sink-configuration-properties>` section.
 
 Fundamentals
 ------------
 
-To learn how features of the sink connector work and how to configure them, see the 
+To learn how features of the sink connector work and how to configure them, see the
 :ref:`Fundamentals <kafka-sink-fundamentals>` section.

source/sink-connector/configuration-properties/post-processors.txt

@@ -123,12 +123,6 @@ Settings
        | **Description:**
        | The class that specifies the ``WriteModelStrategy`` the connector should
          use for :manual:`Bulk Writes </core/bulk-write-operations/index.html>`.
-
-       .. seealso::
-
-         For information on how to create your own strategy, see the tutorial
-         on :doc:`Write Strategies </tutorials/write-strategies/>`.
-
        |
        | **Default**:
 

source/tutorials.txt

@@ -6,11 +6,6 @@ Tutorials
    :titlesonly:
    :maxdepth: 1
 
-   Set up Sink and Source Connectors on Docker </tutorials/set-up-on-docker>
-   Set up a Development Environment for Customization </tutorials/set-up-development-environment>
-   Use Built-In and Custom Write Strategies </tutorials/write-strategies>
-   Use Built-In and Custom Post-processors </tutorials/post-processors>
-   Handle Errors in the Sink and Source Connectors </tutorials/handle-errors>
-   Replicate data with the Change Data Capture Handler </tutorials/replicate-with-cdc>
-
-asdf
+   Replicate Data with the Change Data Capture Handler </tutorials/replicate-with-cdc>
+   Migrate an Existing Collection to a Time Series Collection </tutorials/migrate-time-series>
+   

source/tutorials/handle-errors.txt

@@ -0,0 +1,198 @@
+.. _tutorial-migrate-time-series:
+
+==========================================================
+Migrate an Existing Collection to a Time Series Collection
+==========================================================
+
+.. default-domain:: mongodb
+
+In this tutorial, you can learn how to convert an existing MongoDB
+collection to a **time series collection** using the {+mkc+}.
+
+Time series collections efficiently store sequences of measurements
+over a period of time. Time series data consists of measurement data collected
+over time, metadata that describes the measurement, and the time of the
+measurement.
+
+You can configure the source connector to read your existing MongoDB
+collection and the sink connector to write Kafka topic data into a MongoDB
+time series collection.
+
+To learn more about MongoDB time series collections, see the MongoDB
+manual page on :manual:`Time Series Collections </core/timeseries-collections/>`.
+
+Scenario
+--------
+
+Suppose you accumulated stock price data in a MongoDB collection and have
+the following needs:
+
+- More efficient storage of the price data
+- Maintain the ability to analyze stock performance over time using
+  aggregation operators
+
+After reading about MongoDB time series collections, you decide to migrate
+your existing collection into a time series one. Learn how to perform this
+migration in the following sections.
+
+Steps to Migrate to a Time Series Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To migrate an existing MongoDB collection to a time series collection, you
+need to perform the following tasks:
+
+1) :ref:`Identify the time field <time-series-identify-field>` in the existing
+   stock price data document.
+2) :ref:`Configure a source connector <time-series-source-config>` to copy
+   the existing collection data to a Kafka topic.
+3) :ref:`Configure a sink connector <time-series-sink-config>` to copy the
+   Kafka topic data to the time series collection.
+4) :ref:`Verify the connector migrated the data <time-series-verify-collection>` to the time
+   series collection.
+
+.. _time-series-identify-field:
+
+Identify the Time Field
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Before you create a time series collection, you need to identify the
+**time field**. The time field is the document field that MongoDB uses to
+distinguish the time of the measurement. The value of this field can be
+a string, integer, or ISO date. Make sure to set the
+``timeseries.timefield.auto.convert`` setting to instruct the connector to
+automatically convert the value to a date.
+
+The following document shows the format of stock price data documents in
+the existing MongoDB collection:
+
+.. code-block:: javascript
+  :copyable: false
+
+   {
+     tx_time: 2021-07-12T05:20:35Z,
+     symbol: 'WSV',
+     company_name: 'WORRIED SNAKEBITE VENTURES',
+     price: 21.22,
+     _id: ObjectId("...")
+   }
+
+For this scenario, assume you stored these documents in a collection named
+``PriceData`` in the ``Stocks`` database.
+
+You identify that the ``tx_time`` field distinguishes the time of the
+measurements, and specify it as your time field in your sink connector
+configuration.
+
+Learn how to set your time field and field conversion in the
+:ref:`time series configuration guide <time-series-sink-config>`.
+
+.. _time-series-source-config:
+
+Configure the Source Connector
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To copy data from the ``PriceData`` MongoDB collection data and publish it
+to the ``marketdata.Stocks.PriceData`` Kafka topic, create a source
+connector with the following configuration:
+
+.. code-block:: json
+
+   {
+     "name": "mongo-source-marketdata",
+     "config": {
+       "tasks.max":"1",
+       "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
+       "key.converter":"org.apache.kafka.connect.storage.StringConverter",
+       "value.converter":"org.apache.kafka.connect.json.JsonConverter",
+       "publish.full.document.only":"true",
+       "connection.uri":"<your connection uri>",
+       "topic.prefix":"marketdata",
+       "database":"Stocks",
+       "collection":"PriceData",
+       "copy.existing":"true"
+   }}
+
+.. note::
+
+   If you insert documents into a collection during the copying process,
+   the connector inserts them after the process is complete.
+
+After you start your source connector with the preceding configuration,
+the connector starts the copying process. Once the process is complete,
+you should see the following message in the log:
+
+.. code-block:: none
+   :copyable: false
+
+   Finished copying existing data from the collection(s).
+
+Your data from the ``PriceData`` MongoDB collection is now available in
+the ``marketdata.Stocks.PriceData`` Kafka topic.
+
+.. _time-series-sink-config:
+
+Configure the Sink Connector
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To consume data from the ``marketdata.Stocks.PriceData`` Kafka topic and write
+it to a time series collection named ``StockDataMigrate`` in a database
+named ``Stocks``, you can create the following source connector configuration:
+
+.. code-block:: json
+   :emphasize-lines: 12-14
+
+   {
+     "name": "mongo-sink-marketdata",
+     "config": {
+      "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
+       "tasks.max":"1",
+       "topics":"marketdata.Stocks.PriceData",
+       "connection.uri":"<your connection uri>",
+       "database":"Stocks",
+       "collection":"StockDataMigrate",
+       "key.converter":"org.apache.kafka.connect.storage.StringConverter",
+       "value.converter":"org.apache.kafka.connect.json.JsonConverter",
+       "timeseries.timefield":"tx_time",
+       "timeseries.timefield.auto.convert":"true",
+       "timeseries.timefield.auto.convert.date.format":"yyyy-MM-dd'T'HH:mm:ss'Z'"
+    }}
+
+.. tip::
+
+   The sink connector configuration above uses the time field date
+   format converter. Alternatively, you can use the ``TimestampConverter``
+   Single Message Transform (SMT) to convert the ``tx_time`` field from a
+   ``String`` to an ``ISODate``. When using the ``TimestampConverter`` SMT,
+   you must define a schema for the data in the Kafka topic.
+
+   For information on how to use the ``TimestampConverter`` SMT, see the
+   `TimestampConverter <https://docs.confluent.io/platform/current/connect/transforms/timestampconverter.html#timestampconverter>`__
+   Confluent documentation.
+
+After your sink connector finishes processing the topic data, the documents
+in the ``StockDataMigrate`` time series collection contain the ``tx_time``
+field with an ``ISODate`` type value.
+
+.. _time-series-verify-collection:
+
+Verify the Collection Data
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By this step, your time series collection should contain all the market data
+from your ``PriceData`` collection. The following shows the format of the
+documents in the ``StockDataMigrate`` time series collection:
+
+.. code-block:: javascript
+
+  {
+    tx_time: 2021-07-12T20:05:35.000Z,
+    symbol: 'WSV',
+    company_name: 'WORRIED SNAKEBITE VENTURES',
+    price: 21.22,
+    _id: ObjectId("...")
+  }
+
+To learn how to verify a collection is of type **timeseries**, see the
+instructions on how to :manual:`Check if a Collection is of Type Time Series </core/timeseries-collections/#check-if-a-collection-is-of-type-time-series>`
+in the MongoDB manual.
+

source/tutorials/migrate-time-series.txt

@@ -55,7 +55,8 @@ Sink Connector
 
 - Added support for :ref:`automatic time-series collection creation <sink-configuration-time-series>`
   in MongoDB 5.0 to efficiently store sequences of measurements over a period
-  of time
+  of time. Learn how to configure connectors to :ref:`<tutorial-migrate-time-series>`.
+
 - Improved the error logging for bulk write exceptions
 
 Source Connector