(DOCSP-18607) Usage Example Schema (#164)

Co-authored-by: Chris Cho <[email protected]>

Author: 2 people

snooty.toml

@@ -8,6 +8,8 @@ mkc = "MongoDB Kafka Connector"
 connector-long = "MongoDB Connector for Apache Kafka"
 ak = "Apache Kafka"
 kc = "Kafka Connect"
+avro = "Apache Avro"
+avro-short = "Avro"
 connector_version="1.6"
 connector_driver_version="4.3"
 connector_driver_url_base="https://docs.mongodb.com/drivers/java/sync/v{+connector_driver_version+}/"

source/includes/source-connector/usage-examples/customer.json

@@ -0,0 +1,15 @@
+{
+  "name": "Zola",
+  "visits": [
+    {
+      "$date": "2021-07-25T17:30:00.000Z"
+    },
+    {
+      "$date": "2021-10-03T14:06:00.000Z"
+    }
+  ],
+  "goods_purchased": {
+    "apples": 1,
+    "bananas": 10
+  }
+}

source/includes/source-connector/usage-examples/customers.avro

@@ -0,0 +1,24 @@
+{
+  "type": "record",
+  "name": "Customer",
+  "fields": [{
+      "name": "name",
+      "type": "string"
+    },{
+      "name": "visits",
+      "type": {
+        "type": "array",
+        "items": {
+          "type": "long",
+          "logicalType": "timestamp-millis"
+        }
+      }
+    },{
+      "name": "goods_purchased",
+      "type": {
+        "type": "map",
+        "values": "int"
+      }
+    }
+  ]
+}

source/includes/source-connector/usage-examples/schema.properties

@@ -0,0 +1,10 @@
+connector.class="com.mongodb.kafka.connect.MongoSourceConnector"
+connection.uri="<your MongoDB connection URI>"
+database="customers"
+collection="purchases"
+publish.full.document.only=true
+output.format.value="schema"
+output.schema.value="{\"type\": \"record\", \"name\": \"Customer\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}, {\"name\": \"visits\", \"type\": {\"type\": \"array\", \"items\": {\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}}}, {\"name\": \"goods_purchased\", \"type\": {\"type\": \"map\", \"values\": \"int\"}}]}"
+value.converter.schemas.enable=true
+value.converter="org.apache.kafka.connect.json.JsonConverter"
+key.converter="org.apache.kafka.connect.storage.StringConverter"

source/introduction/converters.txt

@@ -1,3 +1,5 @@
+.. _intro-converters:
+
 ==========
 Converters
 ==========

source/introduction/data-formats.txt

@@ -120,6 +120,8 @@ You use JSON Schema when you apply JSON Schema converters to your connectors.
 For more information, see the official 
 `JSON Schema website <https://json-schema.org/>`__.
 
+.. _data-formats-avro:
+
 Avro
 ----
 

source/source-connector/fundamentals/change-streams.txt

@@ -54,6 +54,8 @@ For a list of aggregation operators you can use with a change stream, see
 the guide on :manual:`Modify Change Stream Output <changeStreams/#modify-change-stream-output>`
 in the MongoDB manual.
 
+.. _source-connector-fundamentals-change-event:
+
 Change Event Structure
 ~~~~~~~~~~~~~~~~~~~~~~
 

source/source-connector/fundamentals/specify-schema.txt

@@ -1,3 +1,5 @@
+.. _kafka-source-apply-schemas:
+
 =============
 Apply Schemas
 =============
@@ -15,20 +17,20 @@ Overview
 --------
 
 In this guide, you can learn how to apply schemas to incoming
-documents in a {+mkc+} source connector. 
+documents in a {+mkc+} source connector.
 
-There are two types of schema in Kafka Connect, **key schema** and 
+There are two types of schema in Kafka Connect, **key schema** and
 **value schema**. Kafka Connect sends messages to Apache Kafka containing both
 your value and a key. A key schema enforces a structure for keys in messages
 sent to Apache Kafka. A value schema enforces a structure for values in messages
-sent to Apache Kafka. 
+sent to Apache Kafka.
 
 .. important:: Note on Terminology
 
    The word "key" has a slightly different meaning in the context
    of BSON and Apache Kafka. In BSON, a "key" is a unique string identifier for
-   a field in a document. 
-   
+   a field in a document.
+
    In Apache Kafka, a "key" is a byte array sent in a message used to determine
    what partition of a topic to write the message to. Kafka keys can be
    duplicates of other keys or ``null``.
@@ -49,7 +51,7 @@ following combinations of schemas:
 To see full properties files for specifying a schema, see our specify a schema
 usage example. <TODO: link to example>
 
-To learn more about keys and values in Apache Kafka, see the 
+To learn more about keys and values in Apache Kafka, see the
 `official Apache Kafka introduction <http://kafka.apache.org/intro#intro_concepts_and_terms>`__.
 
 Default Schemas
@@ -60,10 +62,10 @@ The {+mkc+} provides two default schemas:
 - :ref:`A key schema for the _id field of MongoDB change event documents. <source-default-key-schema>`
 - :ref:`A value schema for MongoDB change event documents. <source-default-value-schema>`
 
-To learn more about change events, see our 
+To learn more about change events, see our
 :doc:`guide on change streams </source-connector/fundamentals/change-streams>`.
 
-To learn more about default schemas, see the default schemas 
+To learn more about default schemas, see the default schemas
 :github:`here in the MongoDB Kafka Connector source code <mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/source/schema/AvroSchemaDefaults.java>`.
 
 .. _source-default-key-schema:
@@ -73,16 +75,16 @@ Key Schema
 
 The {+mkc+} provides a default key schema for the ``_id`` field of change
 event documents. You should use the default key schema unless you remove the
-``_id`` field from your change event document using either of the transformations 
-:ref:`described in this guide here <source-schema-for-modified-document>`. 
+``_id`` field from your change event document using either of the transformations
+:ref:`described in this guide here <source-schema-for-modified-document>`.
 
 If you specify either of these transformations and would like to use a key
 schema for your incoming documents, you must specify a key schema
-:ref:`as described in the specify a schema section of this guide <source-specify-avro-schema>`.  
+:ref:`as described in the specify a schema section of this guide <source-specify-avro-schema>`.
 
-You can enable the default key schema with the following option: 
+You can enable the default key schema with the following option:
 
-.. code-block:: java 
+.. code-block:: java
 
    output.format.key=schema
 
@@ -93,16 +95,16 @@ Value Schema
 
 The {+mkc+} provides a default value schema for change event documents. You
 should use the default value schema unless you transform your change event
-documents  
-:ref:`as described in this guide here <source-schema-for-modified-document>`. 
+documents
+:ref:`as described in this guide here <source-schema-for-modified-document>`.
 
 If you specify either of these transformations and would like to use a value schema for your
-incoming documents, you must use one of the mechanisms described in the 
-:ref:`schemas for transformed documents section of this guide <source-schema-for-modified-document>`. 
+incoming documents, you must use one of the mechanisms described in the
+:ref:`schemas for transformed documents section of this guide <source-schema-for-modified-document>`.
 
-You can enable the default value schema with the following option: 
+You can enable the default value schema with the following option:
 
-.. code-block:: properties 
+.. code-block:: properties
 
    output.format.value=schema
 
@@ -115,7 +117,7 @@ There are two ways you can transform your change event documents in a
 source connector:
 
 - The ``publish.full.document.only=true`` option
-- An aggregation pipeline that modifies the structure of change event documents 
+- An aggregation pipeline that modifies the structure of change event documents
 
 If you transform your MongoDB change event documents,
 you must do the following to apply schemas:
@@ -164,7 +166,7 @@ Infer a Schema
 
 You can have your source connector infer a schema for incoming documents. This
 option works well for development and for data sources that do not
-frequently change structure, but for most production deployments we recommend that you 
+frequently change structure, but for most production deployments we recommend that you
 :ref:`specify a schema <source-specify-avro-schema>`.
 
 You can have the MongoDB Kafka Connector infer a schema by specifying the
@@ -179,8 +181,8 @@ following options:
 
    The {+mkc+} does not support key schema inference. If you want to use a key
    schema and transform your MongoDB change event documents, you must specify a
-   key schema as described in 
-   :ref:`the specify schemas section of this guide <source-specify-avro-schema>`. 
+   key schema as described in
+   :ref:`the specify schemas section of this guide <source-specify-avro-schema>`.
 
 Properties Files
 ----------------

source/source-connector/usage-examples.txt

@@ -10,6 +10,5 @@ Source Connector Usage Examples
    Multiple Sources </source-connector/usage-examples/multiple-sources>
    Topic Naming </source-connector/usage-examples/topic-naming>
    Copy Existing Data </source-connector/usage-examples/copy-existing-data>
-   
+   Specify a Schema </source-connector/usage-examples/schema>
 
-asdf

source/source-connector/usage-examples/schema.txt

@@ -0,0 +1,152 @@
+.. _source-usage-example-schema:
+
+================
+Specify a Schema
+================
+
+.. default-domain:: mongodb
+
+This usage example demonstrates how you can configure your MongoDB Kafka
+source connector to apply a custom **schema** to your data. A schema is a
+definition that specifies the structure and type information about data in an
+{+ak+} topic. Use a schema when you need to ensure the data on the topic populated
+by your source connector has a consistent structure.
+
+To learn more about using schemas with the {+mkc+}, see the 
+:ref:`Apply Schemas <kafka-source-apply-schemas>` guide.
+
+Example
+-------
+
+Suppose your application keeps track of customer data in a MongoDB
+collection, and you need to publish this data to a Kafka topic. You want
+the subscribers of the customer data to receive consistently formatted data.
+You choose to apply a schema to your data.
+
+Your requirements and your solutions are as follows:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 50 50
+
+   * - Requirement
+     - Solution
+
+   * - Receive customer data from a MongoDB collection
+     - | Configure a MongoDB source connector to receive updates to data 
+         from a specific database and collection.
+       | See :ref:`<usage-example-schema-read-collection>`.
+
+   * - Provide the customer data schema
+     - | Specify a schema that corresponds to the structure and data types of
+         the customer data.
+       | See :ref:`<usage-example-schema-custom-schema>`.
+
+   * - Omit Kafka metadata from the customer data
+     - | Include only the data from the ``fullDocument`` field.
+       | See :ref:`<usage-example-schema-omit-metadata>`.
+
+For the full configuration file that meets the requirements above, see
+:ref:`<usage-example-schema-config>`.
+
+.. _usage-example-schema-read-collection:
+
+Receive Data from a Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To configure your source connector to receive data from a MongoDB collection,
+specify the database and collection name. For this example, you can
+configure the connector to read from the ``purchases`` collection in the
+``customers`` database as follows:
+
+.. code-block:: ini
+
+   database="customers"
+   collection="purchases"
+
+.. _usage-example-schema-custom-schema:
+
+Create a Custom Schema
+~~~~~~~~~~~~~~~~~~~~~~
+
+A sample customer data document from your collection contains the following
+information:
+
+.. literalinclude:: /includes/source-connector/usage-examples/customer.json
+   :language: json
+
+From the sample document, you decide your schema should present the fields
+using the following data types:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 30 45
+
+   * - Field name
+     - Data types
+     - Description
+
+   * - **name**
+     - `string <https://avro.apache.org/docs/current/spec.html#schema_primitive>`__
+     - | Name of the customer
+
+   * - **visits**
+     - `array <https://avro.apache.org/docs/current/spec.html#Arrays>`__
+       of `timestamps <https://avro.apache.org/docs/current/spec.html#Timestamp+%28millisecond+precision%29>`__
+     - Dates the customer visited
+
+   * - **goods_purchased**
+     - `map <https://avro.apache.org/docs/current/spec.html#Maps>`__
+       of string (the assumed type) to
+       `integer <https://avro.apache.org/docs/current/spec.html#schema_primitive>`__
+       values
+     - Names of goods and quantity of each item the customer purchased
+
+You can describe your data using the {+avro+} schema format as shown in
+the example schema below:
+
+.. literalinclude:: /includes/source-connector/usage-examples/customers.avro
+   :language: json
+
+.. important:: Converters
+
+   If you want to send your data through {+ak+} with {+avro-short+} binary encoding,
+   you must use an {+avro-short+} converter. For more information, see the guide on
+   :ref:`Converters <intro-converters>`.
+
+.. _usage-example-schema-omit-metadata:
+
+Omit Metadata from Published Records
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The connector publishes the customer data documents and metadata
+that describes the document to a Kafka topic. You can set the connector to
+include only the document data contained in the ``fullDocument`` field of the
+record using the following setting:
+
+.. code-block:: ini
+
+   publish.full.document.only=true
+
+For more information on the ``fullDocument`` field, see the 
+:ref:`Change Streams <source-connector-fundamentals-change-event>` guide.
+
+.. _usage-example-schema-config:
+
+Custom Schema Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Your custom schema connector configuration should resemble the following:
+
+.. literalinclude:: /includes/source-connector/usage-examples/schema.properties
+   :language: properties
+   :emphasize-lines: 3,4,5,6,7
+
+.. note:: Embedded Schema
+
+   In the preceding configuration, the JSON Schema converter embeds the custom
+   schema in your messages. To learn more about the JSON Schema converter, see the
+   :ref:`Converters <json-schema-converter-sample-properties>` guide.
+
+For more information on specifying schemas, see the :ref:`Apply
+Schemas <kafka-source-apply-schemas>` guide.