DOCSP-14317 Add SQL schema generation for wildcard collections (#124)

* DOCSP-14317 Add SQL schema generation for wildcard collections

* DOCSP-14317 updates for review feedback

Author: kanchana-mongodb

source/admin/query-with-sql.txt

@@ -10,54 +10,64 @@ Querying with SQL
 
 {+adl+} supports SQL format queries through the :ref:`JDBC driver 
 <jdbc-driver>` for {+adl+} and using the :ref:`adl-sql-stage` 
-:ref:`aggregation pipeline <adl-aggregation-pipeline>` stage. To support 
-SQL format queries, {+adl+} automatically creates a |json| schema that maps 
-to a relational schema of columns, tables, and databases for all new 
-collections, except wildcard (``*``) collections, and views in the {+dl+} 
-storage configuration. To learn more about the schema, see 
+:ref:`aggregation pipeline <adl-aggregation-pipeline>` stage. To 
+support SQL format queries, {+adl+} automatically creates a |json| 
+schema that maps to a relational schema of columns, tables, and 
+databases for all new collections and views in the {+dl+} storage 
+configuration. To learn more about the schema, see 
 :ref:`sql-schema-format`.
 
-{+dl+} automatically generates a schema for a new non-wildcard collection or 
-view in the storage configuration when you: 
+{+dl+} automatically generates a schema for a collection or view in the 
+storage configuration when you: 
 
 - :ref:`Create <dl-create-collection-views-cmd>` the collection or view 
   in the storage configuration.
 
-- :ref:`Rename <dl-rename-collection-cmd>` a collection or view that does not 
-  already have a schema. If you rename a collection or view that already has 
-  a schema, the schema is also renamed. {+dl+} does not generate a new schema 
-  for a renamed collection or view if it already exists.
+- :ref:`Rename <dl-rename-collection-cmd>` a collection or view that 
+  does not already have a schema. If you rename a collection or view 
+  that already has a schema, the schema is also renamed. {+dl+} does 
+  not generate a new schema for a renamed collection or view if it 
+  already exists.
 
 - :ref:`Set <datalake-setstorageconfig>` the storage configuration.
 
+In addition, for wildcard (``*``) collections, {+dl+} generates a 
+schema when it discovers the collections in the :ref:`namespace catalog 
+<manage-ns-catalog-cli>` for the wildcard (``*``) collections.
+
 .. include:: /includes/fact-schema-for-existing-collections.rst
 
-By default, {+dl+} samples data from only one randomly selected document in 
-your non-wildcard collection or view to generate a |json| schema. If your 
-collection or view contains polymorphic data, you can provide a larger 
-sampling size to {+dl+} to generate a new schema or you can manually 
-construct and set the schema. 
-
-You can manually generate schemas for all collections and views using the 
-:ref:`sqlgenerateschema-cmd` command, set or update the schema for your 
-collections or views using the :ref:`sqlsetschema-cmd` command, and view 
-the stored schema using the :ref:`sqlgetschema-cmd` command. 
-
-Once the SQL schema is set up, you can query your {+adl+} collections or views 
-through the :ref:`JDBC driver <jdbc-driver>` for {+adl+} and using the 
-:ref:`adl-sql-stage` :ref:`aggregation pipeline <adl-aggregation-pipeline>` 
-stage. 
-
-You can manually delete a schema for a collection or view by running the 
-:ref:`sqlsetschema-cmd` command with an empty schema document. {+dl+} 
-automatically removes the schema for a collection or view when you: 
-
-- :ref:`Drop the collection or view <dl-drop-collection-views-cmd>` from the 
-  storage configuration.
-- :ref:`Modify the storage configuration <datalake-setstorageconfig>` to 
-  remove the collection or view from the storage configuration.
-- :ref:`Drop the database <dl-drop-database-cmd>` that contains the collection 
-  or view from the storage configuration.
+By default, {+dl+} samples data from only one randomly selected 
+document in your collection or view to generate a |json| schema. If 
+your collection or view contains polymorphic data, you can provide a 
+larger sampling size to {+dl+} to generate a new schema or you can 
+manually construct and set the schema. 
+
+You can manually generate schemas for all collections and views using 
+the :ref:`sqlgenerateschema-cmd` command, set or update the schema for 
+your collections or views using the :ref:`sqlsetschema-cmd` command, 
+and view the stored schema using the :ref:`sqlgetschema-cmd` command. 
+
+Once the SQL schema is set up, you can query your {+adl+} collections 
+or views through the :ref:`JDBC driver <jdbc-driver>` for {+adl+} and 
+using the :ref:`adl-sql-stage` :ref:`aggregation pipeline 
+<adl-aggregation-pipeline>` stage. 
+
+You can manually delete a schema for a collection or view by running 
+the :ref:`sqlsetschema-cmd` command with an empty schema document. 
+{+dl+} automatically removes the schema for a collection or view when 
+you: 
+
+- :ref:`Drop the collection or view <dl-drop-collection-views-cmd>` 
+  from the storage configuration.
+- :ref:`Modify the storage configuration <datalake-setstorageconfig>` 
+  to remove the collection or view from the storage configuration.
+- :ref:`Drop the database <dl-drop-database-cmd>` that contains the 
+  collection or view from the storage configuration.
+
+In addition, for a wildcard (``*``) collection, {+dl+} deletes the 
+schema when it discovers that the collection has been removed from 
+the :ref:`namespace catalog <manage-ns-catalog-cli>`.
 
 .. toctree::
    :titlesonly:

source/includes/fact-schema-for-existing-collections.rst

@@ -1,10 +1,13 @@
 .. note:: 
 
    {+dl+} automatically generates schemas for only new collections and 
-   views in the :ref:`storage configuration <datalake-configuration-file>`.
-   Existing :manual:`namespaces </reference/limits/#faq-dev-namespace>` 
-   will not have auto-generated schemas. If you want {+dl+} to automatically 
-   generate schemas for your existing non-wildcard collections and views in 
-   the storage configuration, :ref:`remove <datalake-setstorageconfig>` the :datalakeconf:`databases` in your {+dl+} storage configuration and then 
-   :ref:`update <datalake-setstorageconfig>` your {+dl+} storage 
+   views in the :ref:`storage configuration 
+   <datalake-configuration-file>` or :ref:`namespace catalog 
+   <manage-ns-catalog-cli>`. Existing :manual:`namespaces 
+   </reference/limits/#faq-dev-namespace>` will not have auto-generated 
+   schemas. If you want {+dl+} to automatically generate schemas for 
+   your existing collections and views in the storage configuration, 
+   :ref:`remove <datalake-setstorageconfig>` the 
+   :datalakeconf:`databases` in your {+dl+} storage configuration and 
+   then :ref:`update <datalake-setstorageconfig>` your {+dl+} storage 
    configuration with the old configuration.

source/reference/cli/sql/sqlgenerateschema.txt

@@ -142,6 +142,7 @@ more about the fields in the output, see
 :ref:`sqlgenerateschema-output`.
 
 .. code-block:: json 
+   :copyable: false
 
    {
 	 "ok" : 1,
@@ -218,6 +219,7 @@ more about the fields in the output, see
 :ref:`sqlgenerateschema-output`.
 
 .. code-block:: json 
+   :copyable: false
 
    {
 	 "ok" : 1,

source/reference/format/sql-schema-format.txt

@@ -19,29 +19,31 @@ SQL Schema Format
 Overview 
 --------
 
-To query data using SQL, {+adl+} needs to be aware of the schema for that 
-data. {+dl+} automatically generates a |json| schema for all new collections, 
-except wildcard collections, and views. To learn more about auto-generated 
+To query data using SQL, {+adl+} needs to be aware of the schema for 
+that data. {+dl+} automatically generates a |json| schema for all new 
+collections and views. To learn more about auto-generated 
 schemas, see :ref:`query-with-sql`.
 
-By default, {+dl+} samples data from a single document in your collection 
-or view to generate a |json| schema. If your collection or view contains 
-polymorphic data, you can provide a larger sampling size to {+dl+} when 
-manually :ref:`generating <sqlgenerateschema-cmd>` the schema. 
-
-{+dl+} maps |json| schemas to relational schemas. MongoDB's :manual:`flexible 
-schema model </core/data-modeling-introduction/>` allows a given field to 
-contain data of multiple types, while relational databases restrict columns to 
-a single data type. The following sections describe the fields supported in 
-the |json| schema, the |bson| types that are supported in a relational schema, 
-and how {+dl+} resolves conflicts for polymorphic fields when mapped to 
+By default, {+dl+} samples data from a single document in your 
+collection or view to generate a |json| schema. If your collection or 
+view contains polymorphic data, you can provide a larger sampling size 
+to {+dl+} when manually :ref:`generating <sqlgenerateschema-cmd>` the 
+schema. 
+
+{+dl+} maps |json| schemas to relational schemas. MongoDB's 
+:manual:`flexible schema model </core/data-modeling-introduction/>` 
+allows a given field to contain data of multiple types, while 
+relational databases restrict columns to a single data type. The 
+following sections describe the fields supported in the |json| schema, 
+the |bson| types that are supported in a relational schema, and how 
+{+dl+} resolves conflicts for polymorphic fields when mapped to
 relational schema.
 
 |json| Schema Format 
 --------------------
 
-The schema for a collection is a document with two fields: ``jsonSchema`` 
-and ``version``. 
+The schema for a collection is a document with two fields:  
+``jsonSchema`` and ``version``. 
 
 .. code-block:: json 
 
@@ -50,9 +52,9 @@ and ``version``.
 		 "jsonSchema" : {}
 	 }
 
-The ``version`` field represents the version of the schema format used by 
-the document and the value is always ``1``. The ``jsonSchema`` field is 
-a document that describes the schema of the :manual:`namespace 
+The ``version`` field represents the version of the schema format used 
+by the document and the value is always ``1``. The ``jsonSchema`` field 
+is a document that describes the schema of the :manual:`namespace 
 </reference/limits/#faq-dev-namespace>`. 
 
 .. _sql-json-schema-fields:
@@ -77,18 +79,18 @@ Supported BSON Types
 {+dl+} only supports the following |bson| types when mapping |json| 
 schema to relational schema:
 
-- ``double``
-- ``string``
-- ``object``
 - ``array``
 - ``binData``
-- ``objectId``
 - ``bool``
 - ``date``
-- ``null``
+- ``decimal``
+- ``double``
 - ``int``
 - ``long``
-- ``decimal``
+- ``null``
+- ``object``
+- ``objectId``
+- ``string``
 
 Other types are ignored in the relational schema. Fields with 
 composite types, such as objects and arrays, are handled specially. 
@@ -97,8 +99,8 @@ Object Fields
 ~~~~~~~~~~~~~
 
 Object fields are flattened such that each nested field maps to its 
-own column in the relational schema. For example, consider the following 
-``eg`` collection: 
+own column in the relational schema. For example, consider the 
+following ``eg`` collection: 
 
 .. code-block:: json 
 
@@ -237,12 +239,13 @@ are representations of the above schema:
 Type Conversion Conflicts 
 -------------------------
 
-MongoDB's :manual:`flexible schema model </core/data-modeling-introduction/>` 
-allows a given field to contain data of multiple types, while relational 
-databases restrict columns to a single data type. When {+dl+} maps the 
-|json| schema to relational schema, type conflicts can occur if a field 
-is polymorphic. There are two main categories of type conversion conflicts 
-that might occur when there are multiple data types: 
+MongoDB's :manual:`flexible schema model 
+</core/data-modeling-introduction/>` allows a given field to contain 
+data of multiple types, while relational databases restrict columns to 
+a single data type. When {+dl+} maps the |json| schema to relational 
+schema, type conflicts can occur if a field is polymorphic. There are 
+two main categories of type conversion conflicts that might occur when 
+there are multiple data types: 
 
 - Conflicts between scalar types
 - Conflicts involving composite types like documents and arrays
@@ -281,9 +284,9 @@ Document Conflicts
 ##################
 
 When a conflict occurs involving a document, {+dl+} displays the fields 
-of the document type as separate columns using dot notation. For example, 
-consider a collection named ``conflict`` that contains the following 
-documents:
+of the document type as separate columns using dot notation. For 
+example, consider a collection named ``conflict`` that contains the 
+following documents:
 
 .. code-block:: javascript
    :copyable: false