Skip to content

Commit 2a83931

Browse files
jeff-allen-mongojeff-allen-mongo
committed
(DOCSP-6533): Data preparation updates for saas (#238)
* (DOCSP-6533): Data preparation updates for saas * Updates per Steve's feedback * updates per Tom's feedback
1 parent 45ef5f2 commit 2a83931

8 files changed

+308
-48
lines changed

‎conf.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,8 @@
7878
'v0.10': ('https://docs.mongodb.com/charts/v0.10%s', ''),
7979
'atlas': ('https://docs.atlas.mongodb.com%s', ''),
8080
'stitch': ('https://docs.mongodb.com/stitch%s', ''),
81-
'onprem': ('https://docs.mongodb.com/charts/onprem%s', '')
81+
'onprem': ('https://docs.mongodb.com/charts/onprem%s', ''),
82+
'compass': ('https://docs.mongodb.com/compass/master%s', '')
8283
}
8384

8485
source_constants = {

‎source/data-source-pipeline.txt

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
.. _data-sources-pipeline:
2+
3+
=====================
4+
Data Source Pipelines
5+
=====================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 2
13+
:class: singlecol
14+
15+
If you are the :data:`Owner` of a data source, you can
16+
apply an :manual:`aggregation pipeline </core/aggregation-pipeline/>`
17+
to a data source to pre-process data before it reaches the
18+
chart builder.
19+
20+
Use Cases
21+
---------
22+
23+
Pre-processing a data source with a pipeline is useful when your
24+
collection data is not yet optimized for data visualization. While you
25+
can still pre-process your data by creating a
26+
:manual:`database view </core/views>` or using the |charts-short|
27+
:ref:`query bar <query-bar>`, adding a pipeline to a data source
28+
ensures that any charts using the data source receive the same
29+
data transformations.
30+
31+
Pre-processing your data can help accomplish the following tasks:
32+
33+
- Join data from multiple collections into a single data source using
34+
:pipeline:`$lookup`. :ref:`See example <join-data>`.
35+
36+
- Hide fields which may contain sensitive data from |charts-short| users
37+
using :pipeline:`$project`. :ref:`See example <hide-data>`.
38+
39+
- Convert data types, such as a string field to a date.
40+
:ref:`See example <convert-data-tye>`.
41+
42+
Add a Pipeline to a Data Source
43+
-------------------------------
44+
45+
.. include:: /includes/steps/add-data-source-pipeline.rst
46+
47+
Example Pipelines
48+
-----------------
49+
50+
.. _join-data:
51+
52+
Join Data from Two Collections
53+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
54+
55+
The following example adds a pipeline to a data source pointing to
56+
the ``sample_mflix.movies`` collection from the
57+
:atlas:`Sample Mflix Dataset </sample-data/sample-mflix/>` provided by
58+
|service|. The pipeline performs a :pipeline:`$lookup` to join data
59+
from the ``comments`` collection based on ``movie_id``. The result is
60+
a new field in this data source called ``comments`` containing data
61+
from a separate collection.
62+
63+
.. code-block:: js
64+
65+
[
66+
{
67+
$lookup:
68+
{
69+
from: 'comments',
70+
localField: '_id',
71+
foreignField: 'movie_id',
72+
as: 'comments'
73+
}
74+
}
75+
]
76+
77+
.. _hide-data:
78+
79+
Hide Sensitive Data
80+
~~~~~~~~~~~~~~~~~~~
81+
82+
The following example adds a pipeline to a data source pointing to the
83+
``sample_supplies.sales`` collection from the
84+
:atlas:`Sample Supply Store Dataset </sample-data/sample-supplies/>`
85+
provided by |service|. The pipeline hides the ``customer.email`` field,
86+
making that field unavailable to charts using this data source:
87+
88+
.. code-block:: js
89+
90+
[
91+
{
92+
$project: { "customer.email": 0 }
93+
}
94+
]
95+
96+
To learn more about the ``$project`` aggregation stage,
97+
see :pipeline:`$project`.
98+
99+
.. _convert-data-tye:
100+
101+
Convert the Type of a Data Field
102+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
103+
104+
The following example adds a pipeline to a data source containing a
105+
string field that we want to convert to a date. The pipeline adds a new
106+
field to the data source with the string field ``saleDate`` converted to
107+
a date field called ``saleDateAsDate``:
108+
109+
.. code-block:: js
110+
111+
[
112+
{
113+
$addFields: {
114+
saleDateAsDate: { $toDate: '$saleDate' }
115+
}
116+
}
117+
]
118+
119+
Modify an Existing Data Source Pipeline
120+
---------------------------------------
121+
122+
.. admonition:: Considerations
123+
:class: important
124+
125+
You can only modify pipelines on data sources of which you are
126+
an :data:`Owner`.
127+
128+
|charts-short| displays a warning if the data source's pipeline you
129+
attempt to modify is in use by at least one chart. Modifying a
130+
pipeline which is currently in use may result in charts using
131+
the pipeline rendering incorrectly or breaking entirely.
132+
133+
.. include:: /includes/steps/modify-data-source-pipeline.rst
134+
135+
Delete an Existing Data Source Pipeline
136+
---------------------------------------
137+
138+
.. admonition:: Considerations
139+
:class: important
140+
141+
You can only delete pipelines on data sources of which you are
142+
an :data:`Owner`.
143+
144+
|charts-short| displays a warning if the data source's pipeline you
145+
attempt to delete is in use by at least one chart. Deleting a
146+
pipeline which is currently in use may result in charts using
147+
the pipeline rendering incorrectly or breaking entirely.
148+
149+
.. include:: /includes/steps/delete-data-source-pipeline.rst

‎source/data-sources.txt

Lines changed: 30 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
:noprevnext:
2-
31
.. _data-sources:
42

53
============
@@ -28,12 +26,13 @@ the top navigation bar.
2826
Data Sources View
2927
-----------------
3028

31-
The :guilabel:`Data Sources` view displays existing data sources that
32-
the currently logged-in user has permission to view. The data sources
33-
are displayed in a tabular format with the following columns:
29+
The :guilabel:`Data Sources` list displays existing data sources
30+
for which the currently logged-in user has :data:`Author` or
31+
:data:`Owner` permissions. The data sources are displayed in a tabular
32+
format with the following columns:
3433

3534
.. figure:: /images/charts/data-sources-saas.png
36-
:alt: Diagram of the "Data Sources" view
35+
:alt: Diagram of the "Data Sources" list
3736
:figwidth: 750px
3837

3938
.. list-table::
@@ -43,29 +42,31 @@ are displayed in a tabular format with the following columns:
4342
* - Column
4443
- Description
4544

46-
* - :guilabel:`Deployment`
47-
- The MongoDB deployment containing the source database and
48-
collection. Identified by the |service| cluster associated
49-
with the |charts-short| instance.
45+
* - :guilabel:`Name`
46+
- Name of the data source. By default, this is the data source's
47+
corresponding database and collection name. To learn how to
48+
rename a data source, see :ref:`mod-data-source-alias`.
5049

51-
* - :guilabel:`Database`
52-
- The database containing the source collection or view.
50+
* - :guilabel:`Collection & Deployment`
51+
- Contains two pieces of information for the data source:
52+
53+
- Database and collection from which |charts-short| retrieves
54+
data for this data source.
5355

54-
* - :guilabel:`Collection`
55-
- The collection or view from where |charts-short| retrieves data.
56+
- Name of the |service| deployment which contains the
57+
collection.
5658

57-
* - :guilabel:`Alias`
58-
- Optional. The display name of the data source used by the
59-
Chart Builder. Providing an :guilabel:`Alias` may result in a
60-
shorter and more identifiable name when selecting a data source
61-
in the :doc:`chart builder </build-charts>`.
59+
* - :guilabel:`Applied Aggregations`
60+
- Aggregation stages applied to this data source. Data source
61+
:data:`Owners <Owner>` can define an
62+
:manual:`aggregation pipeline </core/aggregation-pipeline/>`
63+
to pre-process data before it reaches the chart builder.
6264

63-
To modify the :guilabel:`Alias` for a data source, see the
64-
:ref:`Modify a Data Source <mod-data-source-alias>`
65-
section.
65+
For more information on defining aggregation pipelines for a
66+
data source, see :ref:`data-sources-pipeline`.
6667

67-
* - :guilabel:`Last modified`
68-
- The date the data source was last modified.
68+
* - :guilabel:`Created`
69+
- When the data source was created.
6970

7071
* - :guilabel:`Permissions`
7172
- The icon next to the :guilabel:`Access` button denotes the
@@ -88,7 +89,7 @@ Filter Data Sources
8889
~~~~~~~~~~~~~~~~~~~
8990

9091
To search for specific data sources, use the search bar at the
91-
upper right corner of the :guilabel:`Data Sources` view. You can search
92+
upper right corner of the :guilabel:`Data Sources` list. You can search
9293
by :guilabel:`Deployment`, :guilabel:`Database`,
9394
:guilabel:`Collection`, or :guilabel:`Alias`.
9495

@@ -103,7 +104,7 @@ information on data source permissions, see
103104
Add a Data Source
104105
-----------------
105106

106-
1. From the :guilabel:`Data Sources` view, click the
107+
1. From the :guilabel:`Data Sources` list, click the
107108
:guilabel:`New Data Source` button.
108109

109110
#. Select one of the |service| clusters.
@@ -138,7 +139,7 @@ Add a Data Source
138139
#. Click :guilabel:`Publish Data Source`.
139140

140141
You will see your newly added data source listed in the
141-
:guilabel:`Data Sources` view. To learn how to manage existing
142+
:guilabel:`Data Sources` list. To learn how to manage existing
142143
data sources, see :doc:`manage-data-sources`.
143144

144145
Read Preference
@@ -154,12 +155,13 @@ the newly created data source has the same read preference as the
154155
existing data source.
155156

156157
You can :ref:`change the read preference <mod-data-source-permissions>`
157-
of a data source in the :guilabel:`Data Sources` view.
158+
of a data source in the :guilabel:`Data Sources` list.
158159

159160
.. class:: hidden
160161

161162
.. toctree::
162163
:titlesonly:
163164

165+
/data-source-pipeline
164166
/manage-data-sources
165167
/data-source-permissions

‎source/images/charts/data-sources-saas.png

154 KB
Loading

‎source/includes/steps-add-data-source-pipeline.yaml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
title: Open the :guilabel:`Data Sources` view.
2+
ref: open-data-sources
3+
level: 4
4+
stepnum: 1
5+
content: |
6+
Click :guilabel:`Data Sources` at the top of |charts|.
7+
---
8+
title: "For the data source you wish to pre-process, click
9+
:guilabel:`Add Pipeline` in the :guilabel:`Applied Aggregations`
10+
column."
11+
ref: click-applied-aggregations
12+
level: 4
13+
---
14+
title: "In the :guilabel:`Applied Aggregation Pipeline` modal, create
15+
your aggregation pipeline."
16+
ref: create-pipeline
17+
level: 4
18+
content: |
19+
Your pipeline must be an array in square brackets. Specify each
20+
stage of your pipeline as an object within the array.
21+
22+
|charts| validates your pipeline before you save. |charts-short|
23+
reports any errors in your pipeline below the editor.
24+
25+
.. tip::
26+
27+
For help creating your pipeline, you can utilize the
28+
:atlas:`Aggregation Pipeline Builder
29+
</data-explorer/cloud-agg-pipeline/>` in the |service| Data
30+
Explorer.
31+
32+
You can create your pipeline using the |service| pipeline builder
33+
and paste it into the |charts| pipeline editor.
34+
---
35+
title: Click :guilabel:`Apply to Data Source`.
36+
ref: click-apply
37+
level: 4
38+
39+
...

‎source/includes/steps-delete-data-source-pipeline.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
title: Open the :guilabel:`Data Sources` view.
2+
ref: open-data-sources
3+
level: 4
4+
stepnum: 1
5+
content: |
6+
Click :guilabel:`Data Sources` at the top of |charts|.
7+
---
8+
title: "For the pipeline you want to modify, click the :guilabel:`Gear`
9+
icon."
10+
ref: click-gear
11+
level: 4
12+
---
13+
title: "Click :guilabel:`Delete Pipeline`."
14+
ref: click-delete
15+
level: 4
16+
---
17+
title: Click :guilabel:`Delete` to confirm.
18+
ref: confirm-delete
19+
level: 4
20+
21+
...

‎source/includes/steps-modify-data-source-pipeline.yaml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
title: Open the :guilabel:`Data Sources` view.
2+
ref: open-data-sources
3+
level: 4
4+
stepnum: 1
5+
content: |
6+
Click :guilabel:`Data Sources` at the top of |charts|.
7+
---
8+
title: "Open the pipeline modification dialog."
9+
ref: open-pipeline-dialog
10+
level: 4
11+
content: |
12+
a. For the pipeline you want to modify, click the
13+
:guilabel:`Gear` icon.
14+
15+
#. Click :guilabel:`Modify Pipeline`.
16+
---
17+
title: "In the :guilabel:`Applied Aggregation Pipeline` modal, make
18+
the desired changes to your pipeline."
19+
ref: edit-pipeline
20+
level: 4
21+
content: |
22+
Your pipeline must be an array in square brackets. Specify each
23+
stage of your pipeline as an object within the array.
24+
25+
|charts| validates your pipeline before you save. |charts-short|
26+
reports any errors in your pipeline below the editor.
27+
28+
.. tip::
29+
30+
For help creating your pipeline, you can utilize the
31+
:compass:`Aggregation Pipeline Builder
32+
</aggregation-pipeline-builder/>` in
33+
`MongoDB Compass <https://www.mongodb.com/products/compass>`__.
34+
35+
You can create your pipeline using Compass's pipeline builder
36+
and paste it into the |charts| pipeline editor.
37+
---
38+
title: Click :guilabel:`Apply to Data Source`.
39+
ref: click-apply
40+
level: 4
41+
42+
...

0 commit comments

Comments
 (0)