Skip to content

Commit e539b3b

Browse files
kanchana-mongodbkanchana-mongodb
authored
DOCSP-12069 Require HTTPS for HTTP stores (by default) (#72)
* DOCSP-12069 Require HTTPS for HTTP stores (by default) DOCSP-12069 updates * DOCSP-12069 update for tech review feedback DOCSP-12069 minor fix DOCSP-12069 minor fix DOCSP-12069 minor fix DOCSP-12069 minor fix * DOCSP-12069 updates for review feedback DOCSP-12069 minor fixes
1 parent 3d59ae7 commit e539b3b

File tree

6 files changed

+129
-13
lines changed

6 files changed

+129
-13
lines changed

‎source/includes/extracts-common-cli-params.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,27 @@ content: |
1414
1515
If omitted, {+data-lake-short+} attempts to detect the file type by
1616
processing a few bytes of the file.
17+
---
18+
ref: cli-param-allow-insecure
19+
content: |
20+
21+
Validates the scheme in the specified |url|\s. Value can be one of the
22+
following:
23+
24+
- ``true`` to allow insecure |http| scheme
25+
- ``false`` to only allow secure |https| scheme (default)
26+
27+
If true, {+adl+}:
28+
29+
- Does not verify the server's certificate chain and hostname.
30+
- Accepts any certificate with any hostname presented by the server.
31+
32+
.. warning::
33+
34+
If you set this to ``true``, your data might become vulnerable to a
35+
man-in-the-middle attack, which can compromise the confidentiality
36+
and integrity of your data. Set this to ``true`` only for testing
37+
and getting started with {+adl+}.
38+
39+
If omitted, defaults to ``false``.
1740
...

‎source/includes/extracts-common-conf-params.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,34 @@ content: |
2323
*Optional.* Maximum number of wildcard ``*`` collections in the database.
2424
Each wildcard collection can have only one data source. Value can be between
2525
``1`` and ``1000``, inclusive. If omitted, defaults to ``100``.
26+
---
27+
ref: param-allow-insecure
28+
content: |
29+
30+
*Optional.* Validates the scheme in the specified |url|\s. Value can be one
31+
of the following:
32+
33+
- ``true`` to allow insecure |http| scheme
34+
- ``false`` to only allow secure |https| scheme (default)
35+
36+
If true, {+adl+}:
37+
38+
- Does not verify the server's certificate chain and hostname.
39+
- Accepts any certificate with any hostname presented by the server.
40+
41+
.. warning::
42+
43+
If you set this to ``true``, your data might become vulnerable to a
44+
man-in-the-middle attack, which can compromise the confidentiality
45+
and integrity of your data. Set this to ``true`` only for testing
46+
and getting started with {+adl+}.
47+
48+
If omitted, defaults to ``false``. If set to ``false``, {+dl+} returns an
49+
error similar to the following if a specified URL contains insecure |http|
50+
scheme:
51+
52+
.. code-block:: sh
53+
:copyable: false
54+
55+
The insecure HTTP scheme is not supported by default - please add a "allowInsecure: true" flag to query from such URLs.
2656
...

‎source/query/query-data-lake.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,7 @@ define:
179179
"provider": "http",
180180
"urls": ["<url>"],
181181
"defaultFormat" : "<string>"
182+
"allowInsecure": <boolean>,
182183
}
183184
],
184185
"databases" : [
@@ -192,6 +193,7 @@ define:
192193
"storeName" : "<store-name>",
193194
"urls" : ["<url>"],
194195
"defaultFormat" : "<string>"
196+
"allowInsecure" : <boolean>,
195197
}
196198
]
197199
}
@@ -247,6 +249,7 @@ contain the settings that define:
247249
"provider": "http",
248250
"urls": ["<url>"],
249251
"defaultFormat" : "<string>"
252+
"allowInsecure": <boolean>,
250253
}
251254
],
252255
"databases" : [
@@ -269,6 +272,7 @@ contain the settings that define:
269272
"storeName" : "<store-name>",
270273
"urls" : ["<url>"],
271274
"defaultFormat" : "<string>"
275+
"allowInsecure" : <boolean>,
272276
}
273277
]
274278
}

‎source/reference/cli/collections/create-collections-views.txt‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ Syntax
9797

9898
.. code-block:: sh
9999

100-
db.runCommand({ "create" : "<collection-name>", "dataSources" : [{ "storeName" : "<store-name>", "urls" : [ "<url>" ], "defaultFormat" : "<file-extension>" }]})
100+
db.runCommand({ "create" : "<collection-name>", "dataSources" : [{ "storeName" : "<store-name>", "allowInsecure" : true|false, "urls" : [ "<url>" ], "defaultFormat" : "<file-extension>" }]})
101101

102102
.. tab:: Views
103103
:tabid: views
@@ -240,6 +240,11 @@ Parameters
240240

241241
.. list-table::
242242

243+
* - ``dataSources.allowInsecure``
244+
- boolean
245+
- .. include:: /includes/extracts/cli-param-allow-insecure.rst
246+
- no
247+
243248
* - ``dataSources.urls``
244249
- array of strings or empty array
245250
- The |url|\s of the publicly accessible data files. You
@@ -500,13 +505,14 @@ Examples
500505
.. tab:: HTTP Example
501506
:tabid: http
502507

503-
The ``http-collection`` collection includes a partition for
504-
each |url| in the collection.
508+
The ``airbnb`` collection includes a partition for
509+
each |url| in the collection. The ``allowInsecure`` flag is
510+
not set and defaults to ``false``.
505511

506512
.. code-block:: json
507513

508514
use sampleDB
509-
db.runCommand({ "create" : "http-collection", "dataSources" : [{ "storeName" : "http-store", "urls" : [ "https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews.json","https://atlas-data-lake.s3.amazonaws.com/json/sample_weatherdata/data.json" ], "defaultFormat" : ".json" }]})
515+
db.runCommand({ "create" : "airbnb", "dataSources" : [{ "storeName" : "http-store", "urls": ["https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews.json","https://atlas-data-lake.s3.amazonaws.com/json/sample_weatherdata/data.json"], "defaultFormat" : ".json" }]})
510516

511517
The previous command returns the following output:
512518

@@ -522,7 +528,7 @@ Examples
522528
:copyable: false
523529

524530
> show collections
525-
http-collection
531+
airbnb
526532
> db.runCommand({"storageGetConfig":1})
527533
{
528534
"ok" : 1,
@@ -540,10 +546,10 @@ Examples
540546
],
541547
"databases" : [
542548
{
543-
"name" : "http-db",
549+
"name" : "sampleDb",
544550
"collections" : [
545551
{
546-
"name" : "http-collection",
552+
"name" : "airbnb",
547553
"dataSources" : [
548554
{
549555
"storeName" : "http-store",

‎source/reference/cli/stores/create-store.txt‎

Lines changed: 40 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Syntax
6767

6868
.. code-block:: sh
6969

70-
db.runCommand({ createStore: <store-name>, provider: <storage-provider>, urls: [ <url> ], defaultFormat: <file-extension> })
70+
db.runCommand({ createStore: <store-name>, provider: <storage-provider>, allowInsecure: true|false, urls: [ <url> ], defaultFormat: <file-extension> })
7171

7272
.. _dl-create-store-cmd-params:
7373

@@ -179,6 +179,11 @@ Parameters
179179
.. list-table::
180180
:widths: 10 10 70 10
181181

182+
* - ``allowInsecure``
183+
- boolean
184+
- .. include:: /includes/extracts/cli-param-allow-insecure.rst
185+
- no
186+
182187
* - ``urls``
183188
- array of strings or an empty array
184189
- One or more publicly accessible |url|\s. You
@@ -253,6 +258,7 @@ fails, see :ref:`dl-create-store-cmd-errors` for recommended solutions.
253258
"store" : {
254259
"name" : "<store-name>",
255260
"provider" : "<storage-provider>",
261+
"allowInsecure" : true|false,
256262
"urls" : [
257263
"<url>"
258264
],
@@ -329,7 +335,7 @@ The following example uses the ``createStore`` command to create a new
329335
.. code-block:: json
330336

331337
use sample
332-
db.runCommand({ createStore: "myStore", provider: "http", urls: ["https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews.json"], defaultFormat: ".json" })
338+
db.runCommand({ createStore: "myStore", provider: "http", urls: ["https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews","http://example.mongodb.com/sampleData"], allowInsecure: true, defaultFormat: ".json" })
333339

334340
The previous command prints the following:
335341

@@ -342,9 +348,11 @@ The following example uses the ``createStore`` command to create a new
342348
"name" : "http-store",
343349
"provider" : "http",
344350
"urls" : [
345-
"https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews.json"
351+
"https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews",
352+
"http://example.mongodb.com/sampleData"
346353
],
347354
"defaultFormat" : ".json"
355+
"allowInsecure" : true
348356
}
349357
}
350358

@@ -383,3 +391,32 @@ If the command fails, it returns one of the following errors.
383391
}
384392

385393
**Solution:** Ensure that you specify a valid storage provider.
394+
395+
.. tabs::
396+
:hidden:
397+
398+
.. tab:: S3 Configuration
399+
:tabid: s3
400+
401+
.. tab:: Atlas Configuration
402+
:tabid: atlas
403+
404+
.. tab:: HTTP Configuration
405+
:tabid: http
406+
407+
**Reason:** One or more given |url|\s contain insecure |http| scheme.
408+
409+
.. code-block:: json
410+
:copyable: false
411+
412+
{
413+
"ok" : 0,
414+
"errmsg" : "store 'httpStore': the insecure HTTP scheme is not supported by default - please add a 'allowInsecure: true' flag to the store or datasource to query from such URLs, correlationID = 16332c6eddf7b32776fde638",
415+
"code" : 72,
416+
"codeName" : "InvalidOptions"
417+
}
418+
419+
**Solution:** Specify |url|\s with the secure |https| scheme. If the
420+
specified |url|\s have the insecure |http| scheme, set the ``allowInsecure`` flag to ``true``. Note that setting the
421+
``allowInsecure`` flag to ``true`` leaves your data vulnerable to
422+
man-in-the-middle attacks.

‎source/reference/format/data-lake-configuration.txt‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,7 @@ Click on the tab below to learn more about the {+dl+} configuration for that dat
205205
{
206206
"name" : "httpStore",
207207
"provider" : "http",
208+
"allowInsecure" : false,
208209
"urls" : [
209210
"https://www.datacenter-hardware.com/data.json",
210211
"https://www.datacenter-software.com/data.json"
@@ -221,6 +222,7 @@ Click on the tab below to learn more about the {+dl+} configuration for that dat
221222
"dataSources" : [
222223
{
223224
"storeName" : "httpStore",
225+
"allowInsecure" : false,
224226
"urls" : [
225227
"https://www.datacenter-metrics.com/data"
226228
],
@@ -274,7 +276,8 @@ to run federated queries.
274276
{
275277
"name" : "httpStore",
276278
"provider" : "http",
277-
"urls": [
279+
"allowInsecure" : false,
280+
"urls" [
278281
"https://www.datacenter-hardware.com/data.json",
279282
"https://www.datacenter-software.com/data.json"
280283
],
@@ -299,6 +302,7 @@ to run federated queries.
299302
},
300303
{
301304
"storeName" : "httpStore",
305+
"allowInsecure" : false,
302306
"urls": [
303307
"https://www.datacenter-metrics.com/data.json"
304308
],
@@ -426,8 +430,9 @@ The {+data-lake-short+} configuration has the following format:
426430
{
427431
"name" : "<string>",
428432
"provider": "<string>",
429-
"urls": ["<string>"],
430-
"defaultFormat" : "<string>"
433+
"defaultFormat" : "<string>",
434+
"allowInsecure": <boolean>,
435+
"urls": ["<string>"]
431436
}
432437
],
433438
"databases" : [
@@ -439,6 +444,7 @@ The {+data-lake-short+} configuration has the following format:
439444
"dataSources" : [
440445
{
441446
"storeName" : "<string>",
447+
"allowInsecure" : <boolean>,
442448
"urls" : ["<string>"],
443449
"defaultFormat" : "<string>"
444450
}
@@ -515,6 +521,7 @@ The {+data-lake-short+} configuration has the following format:
515521
{
516522
"name" : "<string>",
517523
"provider" : "<string>",
524+
"allowInsecure" " <boolean>,
518525
"urls" : ["<string>"],
519526
"defaultFormat" : "<string>"
520527
}
@@ -659,6 +666,10 @@ The {+data-lake-short+} configuration has the following format:
659666
.. tab:: HTTP
660667
:tabid: http
661668

669+
.. datalakeconf:: stores.[n].allowInsecure
670+
671+
.. include:: /includes/extracts/param-allow-insecure.rst
672+
662673
.. datalakeconf:: stores.[n].urls
663674

664675
*Optional.* Comma-separated list of publicly accessible
@@ -755,6 +766,7 @@ The {+data-lake-short+} configuration has the following format:
755766
"dataSources" : [
756767
{
757768
"storeName" : "<string>",
769+
"allowInsecure" : <boolean>,
758770
"urls" : ["<string>"],
759771
"defaultFormat" : "<string>"
760772
}
@@ -905,6 +917,10 @@ The {+data-lake-short+} configuration has the following format:
905917
.. tab:: HTTP
906918
:tabid: http
907919

920+
.. datalakeconf:: databases.[n].collections.[n].dataSources.[n].allowInsecure
921+
922+
.. include:: /includes/extracts/param-allow-insecure.rst
923+
908924
.. datalakeconf:: databases.[n].collections.[n].dataSources.[n].urls
909925

910926
*Optional*. Comma-separated list of publicly accessible |url|\s

0 commit comments

Comments
 (0)