PYTHON-2545 Test serverless proxy #664

prashantmital · 2021-07-07T01:38:23Z

TO DO:

Update spec tests once DRIVERS-1836 Skip tests with $out stage on serverless specifications#1036 is merged
Add logic to skip non unified format tests that use features not supported by serverless (i.e. collation and $out)

ShaneHarvey · 2021-07-07T22:45:16Z

.evergreen/config.yml


-tasks:
-
+task_groups:


Interesting. I haven't since "task_groups" before. I gather it's a way to add arbitrary setup/teardown code to an existing task?

Yes. I too, haven't used them before. I picked this right out of the sample in https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/config.yml

.evergreen/config.yml

test/__init__.py

ShaneHarvey · 2021-07-07T23:02:56Z

test/__init__.py

-            elif 'parsed' in self.cmd_line:
-                params = self.cmd_line['parsed'].get('setParameter', [])
-                if 'enableTestCommands=1' in params:
+                self.has_ipv6 = False


Serverless doesn't support ipv6?

I am not sure... It seems we only use this in one test (TestClient.test_ipv6) - do we want to run that test class against serverless?

Ah okay, I don't think we need to run that test on serverless. A maybe useful test would be to manually resolve the serverless host to ipv6 and then attempt to connect. Up to you. I don't think it's needed. Should be something that's tested on the cloud team's side.

ShaneHarvey · 2021-07-12T20:47:40Z

Looks like there are still a large number of failures:

...
 [2021/07/09 20:16:42.584] FAIL [0.916s]: test_retryable_reads_listIndexes_serverErrors_ListIndexes_succeeds_after_SocketException (test_retryable_reads.TestSpec)
 [2021/07/09 20:16:42.584] ----------------------------------------------------------------------
 [2021/07/09 20:16:42.584] Traceback (most recent call last):
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/__init__.py", line 534, in wrap
 [2021/07/09 20:16:42.584]     return f(*args, **kwargs)
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/__init__.py", line 534, in wrap
 [2021/07/09 20:16:42.584]     return f(*args, **kwargs)
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/test_retryable_reads.py", line 104, in run_scenario
 [2021/07/09 20:16:42.584]     self.run_scenario(scenario_def, test)
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/utils_spec_runner.py", line 557, in run_scenario
 [2021/07/09 20:16:42.584]     self.check_events(test, listener, session_ids)
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/utils_spec_runner.py", line 380, in check_events
 [2021/07/09 20:16:42.584]     len(res['started']), len(test['expectations']), cmds)
 [2021/07/09 20:16:42.584] AssertionError: 1 != 2 : [{'listIndexes': 'coll', 'cursor': {}, 'lsid': {'id': UUID('3045bd69-a342-4e1c-9442-ae30e0764147')}, '$clusterTime': {'clusterTime': Timestamp(1625861782, 4), 'signature': {'hash': b'\xbe[\xde\xe4%\xf8&\xd1\x84\xef_\xc1Y\x1c\xcc\x12\x0c0\xec\xe2', 'keyId': 6982211488521912322}}, '$db': 'retryable-reads-tests', '$readPreference': {'mode': 'primary'}}]
 [2021/07/09 20:16:42.584] ======================================================================
 [2021/07/09 20:16:42.584] FAIL [0.644s]: test_cluster_time_no_server_support (test_session.TestCausalConsistency)
 [2021/07/09 20:16:42.584] ----------------------------------------------------------------------
 [2021/07/09 20:16:42.584] Traceback (most recent call last):
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/__init__.py", line 534, in wrap
 [2021/07/09 20:16:42.584]     return f(*args, **kwargs)
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/test_session.py", line 1057, in test_cluster_time_no_server_support
 [2021/07/09 20:16:42.584]     self.assertIsNone(after_cluster_time)
 [2021/07/09 20:16:42.584] AssertionError: {'clusterTime': Timestamp(1625861787, 5), 'signature': {'hash': b"\xee\n<c\xa3\xce\xb0o\xf7\x18'P\x1c\x1d\x04\xe6\xcd\xb0\x1e\x8f", 'keyId': 6982211488521912322}} is not None
 [2021/07/09 20:16:42.584] ======================================================================
 [2021/07/09 20:16:42.584] FAIL [0.083s]: test_server_not_causal (test_session.TestCausalConsistency)
 [2021/07/09 20:16:42.584] ----------------------------------------------------------------------
 [2021/07/09 20:16:42.584] Traceback (most recent call last):
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/__init__.py", line 534, in wrap
 [2021/07/09 20:16:42.584]     return f(*args, **kwargs)
 [2021/07/09 20:16:42.584]   File "/data/mci/9b6fda5b61ca43196c9b54664e292149/src/test/test_session.py", line 1016, in test_server_not_causal
 [2021/07/09 20:16:42.584]     self.assertIsNone(act)
 [2021/07/09 20:16:42.584] AssertionError: Timestamp(1625861787, 6) is not None
 [2021/07/09 20:16:42.690] ----------------------------------------------------------------------
 [2021/07/09 20:16:42.690] Ran 2042 tests in 426.268s
 [2021/07/09 20:16:42.690] FAILED (failures=198, errors=77, skipped=762)

Waiting for those to be resolved before reviewing again.

ShaneHarvey · 2021-07-13T04:30:57Z

test/__init__.py

                    **self.default_client_options)

                # May not have this if OperationFailure was raised earlier.
                self.cmd_line = self.client.admin.command('getCmdLineOpts')


Does getCmdLineOpts actually work on serverless?

It works but doesn't give us any useful info:

self.client.admin.command('getCmdLineOpts') {'argv': [], 'parsed': {}, 'ok': 1}

ShaneHarvey · 2021-07-13T04:36:15Z

test/test_crud_unified.py

 # Generate unified tests.
-globals().update(generate_test_classes(TEST_PATH, module=__name__))
+globals().update(generate_test_classes(
+    TEST_PATH, module=__name__, RUN_ON_SERVERLESS=True))


Shouldn't RUN_ON_SERVERLESS be true for all unified tests?

Not as per the wording of the spec. It only needs to be enabled for spec tests pertaining to certain driver specifications.

ShaneHarvey · 2021-07-13T04:37:35Z

test/test_crud_v1.py

+                self.skipTest("MongoDB Serverless does not support $out")
+
+    if "collation" in test['operation']['arguments']:
+        self.skipTest("MongoDB Serverless does not support collations")


I thought they already updated the spec tests themselves. Is this needed anymore? Or did they only update the unified tests and not the v1 ones?

The changes to make tests serverless friendly were only for unified format.

ShaneHarvey · 2021-07-13T04:42:00Z

test/test_retryable_writes.py


 class TestWriteConcernError(IntegrationTest):
    RUN_ON_LOAD_BALANCER = True
+    RUN_ON_SERVERLESS = True


I think IgnoreDeprecationsTest also needs RUN_ON_SERVERLESS=True.

…nTest

prashantmital · 2021-07-27T00:01:17Z

@ShaneHarvey this is ready for final review.

ShaneHarvey

Can you ensure the test suite fails in some way if the driver cannot connect to the serverless instance when TEST_SERVERLESS==True? For example you could add an extra test to TestClientContext. Or is this case already covered?

ShaneHarvey · 2021-07-27T00:24:12Z

test/test_retryable_reads.py

+                                "MongoDB Serverless does not support $out")
+                if "collation" in operation['arguments']:
+                    self.skipTest(
+                        "MongoDB Serverless does not support collations")


DRIVERS-1836 updated the unified tests and DRIVERS-1851 updated the crud v1 tests. Is there any reason why we did not update the retryable reads/writes tests? Can you file a drivers ticket for it?

I think this is only a concern for Retrayble Reads. I opened DRIVERS-1869.

…VERLESS=1

ShaneHarvey · 2021-07-27T17:57:28Z

test/test_client_context.py

+        if 'TEST_SERVERLESS' not in os.environ:
+            raise SkipTest('TEST_SERVERLESS is not set')
+
+        self.assertTrue(client_context.connected,


Suggest client_context.connected and client_context.serverless? Or is that redundant?

Somewhat redundant as client_context.serverless is set solely based on the value of TEST_SERVERLESS but it doesn't hurt. Updated.

(cherry picked from commit f07da34)

prashantmital requested a review from ShaneHarvey July 7, 2021 01:54

ShaneHarvey reviewed Jul 7, 2021

View reviewed changes

prashantmital marked this pull request as ready for review July 8, 2021 00:38

prashantmital force-pushed the PYTHON-2545/test-proxy-as-mongos branch from 37d5f0c to cedc355 Compare July 12, 2021 23:48

ShaneHarvey reviewed Jul 13, 2021

View reviewed changes

prashantmital added 19 commits July 26, 2021 16:01

PYTHON-2545 Run test-suite against proxy as a mongos

258a0cb

Get trivial example working locally

3791da5

cleanup evg config

d4ae3bc

ensure driver is able to connect to serverless instance

5f069c6

remove debugging cruft

00ccaf2

Skip some crud V1 tests on serverless

dc09d95

reync spec tests

b0c5776

augment username/password information to URI

fd6ac53

Only run on unified format crud tests

199f751

skip more tests

8c2c812

ensure tests that require server connectivity inherit from Integratio…

8cab22c

…nTest

connect to a single mongos

893f728

skip changestream tests

c95cd66

skip Map Reduce tests

5fcfaa8

ensure runOn is honored

0bfdf88

skip currentOp tests on serverless

86f9a2e

skip out and collation on retryable reads spec tests

ea21540

only skip on serverless

153188e

update crud tests

82e3b1d

prashantmital force-pushed the PYTHON-2545/test-proxy-as-mongos branch from eb61ba8 to 82e3b1d Compare July 26, 2021 23:02

properly skip tests

2f8e405

prashantmital requested a review from ShaneHarvey July 27, 2021 00:00

run ignore deprecations tests on serverless

2df9fde

ShaneHarvey reviewed Jul 27, 2021

View reviewed changes

ensure test suite fails when not connected to serverless and TEST_SER…

5544ae0

…VERLESS=1

ShaneHarvey reviewed Jul 27, 2021

View reviewed changes

edit check

8232d17

prashantmital requested a review from ShaneHarvey July 27, 2021 23:06

ShaneHarvey approved these changes Jul 27, 2021

View reviewed changes

prashantmital merged commit f07da34 into mongodb:master Jul 27, 2021

prashantmital deleted the PYTHON-2545/test-proxy-as-mongos branch July 27, 2021 23:35

prashantmital added a commit to prashantmital/mongo-python-driver that referenced this pull request Jul 27, 2021

PYTHON-2545 Test Atlas Serverless (mongodb#664)

a89a68c

(cherry picked from commit f07da34)

prashantmital added a commit that referenced this pull request Jul 28, 2021

PYTHON-2545 Test Atlas Serverless (#664) (#680)

cd033e7

(cherry picked from commit f07da34)


		tasks:

		task_groups:

PYTHON-2545 Test serverless proxy #664

PYTHON-2545 Test serverless proxy #664

Uh oh!

Conversation

prashantmital commented Jul 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShaneHarvey commented Jul 12, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prashantmital commented Jul 27, 2021

Uh oh!

ShaneHarvey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

prashantmital commented Jul 7, 2021 •

edited

Loading