DRIVERS-1934: withTransaction API retries too frequently #1851

sleepyStick · 2025-10-17T23:04:58Z

Please complete the following before merging:

Is the relevant DRIVERS ticket in the PR title?

Update changelog.
Test changes in at least one language driver.
Test these changes against all server versions and topologies (including standalone, replica set, and sharded
clusters).

sleepyStick · 2025-10-17T23:06:30Z

source/transactions-convenient-api/transactions-convenient-api.md

 callbacks.

-A previous design had no limits for retrying commits or entire transactions. The callback is always able indicate that
+A previous design had no limits for retrying commits or entire transactions. The callback is always able to indicate that


Maybe I read it wrong, or maybe its a typo? Not sure.

sleepyStick · 2025-10-17T23:08:32Z

source/transactions-convenient-api/transactions-convenient-api.md

        [abortTransaction](../transactions/transactions.md#aborttransaction) on the session.
    2. If the callback's error includes a "TransientTransactionError" label and the elapsed time of `withTransaction` is
-        less than 120 seconds, jump back to step two.
+        less than 120 seconds, sleep for `jitter * min(BACKOFF_INITIAL * (1.25**retry), BACKOFF_MAX)` where: 


I know python uses ** as the exponential operator. I don't believe that is a standard across other programming languages. Is it clear that its exponent? Is there a different preferred symbol for exponent? (I know math commonly uses ^ but it typically also means bitwise XOR in code so I felt like that could be confusing.)

** seems fine to me, but I might also be biased because that's what Node does 🙂. But I don't think people will have trouble understanding what this means.

ShaneHarvey · 2025-10-20T20:47:35Z

source/transactions-convenient-api/tests/README.md

+### Retry Backoff is Enforced
+
+Drivers should test that retries within `withTransaction` do not occur immediately. Ideally, set BACKOFF_INITIAL 500ms
+and configure a failpoint that forces one retry. Ensure that the operation took more than 500ms so succeed.


This test would not work as described because jitter is non-deterministic.

An alternative test would be to use a failpoint to fail the transaction X times and then assert the overall time is larger than some threshold.

Yeah, I realized that when implementing it. I set the fail point to fail 3 times and that seems to be consistently working (and without jitter failing 3 times would still cause the success to happen within 500ms)

update: it was consistent locally but super flakey on atlas. I modified the with transaction code to append backoff values to make this easier to test and the test is now more stable.

source/transactions-convenient-api/transactions-convenient-api.md

source/transactions-convenient-api/tests/README.md

source/transactions-convenient-api/transactions-convenient-api.md

ShaneHarvey · 2025-11-13T23:12:15Z

source/transactions-convenient-api/tests/README.md

+    },
+    "data": {
+        "failCommands": ["commitTransaction"],
+        "errorCode": 24,


Can we use this error code instead for these tests?:

code: 251, name: NoSuchTransaction

Changed in f40529f

ShaneHarvey · 2025-11-13T23:19:31Z

source/transactions-convenient-api/tests/README.md

+```
+
+Let the callback for the transaction be a simple `insertOne` command. Check that the total time for all retries exceeded
+3.5 seconds.


These two tests are identical. What if we run this test twice, first with backoff disabled (eg jitter set to always be 0%) and second with it enabled and jitter set to always be 100%. Then we can assert the second attempt takes 3 seconds longer?

Ooo i like this idea! Changed in f40529f

I decreased the number of failures to 13 so the test wouldn't take as long (resulting in total backing time to be 2.2 seconds) and added a +/- 1 second window for the difference between the two attempts.

…f takes longer

ShaneHarvey · 2025-11-18T21:05:41Z

source/transactions-convenient-api/tests/README.md

+```
+
+Let the callback for the transaction be a simple `insertOne` command. Let `no_backoff_time` be the time it took for the
+command to succeed.


"command" -> "withTransaction call"

changed in 56e88f0

ShaneHarvey · 2025-11-18T21:09:12Z

source/transactions-convenient-api/tests/README.md

+return `1`. Set the fail point to force 13 retries using the same command as before. Using the same callback as before,
+check that the total time for the withTransaction command is within +/-1 second of `no_backoff_time` plus 2.2 seconds.
+Note that 2.2 seconds is the sum of backoff 13 consecutive backoff values and the 1-second window is just to account for
+potential networking differences between the two runs.


minor nit for tone:

"the 1-second window is just to account for
potential networking differences between the two runs."
->
"the 1-second window accounts for potential variance between the two runs.

changed in 56e88f0

ShaneHarvey · 2025-11-18T21:10:54Z

source/transactions-convenient-api/transactions-convenient-api.md

 This method can be expressed by the following pseudo-code:

 ```typescript
+var BACKOFF_INITIAL = 1  // 1ms initial backoff


Let's update this to use 5ms as described in the tech doc. And the growth rate as well.

changed in 56e88f0

source/transactions-convenient-api/transactions-convenient-api.md

source/transactions-convenient-api/tests/README.md

source/transactions-convenient-api/transactions-convenient-api.md

baileympearson · 2025-11-19T23:02:39Z

source/transactions-convenient-api/tests/README.md

+        > [!NOTE]
+        > errorCode 251 is NoSuchTransaction.


This isn't formatting for me? Maybe it's something to do with the indentation?

hmm wacky? it seemed to render in my preview in pycharm when I was editing this so i assumed it worked. I'll play around with the indentation after looking into CSOT. worse case i'll be boring and just have this note be normal text lol

you were right about it being indentation! but unindenting it messes with the formatting of the bullets below it so no fancy alerts this time ;-;
i gotta keep this fancy alerts thing in mind for a future opportunity -- I want to use it now haha
(made it a normal comment in 1f04505)

source/transactions-convenient-api/tests/README.md

add exponential backoff to withTransactions API

25759c4

sleepyStick commented Oct 17, 2025

View reviewed changes

run pre-commit

bdcd2ef

ShaneHarvey requested changes Oct 20, 2025

View reviewed changes

sleepyStick added 2 commits October 20, 2025 16:07

add design rational for backoff

48890a2

fix prose test

71ba1ba

sleepyStick commented Oct 23, 2025

View reviewed changes

source/transactions-convenient-api/tests/README.md Outdated Show resolved Hide resolved

sleepyStick requested a review from ShaneHarvey October 23, 2025 17:30

ShaneHarvey requested changes Oct 24, 2025

View reviewed changes

source/transactions-convenient-api/tests/README.md Outdated Show resolved Hide resolved

baileympearson requested changes Oct 27, 2025

View reviewed changes

source/transactions-convenient-api/transactions-convenient-api.md Outdated Show resolved Hide resolved

sleepyStick added 2 commits October 27, 2025 15:07

fix pseudocode

b602606

fix test

057fbbf

baileympearson reviewed Oct 29, 2025

View reviewed changes

source/transactions-convenient-api/transactions-convenient-api.md Show resolved Hide resolved

sleepyStick added 3 commits October 29, 2025 15:38

account for CSOT / timeoutMS in algorithm

42e4d94

add more details to tests

c11aef8

add second test that is deterministic

a6b7b95

sleepyStick marked this pull request as ready for review November 12, 2025 21:44

sleepyStick requested a review from a team as a code owner November 12, 2025 21:44

sleepyStick requested review from alcaeus and removed request for a team November 12, 2025 21:44

alcaeus requested review from durran and removed request for alcaeus November 13, 2025 08:18

sleepyStick requested review from ShaneHarvey and baileympearson November 13, 2025 21:59

ShaneHarvey reviewed Nov 13, 2025

View reviewed changes

change test to use no backoff as baseline time and ensure with backof…

f40529f

…f takes longer

sleepyStick requested a review from ShaneHarvey November 14, 2025 00:49

ShaneHarvey requested changes Nov 18, 2025

View reviewed changes

baileympearson requested changes Nov 18, 2025

View reviewed changes

sleepyStick added 2 commits November 18, 2025 16:43

address comments pt 1

56e88f0

address comments pt 2

bc9153e

sleepyStick requested review from ShaneHarvey and baileympearson November 19, 2025 01:42

baileympearson requested changes Nov 19, 2025

View reviewed changes

sleepyStick requested a review from comandeo-mongo November 20, 2025 18:38

sleepyStick added 2 commits November 20, 2025 14:51

update constants (oops) and CSOT changes

5c74235

gh fancy alerts can't be indented ;-;

1f04505

DRIVERS-1934: withTransaction API retries too frequently #1851

Are you sure you want to change the base?

DRIVERS-1934: withTransaction API retries too frequently #1851

Conversation

sleepyStick commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sleepyStick Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sleepyStick commented Oct 17, 2025 •

edited

Loading

sleepyStick Nov 13, 2025 •

edited

Loading