From bab70bd1a1e0a7e11fa6ffb2ed779a480735bdb9 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Wed, 16 Jul 2025 13:02:32 +0300 Subject: [PATCH 1/6] Add recommendations on non-idempotent operations and `request_timeout` param Fixes #5242 Fixes #5286 --- doc/platform/sharding/vshard_admin.rst | 87 +++++++++++++++++++ .../reference_rock/vshard/vshard_router.rst | 6 ++ 2 files changed, 93 insertions(+) diff --git a/doc/platform/sharding/vshard_admin.rst b/doc/platform/sharding/vshard_admin.rst index ae338eca3..0ce6255f7 100644 --- a/doc/platform/sharding/vshard_admin.rst +++ b/doc/platform/sharding/vshard_admin.rst @@ -551,6 +551,93 @@ In a router application, you can define the ``put`` function that specifies how Learn more at :ref:`vshard-process-requests`. +.. _vshard-deduplication: + +Deduplication of non-idempotent requests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Idempotent requests** produce the same result every time they are executed. +For example, a data read request or a multiplication by one are both idempotent. +Therefore, incrementing by one is an example of a non-idempotent operation. +When such an operation is applied again, the value for the field increases by 2 instead of just 1. + +.. note:: + + Any write requests that are intended to be executed repeatedly should be idempotent. + The operations' idempotency ensures that the change is applied **only once**. + +A request may need to be run again if an error occurs on the server or client side. +In this case: + +- Read requests can be executed repeatedly. + For this purpose, :ref:`vshard.router.call() ` (with ``mode=read``) uses the ``request_timeout`` parameter + (since ``vshard`` 0.1.28). + It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: + + .. code-block:: text + + timeout > request_timeout + + + For example, if ``timeout = 10`` and ``request_timeout = 2``, + within 10 seconds the router is able to make 5 attempts (2 seconds each) to send a request to different replicas + until the request finally succeeds. + +- Write requests (:ref:`vshard.router.callrw() `) generally **cannot be re-executed** without verifying + that they have not been applied before. + Lack of such a check might lead to duplicate records or unplanned data changes. + + For example, a client has sent a request to the server. The client is waiting for a response within a specified timeout. + If the server sends a successful response after this time has elapsed, the client will receive an error. + When re-executing this request without additional check, the operation may be applied twice. + + A write request can be executed repeatedly without a check only if the error occurred on the server side -- + for example, `ER_READONLY`. + +**Deduplication examples** + +To ensure that the write requests (INSERT, UPDATE, UPSERT, and autoincrement) are idempotent, +you should implement a check that the request is applied for the first time. + +For example, when you add a new tuple to a space, you can use a unique insert ID to check the request. +In the example below within a single transaction: + +1. It is checked whether a tuple with the ``key`` ID exists in the ``bands`` space. +2. If there is no tuple with this ID in the space, the tuple is inserted. + +.. code-block:: lua + + box.begin() + if box.space.bands:get{key} == nil then + box.space.bands:insert{key, value} + end + box.commit() + +For update and upsert requests, you can create a *deduplication space* where the request IDs will be saved. +*Deduplication space* is a user space that contains a list of unique identifiers. +Each identifier corresponds to one applied request. +This space can have any name, in the example it is called ``deduplication``. + +In the example below, within a single transaction: + +1. It is checked whether the ``deduplication_key`` request ID exists in the ``deduplication`` space. +2. If there is no such ID, The ID is added to the deduplication space. +3. If the request hasn't been applied before, it increments the specified field in the ``bands`` space by one. + +This approach ensures that each data modification request will be executed **only once**. + +.. code-block:: lua + + function update_1(deduplication_key, key) + box.begin() + if box.space.deduplication:get{deduplication_key} == nil then + box.space.deduplication:insert{deduplication_key} + box.space.bands:update(key, {{'+', 'value', 1 }}) + end + box.commit() + end + + .. _vshard-maintenance: Sharded cluster maintenance diff --git a/doc/reference/reference_rock/vshard/vshard_router.rst b/doc/reference/reference_rock/vshard/vshard_router.rst index 66c23d2ad..c7e832f03 100644 --- a/doc/reference/reference_rock/vshard/vshard_router.rst +++ b/doc/reference/reference_rock/vshard/vshard_router.rst @@ -163,6 +163,12 @@ Router public API optional attribute containing a message with the human-readable error description, and other attributes specific for the error code. + .. note:: + + Any write requests that are intended to be executed repeatedly should be idempotent. + The operations' idempotency ensures that the change is applied **only once**. + Read more: :ref:``. + **Examples:** To call ``customer_add`` function from ``vshard/example``, say: From 6875bd80371c6d6ca95d79d31864c81888c319c0 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Fri, 25 Jul 2025 12:57:19 +0300 Subject: [PATCH 2/6] Apply review suggestions --- doc/platform/sharding/vshard_admin.rst | 17 +++++++--- .../reference_rock/vshard/vshard_router.rst | 33 ++++++++++++++++++- 2 files changed, 45 insertions(+), 5 deletions(-) diff --git a/doc/platform/sharding/vshard_admin.rst b/doc/platform/sharding/vshard_admin.rst index 0ce6255f7..5131aa57c 100644 --- a/doc/platform/sharding/vshard_admin.rst +++ b/doc/platform/sharding/vshard_admin.rst @@ -563,7 +563,7 @@ When such an operation is applied again, the value for the field increases by 2 .. note:: - Any write requests that are intended to be executed repeatedly should be idempotent. + Any write requests that are intended to be executed repeatedly (for example, retried after an error) should be idempotent. The operations' idempotency ensures that the change is applied **only once**. A request may need to be run again if an error occurs on the server or client side. @@ -588,17 +588,26 @@ In this case: Lack of such a check might lead to duplicate records or unplanned data changes. For example, a client has sent a request to the server. The client is waiting for a response within a specified timeout. - If the server sends a successful response after this time has elapsed, the client will receive an error. + If the server sends a successful response after this time has elapsed, + the client won't see this response due to a timeout, and will consider the request as failed. When re-executing this request without additional check, the operation may be applied twice. - A write request can be executed repeatedly without a check only if the error occurred on the server side -- - for example, `ER_READONLY`. + A write request can be executed repeatedly without a check in two cases: + - The request is idempotent. + - It's known for sure that the previous request raised an error before executing any write operations. + For example, ER_READONLY was thrown by the server. + In this case, we know that the request couldn't complete due to server in read-only mode. **Deduplication examples** To ensure that the write requests (INSERT, UPDATE, UPSERT, and autoincrement) are idempotent, you should implement a check that the request is applied for the first time. +.. note:: + + There is no built-in deduplication check in Tarantool. + Currently, deduplication can be only implemented by the user in the application code. + For example, when you add a new tuple to a space, you can use a unique insert ID to check the request. In the example below within a single transaction: diff --git a/doc/reference/reference_rock/vshard/vshard_router.rst b/doc/reference/reference_rock/vshard/vshard_router.rst index c7e832f03..0a6c9e4d2 100644 --- a/doc/reference/reference_rock/vshard/vshard_router.rst +++ b/doc/reference/reference_rock/vshard/vshard_router.rst @@ -132,6 +132,15 @@ Router public API * ``timeout`` — a request timeout, in seconds. If the ``router`` cannot identify a shard with the specified ``bucket_id``, it will retry until the timeout is reached. + * ``request_timeout`` (since ``vshard`` 0.1.28) — timeout in seconds that serves as a protection against hung replicas. + The parameter is used in the read requests only (``mode=read``). + It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: + ``timeout > request_timeout``. + + The ``timeout`` parameter controls how much time a single request attempt may take. + When this time is over (the ``TimedOut`` error is raised), router retries the request on the next replica as long + as the ``timeout`` value is not elapsed. + * other :ref:`net.box options `, such as ``is_async``, ``buffer``, ``on_push`` are also supported. @@ -163,12 +172,16 @@ Router public API optional attribute containing a message with the human-readable error description, and other attributes specific for the error code. + .. reference_vshard_note_start + .. note:: - Any write requests that are intended to be executed repeatedly should be idempotent. + Any write requests that are intended to be executed repeatedly (for example, retried after an error) should be idempotent. The operations' idempotency ensures that the change is applied **only once**. Read more: :ref:``. + .. reference_vshard_note_end + **Examples:** To call ``customer_add`` function from ``vshard/example``, say: @@ -205,6 +218,13 @@ Router public API * ``timeout`` — a request timeout, in seconds.If the ``router`` cannot identify a shard with the specified ``bucket_id``, it will retry until the timeout is reached. + * ``request_timeout`` (since ``vshard`` 0.1.28) — timeout in seconds that serves as a protection against hung replicas. + It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: + ``timeout > request_timeout``. + The ``timeout`` parameter controls how much time a single request attempt may take. + When this time is over (the ``TimedOut`` error is raised), router retries the request on the next replica as long + as the ``timeout`` value is not elapsed. + * other :ref:`net.box options `, such as ``is_async``, ``buffer``, ``on_push`` are also supported. @@ -254,6 +274,10 @@ Router public API optional attribute containing a message with the human-readable error description, and other attributes specific for this error code. + .. include:: /reference/reference_rock/vshard/vshard_router.rst + :start-after: reference_vshard_note_start + :end-before: reference_vshard_note_end + .. _router_api-callre: .. function:: vshard.router.callre(bucket_id, function_name, {argument_list}, {options}) @@ -273,6 +297,13 @@ Router public API * ``timeout`` — a request timeout, in seconds. If the ``router`` cannot identify a shard with the specified ``bucket_id``, it will retry until the timeout is reached. + * ``request_timeout`` (since ``vshard`` 0.1.28) — timeout in seconds that serves as a protection against hung replicas. + It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: + ``timeout > request_timeout``. + The ``timeout`` parameter controls how much time a single request attempt may take. + When this time is over (the ``TimedOut`` error is raised), router retries the request on the next replica as long + as the ``timeout`` value is not elapsed. + * other :ref:`net.box options `, such as ``is_async``, ``buffer``, ``on_push`` are also supported. From 2d93dd39110cb398b521b6e3dac67e81a73ade36 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Thu, 16 Oct 2025 12:12:08 +0300 Subject: [PATCH 3/6] Add small fixes --- doc/platform/sharding/vshard_admin.rst | 2 ++ doc/reference/reference_rock/vshard/vshard_router.rst | 6 +++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/platform/sharding/vshard_admin.rst b/doc/platform/sharding/vshard_admin.rst index 5131aa57c..6020a65ff 100644 --- a/doc/platform/sharding/vshard_admin.rst +++ b/doc/platform/sharding/vshard_admin.rst @@ -593,7 +593,9 @@ In this case: When re-executing this request without additional check, the operation may be applied twice. A write request can be executed repeatedly without a check in two cases: + - The request is idempotent. + - It's known for sure that the previous request raised an error before executing any write operations. For example, ER_READONLY was thrown by the server. In this case, we know that the request couldn't complete due to server in read-only mode. diff --git a/doc/reference/reference_rock/vshard/vshard_router.rst b/doc/reference/reference_rock/vshard/vshard_router.rst index 0a6c9e4d2..e6721e72f 100644 --- a/doc/reference/reference_rock/vshard/vshard_router.rst +++ b/doc/reference/reference_rock/vshard/vshard_router.rst @@ -138,7 +138,7 @@ Router public API ``timeout > request_timeout``. The ``timeout`` parameter controls how much time a single request attempt may take. - When this time is over (the ``TimedOut`` error is raised), router retries the request on the next replica as long + When this time is over (the ``TimedOut`` error is raised), the router retries this request on the next replica as long as the ``timeout`` value is not elapsed. * other :ref:`net.box options `, such as ``is_async``, @@ -222,7 +222,7 @@ Router public API It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: ``timeout > request_timeout``. The ``timeout`` parameter controls how much time a single request attempt may take. - When this time is over (the ``TimedOut`` error is raised), router retries the request on the next replica as long + When this time is over (the ``TimedOut`` error is raised), the router retries this request on the next replica as long as the ``timeout`` value is not elapsed. * other :ref:`net.box options `, such as ``is_async``, @@ -301,7 +301,7 @@ Router public API It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: ``timeout > request_timeout``. The ``timeout`` parameter controls how much time a single request attempt may take. - When this time is over (the ``TimedOut`` error is raised), router retries the request on the next replica as long + When this time is over (the ``TimedOut`` error is raised), the router retries this request on the next replica as long as the ``timeout`` value is not elapsed. * other :ref:`net.box options `, such as ``is_async``, From 3173f407c1a12503daadbe6fa7b2f31c9d843095 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Fri, 24 Oct 2025 16:34:29 +0300 Subject: [PATCH 4/6] Add small fix --- doc/reference/reference_rock/vshard/vshard_router.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/reference/reference_rock/vshard/vshard_router.rst b/doc/reference/reference_rock/vshard/vshard_router.rst index e6721e72f..08253e91a 100644 --- a/doc/reference/reference_rock/vshard/vshard_router.rst +++ b/doc/reference/reference_rock/vshard/vshard_router.rst @@ -137,7 +137,7 @@ Router public API It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: ``timeout > request_timeout``. - The ``timeout`` parameter controls how much time a single request attempt may take. + The ``request_timeout`` parameter controls how much time a single request attempt may take. When this time is over (the ``TimedOut`` error is raised), the router retries this request on the next replica as long as the ``timeout`` value is not elapsed. @@ -221,7 +221,7 @@ Router public API * ``request_timeout`` (since ``vshard`` 0.1.28) — timeout in seconds that serves as a protection against hung replicas. It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: ``timeout > request_timeout``. - The ``timeout`` parameter controls how much time a single request attempt may take. + The ``request_timeout`` parameter controls how much time a single request attempt may take. When this time is over (the ``TimedOut`` error is raised), the router retries this request on the next replica as long as the ``timeout`` value is not elapsed. @@ -300,7 +300,7 @@ Router public API * ``request_timeout`` (since ``vshard`` 0.1.28) — timeout in seconds that serves as a protection against hung replicas. It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: ``timeout > request_timeout``. - The ``timeout`` parameter controls how much time a single request attempt may take. + The ``request_timeout`` parameter controls how much time a single request attempt may take. When this time is over (the ``TimedOut`` error is raised), the router retries this request on the next replica as long as the ``timeout`` value is not elapsed. From 038988754a088918b4429540f5600ce489213953 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova <73473519+xuniq@users.noreply.github.com> Date: Mon, 27 Oct 2025 19:18:53 +0300 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Elena Shebunyaeva --- doc/platform/sharding/vshard_admin.rst | 4 ++-- doc/reference/reference_rock/vshard/vshard_router.rst | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/platform/sharding/vshard_admin.rst b/doc/platform/sharding/vshard_admin.rst index 6020a65ff..e17ff5a2d 100644 --- a/doc/platform/sharding/vshard_admin.rst +++ b/doc/platform/sharding/vshard_admin.rst @@ -611,7 +611,7 @@ you should implement a check that the request is applied for the first time. Currently, deduplication can be only implemented by the user in the application code. For example, when you add a new tuple to a space, you can use a unique insert ID to check the request. -In the example below within a single transaction: +In the example below, within a single transaction: 1. It is checked whether a tuple with the ``key`` ID exists in the ``bands`` space. 2. If there is no tuple with this ID in the space, the tuple is inserted. @@ -632,7 +632,7 @@ This space can have any name, in the example it is called ``deduplication``. In the example below, within a single transaction: 1. It is checked whether the ``deduplication_key`` request ID exists in the ``deduplication`` space. -2. If there is no such ID, The ID is added to the deduplication space. +2. If there is no such ID, the ID is added to the deduplication space. 3. If the request hasn't been applied before, it increments the specified field in the ``bands`` space by one. This approach ensures that each data modification request will be executed **only once**. diff --git a/doc/reference/reference_rock/vshard/vshard_router.rst b/doc/reference/reference_rock/vshard/vshard_router.rst index 08253e91a..42c650a92 100644 --- a/doc/reference/reference_rock/vshard/vshard_router.rst +++ b/doc/reference/reference_rock/vshard/vshard_router.rst @@ -133,7 +133,7 @@ Router public API shard with the specified ``bucket_id``, it will retry until the timeout is reached. * ``request_timeout`` (since ``vshard`` 0.1.28) — timeout in seconds that serves as a protection against hung replicas. - The parameter is used in the read requests only (``mode=read``). + The parameter is used in read requests only (``mode=read``). It is necessary to pass the ``request_timeout`` and ``timeout`` parameters together, with the following requirement: ``timeout > request_timeout``. From f788297f1feb4e668ff61c568929231e458de791 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Tue, 28 Oct 2025 12:28:08 +0300 Subject: [PATCH 6/6] Add small fix --- doc/reference/reference_rock/vshard/vshard_router.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/reference/reference_rock/vshard/vshard_router.rst b/doc/reference/reference_rock/vshard/vshard_router.rst index 42c650a92..2cc7f0557 100644 --- a/doc/reference/reference_rock/vshard/vshard_router.rst +++ b/doc/reference/reference_rock/vshard/vshard_router.rst @@ -178,7 +178,7 @@ Router public API Any write requests that are intended to be executed repeatedly (for example, retried after an error) should be idempotent. The operations' idempotency ensures that the change is applied **only once**. - Read more: :ref:``. + Read more: :ref:`Deduplication of non-idempotent requests `. .. reference_vshard_note_end