From cf7f7c9a3a48c82e0036693406da046b4289e1c7 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Wed, 24 Jan 2024 22:35:51 +0300 Subject: [PATCH 01/10] persistence: add wal and snapshot sections --- doc/enterprise/wal_extensions.rst | 3 +- .../cfg_binary_logging_snapshots.rst | 24 +- .../configuration/configuration_reference.rst | 246 ++++++++++++++++++ 3 files changed, 260 insertions(+), 13 deletions(-) diff --git a/doc/enterprise/wal_extensions.rst b/doc/enterprise/wal_extensions.rst index c07cc5d0fb..c5eac42f6b 100644 --- a/doc/enterprise/wal_extensions.rst +++ b/doc/enterprise/wal_extensions.rst @@ -8,13 +8,12 @@ For example, you can enable storing an old and new tuple for each CRUD operation This information might be helpful for implementing a CDC (Change Data Capture) utility that transforms a data replication stream. - .. _wal_extensions_configuration: Configuration ------------- -To configure WAL extensions, use the ``wal_ext`` :ref:`configuration property `. +To configure WAL extensions, use the ``wal.ext`` :ref:`configuration property `. Inside the ``wal_ext`` block, you can enable storing old and new tuples as follows: * Set the ``old`` and ``new`` options to ``true`` to store old and new tuples in a write-ahead log for all spaces. diff --git a/doc/reference/configuration/cfg_binary_logging_snapshots.rst b/doc/reference/configuration/cfg_binary_logging_snapshots.rst index 83f29695b6..0da363eb45 100644 --- a/doc/reference/configuration/cfg_binary_logging_snapshots.rst +++ b/doc/reference/configuration/cfg_binary_logging_snapshots.rst @@ -36,8 +36,8 @@ Since version 1.7.4. The maximum number of bytes in a single write-ahead log file. - When a request would cause an .xlog file to become larger than - ``wal_max_size``, Tarantool creates another WAL file. + When a request would cause an ``.xlog`` file to become larger than + ``wal_max_size``, Tarantool creates a new WAL file. | Type: integer | Default: 268435456 (256 * 1024 * 1024) bytes @@ -58,7 +58,7 @@ locations and moving snapshots to a separate disk. The limit also affects what :ref:`box.stat.vinyl().regulator ` - may show for the write rate of dumps to .run and .index files. + may show for the write rate of dumps to ``.run`` and ``.index`` files. | Type: float | Default: null @@ -73,12 +73,14 @@ Specify fiber-WAL-disk synchronization mode as: - * ``none``: write-ahead log is not maintained. - A node with ``wal_mode = none`` can't be replication master; - * ``write``: :ref:`fibers ` wait for their data to be written to - the write-ahead log (no :manpage:`fsync(2)`); - * ``fsync``: fibers wait for their data, :manpage:`fsync(2)` - follows each :manpage:`write(2)`; + * ``none``: write-ahead log is not maintained. + A node with ``wal_mode = none`` can't be replication master. + + * ``write``: :ref:`fibers ` wait for their data to be written to + the write-ahead log (no :manpage:`fsync(2)`). + + * ``fsync``: fibers wait for their data, :manpage:`fsync(2)` + follows each :manpage:`write(2)`. | Type: string | Default: "write" @@ -129,7 +131,7 @@ Since version :doc:`2.6.3 `. The delay (in seconds) used to prevent the :ref:`Tarantool garbage collector ` - from immediately removing :ref:`write-ahead log` files after a node restart. + from immediately removing :ref:`write-ahead log ` files after a node restart. This delay eliminates possible erroneous situations when the master deletes WALs needed by :ref:`replicas ` after restart. As a consequence, replicas sync with the master faster after its restart and @@ -141,7 +143,7 @@ .. NOTE:: The ``wal_cleanup_delay`` option has no effect on nodes running as - :ref:`anonymous replicas`. + :ref:`anonymous replicas `. | Type: number | Default: 14400 seconds diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index 791a7dc50d..910efbc08f 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2389,3 +2389,249 @@ The ``security`` section defines configuration parameters related to various sec | Type: boolean | Default: false | Environment variable: TT_SECURITY_SECURE_ERASING + +.. _configuration_reference_snapshot: + +snapshot +-------- + +The ``snapshot`` section defines configuration parameters related to + +.. NOTE:: + + ``snapshot`` can be defined in any :ref:`scope `. + +- :ref:`snapshot.by.interval ` +- :ref:`snapshot.by.wal_size ` +- :ref:`snapshot.count ` +- :ref:`snapshot.dir ` +- :ref:`snapshot.snap_io_rate_limit ` + +.. _configuration_reference_snapshot_by_interval: + +.. confval:: snapshot.by.interval + + Description + + | + | Type: number + | Default: 3600 + | Environment variable: TT_SNAPSHOT_BY_INTERVAL + +.. _configuration_reference_snapshot_by_wal_size: + +.. confval:: snapshot.by.wal_size + + Description + + | + | Type: number + | Default: 3600 + | Environment variable: TT_SNAPSHOT_BY_WAL_SIZE + +.. _configuration_reference_snapshot_count: + +.. confval:: snapshot.count + + Description + + | + | Type: integer + | Default: 2 + | Environment variable: TT_SNAPSHOT_COUNT + +.. _configuration_reference_snapshot_dir: + +.. confval:: snapshot.dir + + Description + + | + | Type: string + | Default: 'var/lib/{{ instance_name }}' + | Environment variable: TT_SNAPSHOT_DIR + + +.. _configuration_reference_snapshot_snap_io_rate_limit: + +.. confval:: snapshot.snap_io_rate_limit + + Reduce the throttling effect of :doc:`box.snapshot() ` on + INSERT/UPDATE/DELETE performance by setting a limit on how many + megabytes per second it can write to disk. The same can be + achieved by splitting :ref:`wal.dir ` and + :ref:`memtx_dir ` + locations and moving snapshots to a separate disk. + The limit also affects what + :ref:`box.stat.vinyl().regulator ` + may show for the write rate of dumps to ``.run`` and ``.index`` files. + + | + | Type: number + | Default: box.NULL + | Environment variable: TT_SNAPSHOT_SNAP_IO_RATE_LIMIT + + +.. _configuration_reference_wal: + +wal +--- + +The ``wal`` section defines configuration parameters related to + +.. NOTE:: + + ``wal`` can be defined in any :ref:`scope `. + +- :ref:`wal.cleanup_delay ` +- :ref:`wal.dir ` +- :ref:`wal.dir_rescan_delay ` +- :ref:`wal.ext.new ` +- :ref:`wal.ext.old ` +- :ref:`wal.ext.spaces ` +- :ref:`wal.wal_max_size ` +- :ref:`wal.mode ` +- :ref:`wal.queue_max_size ` + +.. _configuration_reference_wal_cleanup_delay: + +.. confval:: wal.cleanup_delay + + The delay in seconds used to prevent the :ref:`Tarantool garbage collector ` + from immediately removing :ref:`write-ahead log ` files after a node restart. + This delay eliminates possible erroneous situations when the master deletes WALs + needed by :ref:`replicas ` after restart. + As a consequence, replicas sync with the master faster after its restart and + don't need to download all the data again. + + Once all the nodes in the replica set are up and running, + automatic cleanup is started again even if ``wal.cleanup_delay`` has not expired. + + .. NOTE:: + + The option has no effect on nodes running as + :ref:`anonymous replicas `. + + | + | Type: number + | Default: 14400 + | Environment variable: TT_WAL_CLEANUP_DELAY + +.. _configuration_reference_wal_dir: + +.. confval:: wal.dir + + A directory where write-ahead log (.xlog) files are stored. Can be relative + to :ref:`work_dir `. Sometimes ``wal_dir`` and + :ref:`memtx_dir ` are specified with different values, so + that write-ahead log files and snapshot files can be stored on different + disks. If not specified, defaults to ``work_dir``. + + | + | Type: string + | Default: 'var/lib/{{ instance_name }}' + | Environment variable: TT_WAL_DIR + +.. _configuration_reference_wal_dir_rescan_delay: + +.. confval:: wal.dir_rescan_delay + + Number of seconds between periodic scans of the write-ahead-log + file directory, when checking for changes to write-ahead-log + files for the sake of :ref:`replication ` or :ref:`hot standby `. + + | + | Type: number + | Default: 2 + | Environment variable: TT_WAL_DIR_RESCAN_DELAY + +.. _configuration_reference_wal_ext_new: + +.. confval:: wal.ext_new + + (**Enterprise Edition only**) + + Description + + | + | Type: boolean + | Default: nil + | Environment variable: TT_WAL_EXT_NEW + +.. _configuration_reference_wal_ext_old: + +.. confval:: wal.ext_old + + (**Enterprise Edition only**) + Description + + | + | Type: boolean + | Default: nil + | Environment variable: TT_WAL_EXT_OLD + +.. _configuration_reference_wal_ext_spaces: + +.. confval:: wal.ext_spaces + + (**Enterprise Edition only**) + Description + + | + | Type: map + | Default: nil + | Environment variable: TT_WAL_EXT_SPACES + + +.. _configuration_reference_wal_max_size: + +.. confval:: wal.wal_max_size + + The maximum number of bytes in a single write-ahead log file. + When a request would cause an ``.xlog`` file to become larger than + ``wal_max_size``, Tarantool creates a new WAL file. + + | + | Type: integer + | Default: 268435456 + | Environment variable: TT_WAL_MAX_SIZE + +.. _configuration_reference_wal_mode: + +.. confval:: wal.mode + + Specify fiber-WAL-disk synchronization mode as: + + * ``none``: write-ahead log is not maintained. + A node with ``wal.mode = none`` can't be replication master. + + * ``write``: :ref:`fibers ` wait for their data to be written to + the write-ahead log (no :manpage:`fsync(2)`). + + * ``fsync``: fibers wait for their data, :manpage:`fsync(2)` + follows each :manpage:`write(2)`. + + | + | Type: string + | Default: 'write' + | Environment variable: TT_WAL_MODE + +.. _configuration_reference_wal_queue_max_size: + +.. confval:: wal.queue_max_size + + The size of the queue in bytes used by a :ref:`replica ` to submit + new transactions to a :ref:`write-ahead log` (WAL). + This option helps limit the rate at which a replica submits transactions to the WAL. + Limiting the queue size might be useful when a replica is trying to sync with a master and + reads new transactions faster than writing them to the WAL. + + .. NOTE:: + + You might consider increasing the ``wal.queue_max_size`` value in case of + large tuples (approximately one megabyte or larger). + + | + | Type: integer + | Default: 16777216 + | Environment variable: TT_WAL_QUEUE_MAX_SIZE \ No newline at end of file From 9ca5a718e2b071b23cbe4e05302cc167d8579470 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Mon, 5 Feb 2024 18:26:16 +0300 Subject: [PATCH 02/10] persistence: update topic --- doc/book/admin/backups.rst | 6 +- .../persistence_snapshot/config.yaml | 14 + .../persistence_snapshot/instances.yml | 1 + .../persistence_wal/config.yaml | 21 ++ .../persistence_wal/instances.yml | 1 + doc/concepts/atomic/thread_model.rst | 2 +- doc/concepts/configuration.rst | 5 +- .../configuration_persistence.rst | 186 +++++++++++ doc/concepts/data_model/operations.rst | 2 +- doc/concepts/data_model/persistence.rst | 11 +- doc/dev_guide/internals/file_formats.rst | 129 ++++---- doc/enterprise/audit.rst | 20 +- doc/enterprise/flight_recorder.rst | 10 +- doc/enterprise/wal_extensions.rst | 56 ++-- .../configuration/cfg_snapshot_daemon.rst | 15 - .../configuration/configuration_reference.rst | 311 ++++++++++++------ doc/reference/reference_lua/box_snapshot.rst | 4 +- 17 files changed, 564 insertions(+), 230 deletions(-) create mode 100644 doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml create mode 100644 doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/instances.yml create mode 100644 doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml create mode 100644 doc/code_snippets/snippets/config/instances.enabled/persistence_wal/instances.yml create mode 100644 doc/concepts/configuration/configuration_persistence.rst diff --git a/doc/book/admin/backups.rst b/doc/book/admin/backups.rst index 6e34d4c396..a2829db05f 100644 --- a/doc/book/admin/backups.rst +++ b/doc/book/admin/backups.rst @@ -37,15 +37,15 @@ that are made after the last snapshot are incremental backups. Therefore taking a backup is a matter of copying the snapshot and WAL files. 1. Use ``tar`` to make a (possibly compressed) copy of the latest .snap and .xlog - files on the :ref:`memtx_dir ` and - :ref:`wal_dir ` directories. + files on the :ref:`snapshot.dir ` and + :ref:`wal.dir ` directories. 2. If there is a security policy, encrypt the .tar file. 3. Copy the .tar file to a safe place. Later, restoring the database is a matter of taking the .tar file and putting -its contents back in the ``memtx_dir`` and ``wal_dir`` directories. +its contents back in the ``snapshot.dir`` and ``wal.dir`` directories. .. _admin-backups-hot_backup_vinyl_memtx: diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml new file mode 100644 index 0000000000..08f46c1c52 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml @@ -0,0 +1,14 @@ +groups: + group001: + replicasets: + replicaset001: + instances: + instance001: + snapshot: + dir: 'var/lib/{{ instance_name }}/snapshots' + count: 10 + by: + interval: 60 + iproto: + listen: + - uri: '127.0.0.1:3301' \ No newline at end of file diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/instances.yml b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/instances.yml new file mode 100644 index 0000000000..9d7cb5e7c2 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/instances.yml @@ -0,0 +1 @@ +instance001: \ No newline at end of file diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml new file mode 100644 index 0000000000..6350ad3b21 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml @@ -0,0 +1,21 @@ +groups: + group001: + replicasets: + replicaset001: + instances: + instance001: + wal: + mode: 'write' + dir: 'var/lib/{{ instance_name }}/wals' + dir_rescan_delay: 3 + cleanup_delay: 18000 + max_size: 268435456 + ext: + new: true + old: true + spaces: + bands: + new: false + iproto: + listen: + - uri: '127.0.0.1:3301' \ No newline at end of file diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/instances.yml b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/instances.yml new file mode 100644 index 0000000000..9d7cb5e7c2 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/instances.yml @@ -0,0 +1 @@ +instance001: \ No newline at end of file diff --git a/doc/concepts/atomic/thread_model.rst b/doc/concepts/atomic/thread_model.rst index fb3e90d4e9..28f5806850 100644 --- a/doc/concepts/atomic/thread_model.rst +++ b/doc/concepts/atomic/thread_model.rst @@ -69,7 +69,7 @@ There are also several supplementary threads that serve additional capabilities: Separate threads are required because each replica can point to a different position in the log and can run at different speeds. * There is a thread pool for ad hoc asynchronous tasks, - such as a DNS resolver or :ref:`fsync `. + such as a DNS resolver or :ref:`fsync `. * There are OpenMP threads used to parallelize sorting (hence, to parallelize building :ref:`indexes `). diff --git a/doc/concepts/configuration.rst b/doc/concepts/configuration.rst index d1f250bc9f..e709c66ec2 100644 --- a/doc/concepts/configuration.rst +++ b/doc/concepts/configuration.rst @@ -426,7 +426,8 @@ When the limit is reached, ``INSERT`` or ``UPDATE`` requests fail with :ref:`ER_ Snapshots and write-ahead logs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``snapshot.dir`` and ``wal.dir`` options can be used to configure directories for storing snapshots and write-ahead logs. +The :ref:`snapshot.dir ` and :ref:`wal.dir ` +options can be used to configure directories for storing snapshots and write-ahead logs. For example, you can place snapshots and write-ahead logs on different hard drives for better reliability. .. code-block:: yaml @@ -438,6 +439,7 @@ For example, you can place snapshots and write-ahead logs on different hard driv dir: '/media/drive2/wals' To learn more about the persistence mechanism in Tarantool, see the :ref:`Persistence ` section. +Read more about snapshot and WAL configuration: :ref:`Persistence `. @@ -447,6 +449,7 @@ To learn more about the persistence mechanism in Tarantool, see the :ref:`Persis configuration/configuration_etcd configuration/configuration_code + configuration/configuration_persistence configuration/configuration_connections configuration/configuration_credentials configuration/configuration_authentication diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst new file mode 100644 index 0000000000..8f532b1e5f --- /dev/null +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -0,0 +1,186 @@ +.. _configuration_persistence: + +Persistence +=========== + +To ensure data persistence, Tarantool provides the abilities to: + +* record each data change request into a :ref:`write-ahead log ` (WAL) file (``.xlog`` files) +* take :ref:`snapshots ` that contain on-disk copy of the entire data set for a given moment (``.snap`` files) + +During the recovery process, Tarantool can load the latest snapshot file and then read the requests from the WAL files, +produced after this snapshot was made. + +To learn more about the persistence mechanism in Tarantool, see the :ref:`Persistence ` section. +The formats of WAL and snapshot files are described in detail in the :ref:`File formats ` section. + +.. _configuration_persistence_snapshot: + +Configure the snapshots +----------------------- + +**Example on GitHub**: `snapshot `_ + +This section describes how to define snapshot settings in the :ref:`snapshot ` section of a YAML configuration. + +.. _configuration_persistence_snapshot_dir: + +Specify a directory for snapshot files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To configure a directory where the snapshot files are stored, use the :ref:`snapshot.dir ` +configuration option. +The example below shows how to specify a snapshot directory for ``instance001`` explicitly: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: instance001: + :end-at: 'var/lib/{{ instance_name }}/snapshots' + :dedent: + +By default, WAL files and snapshot files are stored in the same directory ``var/lib/{{ instance_name }}``. +However, you can specify different directories for them. +For example, you can place snapshots and write-ahead logs on different hard drives for better reliability: + +.. code-block:: yaml + + instance001: + snapshot: + dir: '/media/drive1/snapshots' + wal: + dir: '/media/drive2/wals' + +.. _configuration_persistence_checkpoint_daemon: + +Configure the automatic snapshot creation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Tarantool, it is possible to set automatic :ref:`snapshot creation `. +To enable it, the :ref:`snapshot.by.interval ` option is used. +The option sets up the :ref:`checkpoint daemon ` (snapshot daemon), which is +a constantly running :ref:`fiber `. +The checkpoint daemon takes new snapshots every ``snapshot.by.interval`` seconds. +When the number of snapshots reaches the limit of :ref:`snapshot.count ` size, +the daemon activates Tarantool garbage collector after the new snapshot is taken. +Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. + +The :ref:`snapshot.by.wal_size ` option defines the maximum size in bytes +for of all WAL files created since the last snapshot taken. +Once this size is exceeded, the checkpoint daemon takes a snapshot and deletes the old WAL files. + +The configuration of the checkpoint daemon might look as follows: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: count: + :end-at: 60 + :dedent: + +If the ``snapshot.by.interval`` option is set to zero, the checkpoint daemon is disabled. +If the ``snapshot.count`` option is set to zero, the checkpoint daemon does not delete old snapshots. + +.. _configuration_persistence_wal: + +Configure the write-ahead log +----------------------------- + +**Example on GitHub**: `wal `_ + +This section describes how to define WAL settings in the :ref:`wal ` section of a YAML configuration. + +.. _configuration_persistence_wal_mode: + +Set the WAL mode +~~~~~~~~~~~~~~~~ + +To be able to recover data in case of a possible instance restart, enable recording to the write-ahead log. +To do it, set the :ref:`wal.mode ` configuration option to ``write`` or ``fsync``. +The example below shows how to specify the ``write`` WAL mode for ``instance001``: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: instance001 + :end-at: 'write' + :dedent: + +The ``write`` mode enables WAL and writes the data without waiting the data to be flushed to the storage device. +The ``fsync`` mode enables WAL and ensures that the record is written to the storage device. + +To turn the WAL writer off, set the ``wal.mode`` option to ``none``. + +.. _configuration_persistence_wal_dir: + +Specify a directory for WAL files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To configure a directory where the WAL files are stored, use the :ref:`wal.dir ` configuration option. +The example below shows how to specify a directory for ``instance001`` explicitly: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: wal: + :end-at: 'var/lib/{{ instance_name }}/wals' + :dedent: + + +.. _configuration_persistence_wal_rescan: + +Set an interval between scans +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If case of :ref:`replication ` or :ref:`hot standby ` mode, +Tarantool scans for changes in the WAL files every :ref:`wal.dir_rescan_delay ` +seconds. The example below shows how to specify the interval between scans: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: dir_rescan_delay + :end-before: cleanup_delay + :dedent: + +.. _configuration_persistence_wal_maxsize: + +Set a maximum size for the WAL file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A new WAL file is created when the current one reaches the :ref:`wal.max_size ` +size. The configuration for this option might look as follows: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: max_size + :end-at: 268435456 + :dedent: + +.. _configuration_persistence_wal_rescan: + +Set a delay for the garbage collector +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Tarantool, the :ref:`checkpoint daemon ` (snapshot daemon) +takes new snapshots at the given interval (see :ref:`snapshot.by.interval `). +After an instance restart, the daemon activates the Tarantool garbage collector that deletes the old WAL files. + +To delay the immediate deletion of WAL files, use the :ref:`wal.cleanup_delay ` +configuration option. The delay eliminates possible erroneous situations when the master deletes WALs +needed by :ref:`replicas ` after restart. +As a consequence, replicas sync with the master faster after its restart and +don't need to download all the data again. + +In the example, the delay is set to 5 hours (18000 seconds): + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: cleanup_delay + :end-at: 18000 + :dedent: + +.. _configuration_persistence_wal_ext: + +Specify the WAL extensions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Tarantool Enterprise, you can store an old and new tuple for each crud operation performed. +The detailed description and examples of the WAL extensions are provided in the :ref:`WAL extensions ` section. + +See also: :ref:`wal.ext.* ` configuration options. diff --git a/doc/concepts/data_model/operations.rst b/doc/concepts/data_model/operations.rst index 602c202539..30a8243066 100644 --- a/doc/concepts/data_model/operations.rst +++ b/doc/concepts/data_model/operations.rst @@ -150,7 +150,7 @@ resource usage of each function. important than the others. * - WAL settings - The important setting for the write-ahead log is - :ref:`wal_mode `. + :ref:`wal.mode `. If the setting causes no writing or delayed writing, this factor is unimportant. If the setting causes every data-change request to wait diff --git a/doc/concepts/data_model/persistence.rst b/doc/concepts/data_model/persistence.rst index 0545fdbfef..dce7443f67 100644 --- a/doc/concepts/data_model/persistence.rst +++ b/doc/concepts/data_model/persistence.rst @@ -12,7 +12,7 @@ In such case, Tarantool restores the data from WAL files by reading them and redoing the requests. This is called the "recovery process". You can change the timing of the WAL writer or turn it off by setting -the :ref:`wal_mode `. +the :ref:`wal.mode `. Tarantool also maintains a set of :ref:`snapshot files `. These files contain an on-disk copy of the entire data set for a given moment. @@ -24,8 +24,8 @@ After creating a new snapshot, the earlier WAL files can be removed to free up s To force immediate creation of a snapshot file, use the :doc:`box.snapshot() ` function. To enable the automatic creation of snapshot files, use Tarantool's -:ref:`checkpoint daemon `. -The checkpoint daemon sets intervals for forced checkpoints. It makes sure that the states +:ref:`checkpoint daemon `. +The checkpoint daemon sets intervals for forced snapshots. It makes sure that the states of both memtx and vinyl storage engines are synchronized and saved to disk, and automatically removes earlier WAL files. @@ -33,10 +33,11 @@ Snapshot files can be created even if there is no WAL file. .. NOTE:: - The memtx engine makes only regular checkpoints with the interval set in - :ref:`checkpoint daemon ` configuration. + The memtx engine takes only regular snapshots with the interval set in + the checkpoint daemon configuration. The vinyl engine runs checkpointing in the background at all times. See the :ref:`Internals ` section for more details about the WAL writer and the recovery process. +To learn more about the configuration of the checkpoint daemon and WAL, check the :ref:`Persistence ` page. \ No newline at end of file diff --git a/doc/dev_guide/internals/file_formats.rst b/doc/dev_guide/internals/file_formats.rst index 72ef740f47..128fa9b99e 100644 --- a/doc/dev_guide/internals/file_formats.rst +++ b/doc/dev_guide/internals/file_formats.rst @@ -1,4 +1,4 @@ -.. _internals-data_persistence: +.. _internals-data_persistence: File formats ============ @@ -10,75 +10,73 @@ Data persistence and the WAL file format To maintain data persistence, Tarantool writes each data change request (insert, update, delete, replace, upsert) into a write-ahead log (WAL) file in the -:ref:`wal_dir ` directory. A new WAL file is created -when the current one reaches the :ref:`wal_max_size ` size. +:ref:`wal.dir ` directory. Each data change request gets assigned a continuously growing 64-bit log sequence number. The name of the WAL file is based on the log sequence number of the first record in the file, plus an extension ``.xlog``. +A new WAL file is created +when the current one reaches the :ref:`wal_max_size ` size. Apart from a log sequence number and the data change request (formatted as in :ref:`Tarantool's binary protocol `), each WAL record contains a header, some metadata, and then the data formatted according to `msgpack `_ rules. -For example, this is what the WAL file looks like after the first INSERT request -("s:insert({1})") for the sandbox database created in our -:ref:`"Getting started" exercises `. -On the left are the hexadecimal bytes that you would see with: - -.. code-block:: console - - $ hexdump 00000000000000000000.xlog - -and on the right are comments. - -.. code-block:: none - - Hex dump of WAL file Comment - -------------------- ------- - 58 4c 4f 47 0a "XLOG\n" - 30 2e 31 33 0a "0.13\n" = version - 53 65 72 76 65 72 3a 20 "Server: " - 38 62 66 32 32 33 65 30 2d [Server UUID]\n - 36 39 31 34 2d 34 62 35 35 - 2d 39 34 64 32 2d 64 32 62 - 36 64 30 39 62 30 31 39 36 - 0a - 56 43 6c 6f 63 6b 3a 20 "Vclock: " - 7b 7d "{}" = vclock value, initially blank - ... (not shown = tuples for system spaces) - d5 ba 0b ab Magic row marker always = 0xab0bbad5 - 19 Length, not including length of header, = 25 bytes - 00 Record header: previous crc32 - ce 8c 3e d6 70 Record header: current crc32 - a7 cc 73 7f 00 00 66 39 Record header: padding - 84 msgpack code meaning "Map of 4 elements" follows - 00 02 element#1: tag=request type, value=0x02=IPROTO_INSERT - 02 01 element#2: tag=server id, value=0x01 - 03 04 element#3: tag=lsn, value=0x04 - 04 cb 41 d4 e2 2f 62 fd d5 d4 element#4: tag=timestamp, value=an 8-byte "Float64" - 82 msgpack code meaning "map of 2 elements" follows - 10 cd 02 00 element#1: tag=space id, value=512, big byte first - 21 91 01 element#2: tag=tuple, value=1-element fixed array={1} + +To see the hexadecimal bytes of the given WAL file, use the ``hexdump`` command: + +.. code-block:: console + + $ hexdump 00000000000000000000.xlog + +For example, the WAL file after the first INSERT request might look the following way: + +.. code-block:: none + + Hex dump of WAL file Comment + -------------------- ------- + 58 4c 4f 47 0a "XLOG\n" + 30 2e 31 33 0a "0.13\n" = version + 53 65 72 76 65 72 3a 20 "Server: " + 38 62 66 32 32 33 65 30 2d [Server UUID]\n + 36 39 31 34 2d 34 62 35 35 + 2d 39 34 64 32 2d 64 32 62 + 36 64 30 39 62 30 31 39 36 + 0a + 56 43 6c 6f 63 6b 3a 20 "Vclock: " + 7b 7d "{}" = vclock value, initially blank + ... (not shown = tuples for system spaces) + d5 ba 0b ab Magic row marker always = 0xab0bbad5 + 19 Length, not including length of header, = 25 bytes + 00 Record header: previous crc32 + ce 8c 3e d6 70 Record header: current crc32 + a7 cc 73 7f 00 00 66 39 Record header: padding + 84 msgpack code meaning "Map of 4 elements" follows + 00 02 element#1: tag=request type, value=0x02=IPROTO_INSERT + 02 01 element#2: tag=server id, value=0x01 + 03 04 element#3: tag=lsn, value=0x04 + 04 cb 41 d4 e2 2f 62 fd d5 d4 element#4: tag=timestamp, value=an 8-byte "Float64" + 82 msgpack code meaning "map of 2 elements" follows + 10 cd 02 00 element#1: tag=space id, value=512, big byte first + 21 91 01 element#2: tag=tuple, value=1-element fixed array={1} Tarantool processes requests atomically: a change is either accepted and recorded -in the WAL, or discarded completely. Let's clarify how this happens, using the -REPLACE request as an example: +in the WAL, or discarded completely. To clarify how this happens, see the example with the REPLACE request below: -1. The server instance attempts to locate the original tuple by primary key. If found, a +#. The server instance attempts to locate the original tuple by primary key. If found, a reference to the tuple is retained for later use. -2. The new tuple is validated. If for example it does not contain an indexed +#. The new tuple is validated. If for example it does not contain an indexed field, or it has an indexed field whose type does not match the type according to the index definition, the change is aborted. -3. The new tuple replaces the old tuple in all existing indexes. +#. The new tuple replaces the old tuple in all existing indexes. -4. A message is sent to the WAL writer running in a separate thread, requesting that +#. A message is sent to the WAL writer running in a separate thread, requesting that the change be recorded in the WAL. The instance switches to work on the next request until the write is acknowledged. -5. On success, a confirmation is sent to the client. On failure, a rollback - procedure is begun. During the rollback procedure, the transaction processor +#. On success, a confirmation is sent to the client. On failure, a rollback + procedure begins. During the rollback procedure, the transaction processor rolls back all changes to the database which occurred after the first failed change, from latest to oldest, up to the first failed change. All rolled back requests are aborted with :errcode:`ER_WAL_IO ` error. No new @@ -91,49 +89,48 @@ database performance doesn't degrade even if all requests refer to the same key in the same space. The transaction processor thread communicates with the WAL writer thread using -asynchronous (yet reliable) messaging; the transaction processor thread, not -being blocked on WAL tasks, continues to handle requests quickly even at high +asynchronous (yet reliable) messaging. +The transaction processor thread, not being blocked on WAL tasks, continues to handle requests quickly even at high volumes of disk I/O. A response to a request is sent as soon as it is ready, even if there were earlier incomplete requests on the same connection. In particular, SELECT performance, even for SELECTs running on a connection packed with UPDATEs and DELETEs, remains unaffected by disk load. The WAL writer employs a number of durability modes, as defined in configuration -variable :ref:`wal_mode `. It is possible to turn the write-ahead -log completely off, by setting -:ref:`wal_mode ` to *none*. Even -without the write-ahead log it's still possible to take a persistent copy of the +variable :ref:`wal.mode `. +It is possible to turn the write-ahead log completely off, by setting the ``wal_mode`` option to *none*. +Even without the write-ahead log it's still possible to take a persistent copy of the entire data set with the :ref:`box.snapshot() ` request. An .xlog file always contains changes based on the primary key. Even if the client requested an update or delete using -a secondary key, the record in the .xlog file will contain the primary key. +a secondary key, the record in the .xlog file contains the primary key. -.. _internals-snapshot: +.. _internals-snapshot: The snapshot file format ------------------------ -The format of a snapshot .snap file is nearly the same as the format of a WAL .xlog file. -However, the snapshot header differs: it contains the instance's global unique identifier +The format of a snapshot .snap file is similar to the format of a WAL .xlog file, except for the header and content. +The snapshot header contains the instance's global unique identifier and the snapshot file's position in history, relative to earlier snapshot files. Also, the content differs: an .xlog file may contain records for any data-change requests (inserts, updates, upserts, and deletes), a .snap file may only contain records of inserts to memtx spaces. -Primarily, the .snap file's records are ordered by space id. Therefore the records of -system spaces -- such as ``_schema``, ``_space``, ``_index``, ``_func``, ``_priv`` -and ``_cluster`` -- will be at the start of the .snap file, before the records of -any spaces that were created by users. +Primarily, the records in the snapshot file have the following order: + +* System spaces (id >= 256 && id <= 511), ordered by ID. +* Non-system spaces, ordered by ID. -Secondarily, the .snap file's records are ordered by primary key within space id. +Secondarily, the .snap file's records are ordered by primary key within space ID. .. _box_protocol-xlog: Example ------- -The header of a ``.snap`` or ``.xlog`` file looks like: +The header of a ``.snap`` or ``.xlog`` file might look in the following way: .. code-block:: none diff --git a/doc/enterprise/audit.rst b/doc/enterprise/audit.rst index 927d5f7266..c2ea40409e 100644 --- a/doc/enterprise/audit.rst +++ b/doc/enterprise/audit.rst @@ -144,21 +144,17 @@ Tarantool records all incoming data in the write-ahead log (WAL). The WAL must be enabled to ensure that data will be recovered in case of a possible instance restart. -Secure values of ``wal_mode`` are ``write`` and ``fsync``: +Secure values of the :ref:`wal.mode ` configuration option are ``write`` and ``fsync``: -.. code-block:: tarantoolsession - - tarantool> box.cfg.wal_mode - --- - - write +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: wal: + :end-at: 'write' + :dedent: An exclusion from this requirement is when the instance is processing data, -which can be freely rejected. -For example, when Tarantool is used for caching. -Then WAL can be disabled to reduce i/o load. - -For more details, see the -:ref:`wal_mode reference `. +which can be freely rejected - for example, when Tarantool is used for caching. +In this case, WAL can be disabled to reduce i/o load. The logging level is INFO or higher ----------------------------------- diff --git a/doc/enterprise/flight_recorder.rst b/doc/enterprise/flight_recorder.rst index 36061b21bd..67184849b7 100644 --- a/doc/enterprise/flight_recorder.rst +++ b/doc/enterprise/flight_recorder.rst @@ -17,8 +17,8 @@ to :ref:`crashing ` a Tarantool instance. .. _enterprise-flight-recorder_enable: -Enabling the flight recorder ----------------------------- +Enable the flight recorder +-------------------------- The flight recorder is disabled by default and can be enabled and configured for a specific Tarantool instance. @@ -33,7 +33,7 @@ configuration option to ``true``. After ``flightrec.enabled`` is set to ``true``, the flight recorder starts collecting data in the flight recording file ``current.ttfr``. -This file is stored in the ``snapshot.dir`` directory. +This file is stored in the :ref:`snapshot.dir ` directory. By default, the directory is ``var/lib/{{ instance_name }}/.ttfr``. If the instance crashes and reboots, Tarantool rotates the flight recording: @@ -49,8 +49,8 @@ Tarantool continues writing to the existing ``current.ttfr`` file after restart. .. _enterprise-flight-recorder_configure: -Configuring a flight recorder ------------------------------ +Configure a flight recorder +--------------------------- When the flight recorder is enabled, you can set the options related to :ref:`logging `, :ref:`metrics `, and storing the :ref:`request and response ` data. diff --git a/doc/enterprise/wal_extensions.rst b/doc/enterprise/wal_extensions.rst index c5eac42f6b..143a014113 100644 --- a/doc/enterprise/wal_extensions.rst +++ b/doc/enterprise/wal_extensions.rst @@ -1,4 +1,4 @@ -.. _wal_extensions: +.. _wal_extensions: WAL extensions ============== @@ -8,47 +8,51 @@ For example, you can enable storing an old and new tuple for each CRUD operation This information might be helpful for implementing a CDC (Change Data Capture) utility that transforms a data replication stream. -.. _wal_extensions_configuration: +See also: :ref:`Configure the write-ahead log `. + +.. _wal_extensions_configuration: Configuration ------------- -To configure WAL extensions, use the ``wal.ext`` :ref:`configuration property `. -Inside the ``wal_ext`` block, you can enable storing old and new tuples as follows: - -* Set the ``old`` and ``new`` options to ``true`` to store old and new tuples in a write-ahead log for all spaces. +WAL extensions are disabled by default. +To configure them, use the :ref:`wal.ext.* ` configuration options. +Inside the ``wal.ext`` block, you can enable storing old and new tuples as follows: - .. code-block:: lua +* To store old and new tuples in a write-ahead log for all spaces, set the + :ref:`wal.ext.old ` and `wal.ext.new ` + options to ``true``: - box.cfg { - wal_ext = { old = true, new = true } - } + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence/config.yaml + :language: yaml + :start-at: wal: + :end-at: old: true + :dedent: -* To adjust these options for specific spaces, use the ``spaces`` option. +* To adjust these options for specific spaces, specify the :ref:`wal.ext.spaces ` option: - .. code-block:: lua + .. code-block:: yaml - box.cfg { - wal_ext = { - old = true, new = true, - spaces = { - space1 = { old = false }, - space2 = { new = false } - } - } - } + wal: + ext: + old: true + new: true + spaces: + space1: + old: false + space2: + new: false - - The configuration for specific spaces has priority over the global configuration, - so only new tuples are added to the log for ``space1`` and only old tuples for ``space2``. + The configuration for specific spaces has priority over the configuration in the ``wal.ext.new`` and ``wal.ext.old`` + options. + It means that only new tuples are added to the log for ``space1`` and only old tuples for ``space2``. Note that records with additional fields are :ref:`replicated ` as follows: * If a replica doesn't support the extended format configured on a master, auxiliary fields are skipped. * If a replica and master have different configurations for WAL records, a master's configuration is ignored. - -.. _wal_extensions_example: +.. _wal_extensions_example: Example ------- diff --git a/doc/reference/configuration/cfg_snapshot_daemon.rst b/doc/reference/configuration/cfg_snapshot_daemon.rst index 0f24fc4495..bff683a573 100644 --- a/doc/reference/configuration/cfg_snapshot_daemon.rst +++ b/doc/reference/configuration/cfg_snapshot_daemon.rst @@ -119,21 +119,6 @@ The checkpoint daemon and the Tarantool garbage collector will not delete a file with calculating how much disk space to allocate for a partition containing WAL files. - For example, suppose - :ref:`checkpoint_interval ` - = 2 and - :ref:`checkpoint_count ` - = 5 - and the average amount that Tarantool writes between each checkpoint interval - = 1 GB. - Then one could calculate that the necessary amount is (2*5*1) 10GB. - But this calculation would be wrong if, instead of writing 1 GB - during one checkpoint interval, - Tarantool encounters an unusual spike and tries to write 11 GB, - causing an operating-system ENOSPC ("no space") error. - By setting ``checkpoint_wal_threshold`` to a lower value, say 9 GB, - an administrator could prevent the error. - | Type: integer | Default: 10^18 (a large number so in effect there is no limit by default) | Environment variable: TT_CHECKPOINT_WAL_THRESHOLD diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index 910efbc08f..eaa373ba1e 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2395,89 +2395,171 @@ The ``security`` section defines configuration parameters related to various sec snapshot -------- -The ``snapshot`` section defines configuration parameters related to +The ``snapshot`` section defines configuration parameters related to the :ref:`snapshot files `. +To learn more about the snapshots' configuration, check the :ref:`Persistence ` page. -.. NOTE:: +.. NOTE:: ``snapshot`` can be defined in any :ref:`scope `. -- :ref:`snapshot.by.interval ` -- :ref:`snapshot.by.wal_size ` -- :ref:`snapshot.count ` - :ref:`snapshot.dir ` - :ref:`snapshot.snap_io_rate_limit ` +- :ref:`checkpoint daemon ` -.. _configuration_reference_snapshot_by_interval: + - :ref:`snapshot.count ` + - :ref:`snapshot.by.interval ` + - :ref:`snapshot.by.wal_size ` -.. confval:: snapshot.by.interval +.. _configuration_reference_snapshot_dir: - Description +.. confval:: snapshot.dir + + A directory where memtx stores snapshot (.snap) files. Can be relative to + ``process.work_dir``. If not specified, defaults to + ``process.work_dir``. See also: :ref:`wal.dir `. | - | Type: number - | Default: 3600 - | Environment variable: TT_SNAPSHOT_BY_INTERVAL + | Type: string + | Default: 'var/lib/{{ instance_name }}' + | Environment variable: TT_SNAPSHOT_DIR -.. _configuration_reference_snapshot_by_wal_size: +.. _configuration_reference_snapshot_snap_io_rate_limit: -.. confval:: snapshot.by.wal_size +.. confval:: snapshot.snap_io_rate_limit - Description + Reduce the throttling effect of :doc:`box.snapshot() ` on + INSERT/UPDATE/DELETE performance by setting a limit on how many + megabytes per second it can write to disk. The same can be + achieved by splitting :ref:`wal.dir ` and + :ref:`snapshot.dir ` + locations and moving snapshots to a separate disk. + The limit also affects what + :ref:`box.stat.vinyl().regulator ` + may show for the write rate of dumps to ``.run`` and ``.index`` files. | | Type: number - | Default: 3600 - | Environment variable: TT_SNAPSHOT_BY_WAL_SIZE + | Default: box.NULL + | Environment variable: TT_SNAPSHOT_SNAP_IO_RATE_LIMIT + +.. _configuration_reference_checkpoint_daemon: + +Checkpoint daemon +~~~~~~~~~~~~~~~~~ + +The checkpoint daemon is a constantly running :ref:`fiber `. +At intervals, it may make new :ref:`snapshot (.snap) files ` and then delete old snapshot files. +The checkpoint daemon may activate the Tarantool garbage collector that deletes old files -- snapshot and WAL files. + +.. NOTE:: + + This garbage collector is distinct from the `Lua garbage collector `_ + which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. + +If the checkpoint daemon deletes an old snapshot file, the Tarantool garbage collector also deletes +any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: + +* The WAL files are older than the snapshot file. +* The WAL files contain information present in the snapshot file. + +Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. + +The checkpoint daemon and the Tarantool garbage collector don't delete a file in the following cases: + +* A **backup** is running, and the file has not been backed up + (see :ref:`"Hot backup" `). + +* **Replication** is running, and the file has not been relayed to a replica + (see :ref:`"Replication architecture" `), + +* A replica is connecting. + +* A replica has fallen behind. + The progress of each replica is tracked; if a replica's position is far + from being up to date, then the server stops to give it a chance to catch up. + If an administrator concludes that a replica is permanently down, then the + correct procedure is to restart the server, or (preferably) :ref:`remove the replica from the cluster `. + .. _configuration_reference_snapshot_count: .. confval:: snapshot.count - Description + The maximum number of snapshots that may exist on the + :ref:`snapshot.dir ` directory + before the checkpoint daemon deletes old snapshots. + If ``snapshot.count`` equals zero, then the checkpoint daemon + does not delete old snapshots. + + Example: + + .. code-block:: yaml + + snapshot: + by: + interval: 60 + count: 10 + + In the example, the checkpoint daemon creates a new snapshot each hour until + it has created ten snapshots. After that, it deletes the oldest snapshot + (and any associated write-ahead-log files) after creating a new one. + + .. NOTE:: + + Snapshots will not be deleted if replication is ongoing and the file has not been relayed to a replica. + Therefore, ``snapshot.count`` has no effect unless all replicas are alive. | | Type: integer | Default: 2 | Environment variable: TT_SNAPSHOT_COUNT -.. _configuration_reference_snapshot_dir: - -.. confval:: snapshot.dir +.. _configuration_reference_snapshot_by_interval: - Description +.. confval:: snapshot.by.interval - | - | Type: string - | Default: 'var/lib/{{ instance_name }}' - | Environment variable: TT_SNAPSHOT_DIR + The interval in seconds between actions by the checkpoint daemon. + If the option is set to a value greater than zero, and there is + activity which causes change to a database, then the checkpoint daemon calls + :doc:`box.snapshot() ` every ``snapshot.by.interval`` + seconds, creating a new snapshot file each time. + If the option is set to zero, the checkpoint daemon is disabled. + Example: -.. _configuration_reference_snapshot_snap_io_rate_limit: - -.. confval:: snapshot.snap_io_rate_limit + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: by: + :end-at: interval: 60 + :dedent: - Reduce the throttling effect of :doc:`box.snapshot() ` on - INSERT/UPDATE/DELETE performance by setting a limit on how many - megabytes per second it can write to disk. The same can be - achieved by splitting :ref:`wal.dir ` and - :ref:`memtx_dir ` - locations and moving snapshots to a separate disk. - The limit also affects what - :ref:`box.stat.vinyl().regulator ` - may show for the write rate of dumps to ``.run`` and ``.index`` files. + In the example, the checkpoint daemon creates a new database snapshot once per minute, if there is activity. | | Type: number - | Default: box.NULL - | Environment variable: TT_SNAPSHOT_SNAP_IO_RATE_LIMIT + | Default: 3600 + | Environment variable: TT_SNAPSHOT_BY_INTERVAL + +.. _configuration_reference_snapshot_by_wal_size: + +.. confval:: snapshot.by.wal_size + The threshold for the total size in bytes of all WAL files created since the last snapshot taken. + Once the configured threshold is exceeded, the WAL thread notifies the + checkpoint daemon that it must make a new snapshot and delete old WAL files. + + | + | Type: integer + | Default: 10^18 + | Environment variable: TT_SNAPSHOT_BY_WAL_SIZE .. _configuration_reference_wal: wal --- -The ``wal`` section defines configuration parameters related to +The ``wal`` section defines configuration parameters related to :ref:`write-ahead log `. +To learn more about the WAL configuration, check the :ref:`Persistence ` page. .. NOTE:: @@ -2486,18 +2568,20 @@ The ``wal`` section defines configuration parameters related to - :ref:`wal.cleanup_delay ` - :ref:`wal.dir ` - :ref:`wal.dir_rescan_delay ` -- :ref:`wal.ext.new ` -- :ref:`wal.ext.old ` -- :ref:`wal.ext.spaces ` -- :ref:`wal.wal_max_size ` +- :ref:`wal.max_size ` - :ref:`wal.mode ` - :ref:`wal.queue_max_size ` +- :ref:`wal.ext.* ` + + - :ref:`wal.ext.new ` + - :ref:`wal.ext.old ` + - :ref:`wal.ext.spaces ` .. _configuration_reference_wal_cleanup_delay: .. confval:: wal.cleanup_delay - The delay in seconds used to prevent the :ref:`Tarantool garbage collector ` + The delay in seconds used to prevent the :ref:`checkpoint daemon ` from immediately removing :ref:`write-ahead log ` files after a node restart. This delay eliminates possible erroneous situations when the master deletes WALs needed by :ref:`replicas ` after restart. @@ -2507,10 +2591,10 @@ The ``wal`` section defines configuration parameters related to Once all the nodes in the replica set are up and running, automatic cleanup is started again even if ``wal.cleanup_delay`` has not expired. - .. NOTE:: + .. NOTE:: The option has no effect on nodes running as - :ref:`anonymous replicas `. + :ref:`anonymous replicas `. | | Type: number @@ -2522,10 +2606,10 @@ The ``wal`` section defines configuration parameters related to .. confval:: wal.dir A directory where write-ahead log (.xlog) files are stored. Can be relative - to :ref:`work_dir `. Sometimes ``wal_dir`` and - :ref:`memtx_dir ` are specified with different values, so + to ``process.work_dir``. Sometimes ``wal.dir`` and + :ref:`snapshot.dir ` are specified with different values, so that write-ahead log files and snapshot files can be stored on different - disks. If not specified, defaults to ``work_dir``. + disks. If not specified, defaults to ``process.work_dir``. | | Type: string @@ -2545,51 +2629,13 @@ The ``wal`` section defines configuration parameters related to | Default: 2 | Environment variable: TT_WAL_DIR_RESCAN_DELAY -.. _configuration_reference_wal_ext_new: - -.. confval:: wal.ext_new - - (**Enterprise Edition only**) - - Description - - | - | Type: boolean - | Default: nil - | Environment variable: TT_WAL_EXT_NEW - -.. _configuration_reference_wal_ext_old: - -.. confval:: wal.ext_old - - (**Enterprise Edition only**) - Description - - | - | Type: boolean - | Default: nil - | Environment variable: TT_WAL_EXT_OLD - -.. _configuration_reference_wal_ext_spaces: - -.. confval:: wal.ext_spaces - - (**Enterprise Edition only**) - Description - - | - | Type: map - | Default: nil - | Environment variable: TT_WAL_EXT_SPACES - - .. _configuration_reference_wal_max_size: -.. confval:: wal.wal_max_size +.. confval:: wal.max_size The maximum number of bytes in a single write-ahead log file. When a request would cause an ``.xlog`` file to become larger than - ``wal_max_size``, Tarantool creates a new WAL file. + ``wal.max_size``, Tarantool creates a new WAL file. | | Type: integer @@ -2606,7 +2652,7 @@ The ``wal`` section defines configuration parameters related to A node with ``wal.mode = none`` can't be replication master. * ``write``: :ref:`fibers ` wait for their data to be written to - the write-ahead log (no :manpage:`fsync(2)`). + the write-ahead log (no ``fsync(2)``). * ``fsync``: fibers wait for their data, :manpage:`fsync(2)` follows each :manpage:`write(2)`. @@ -2621,7 +2667,7 @@ The ``wal`` section defines configuration parameters related to .. confval:: wal.queue_max_size The size of the queue in bytes used by a :ref:`replica ` to submit - new transactions to a :ref:`write-ahead log` (WAL). + new transactions to a :ref:`write-ahead log ` (WAL). This option helps limit the rate at which a replica submits transactions to the WAL. Limiting the queue size might be useful when a replica is trying to sync with a master and reads new transactions faster than writing them to the WAL. @@ -2634,4 +2680,83 @@ The ``wal`` section defines configuration parameters related to | | Type: integer | Default: 16777216 - | Environment variable: TT_WAL_QUEUE_MAX_SIZE \ No newline at end of file + | Environment variable: TT_WAL_QUEUE_MAX_SIZE + +.. _configuration_reference_wal_ext: + +wal.ext.* +~~~~~~~~~ + +.. admonition:: Enterprise Edition + :class: fact + + Configuring ``wal.ext.*`` parameters is available in the `Enterprise Edition `_ only. + +This section describes options related to :ref:`WAL extensions `. +The options allow you to add auxiliary information to each :ref:`write-ahead log ` record. + +Note that records with additional fields are :ref:`replicated ` as follows: + +* If a replica doesn't support the extended format configured on a master, auxiliary fields are skipped. +* If a replica and master have different configurations for WAL records, a master's configuration is ignored. + +.. _configuration_reference_wal_ext_new: + +.. confval:: wal.ext.new + + Enable storing a new tuple for each :ref:`CRUD ` operation performed. + The option is in effect for all spaces. + To adjust the option for specific spaces, use the :ref:`wal.ext.spaces <_configuration_reference_wal_ext_spaces>` + option. + + | + | Type: boolean + | Default: false + | Environment variable: TT_WAL_EXT_NEW + +.. _configuration_reference_wal_ext_old: + +.. confval:: wal.ext.old + + Enable storing an old tuple for each :ref:`CRUD ` operation performed. + The option is in effect for all spaces. + To adjust the option for specific spaces, use the :ref:`wal.ext.spaces <_configuration_reference_wal_ext_spaces>` + option. + + | + | Type: boolean + | Default: false + | Environment variable: TT_WAL_EXT_OLD + +.. _configuration_reference_wal_ext_spaces: + +.. confval:: wal.ext.spaces + + Enable or disable storing an old and new tuple in the :ref:`WAL ` record + for a given space explicitly. + The configuration for specific spaces has priority over the configuration in the + :ref:`wal.ext.new ` and :ref:`wal.ext.old ` + options. + + The option is a key-value pair: + + * The key is a space name (string). + + * The value is a table that includes two optional boolean options: ``old`` and ``new``. + The format and the default value of these options are described in ``wal.ext.old`` and ``wal.ext.new``. + + + Example: + + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml + :language: yaml + :start-at: ext: + :end-at: old: false + :dedent: + + In the example, only new tuples are added to the log for the ``bands`` space. + + | + | Type: map + | Default: nil + | Environment variable: TT_WAL_EXT_SPACES diff --git a/doc/reference/reference_lua/box_snapshot.rst b/doc/reference/reference_lua/box_snapshot.rst index 01e91c524b..c1106d3179 100644 --- a/doc/reference/reference_lua/box_snapshot.rst +++ b/doc/reference/reference_lua/box_snapshot.rst @@ -9,7 +9,7 @@ **Memtx** Take a snapshot of all data and store it in - :ref:`memtx_dir `:samp:`/{}.snap`. + :ref:`snapshot.dir `:samp:`/{}.snap`. To take a snapshot, Tarantool first enters the delayed garbage collection mode for all data. In this mode, the :ref:`Tarantool garbage collector ` @@ -26,7 +26,7 @@ performance (averaging to 80MB/second on modern disks), which means an average database instance gets saved in a matter of minutes. You may restrict the speed by changing - :ref:`snap_io_rate_limit `. + :ref:`snapshot.snap_io_rate_limit `. .. NOTE:: From fef77287a7362609c92e4d4f7b580bae62753936 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Mon, 12 Feb 2024 14:24:29 +0300 Subject: [PATCH 03/10] persistence: update checkpoint daemon section --- .../configuration_persistence.rst | 111 +++++++++++++----- .../configuration/configuration_reference.rst | 56 ++------- doc/reference/reference_lua/box_snapshot.rst | 2 +- 3 files changed, 91 insertions(+), 78 deletions(-) diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst index 8f532b1e5f..5d13a51cd1 100644 --- a/doc/concepts/configuration/configuration_persistence.rst +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -5,8 +5,9 @@ Persistence To ensure data persistence, Tarantool provides the abilities to: -* record each data change request into a :ref:`write-ahead log ` (WAL) file (``.xlog`` files) -* take :ref:`snapshots ` that contain on-disk copy of the entire data set for a given moment (``.snap`` files) +* Record each data change request into a :ref:`write-ahead log ` (WAL) file (``.xlog`` files). +* Take :ref:`snapshots ` that contain on-disk copy of the entire data set for a given moment + (``.snap`` files). It is possible to set automatic snapshot creation using the :ref:`checkpoint daemon `. During the recovery process, Tarantool can load the latest snapshot file and then read the requests from the WAL files, produced after this snapshot was made. @@ -23,6 +24,34 @@ Configure the snapshots This section describes how to define snapshot settings in the :ref:`snapshot ` section of a YAML configuration. +.. _configuration_persistence_snapshot_creation: + +Configure the automatic snapshot creation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In Tarantool, it is possible to set automatic :ref:`snapshot creation `. +To enable it, the :ref:`snapshot.by.interval ` option is used. +The option sets up the :ref:`checkpoint daemon ` that takes new snapshots +every ``snapshot.by.interval`` seconds. +When the number of snapshots reaches the limit of :ref:`snapshot.count ` size, +the daemon activates Tarantool garbage collector after the new snapshot is taken. +Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. + +The :ref:`snapshot.by.wal_size ` option defines the maximum size in bytes +for of all WAL files created since the last snapshot taken. +Once this size is exceeded, the checkpoint daemon takes a snapshot and deletes the old WAL files. + +The configuration of the checkpoint daemon might look as follows: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: count: + :end-at: 60 + :dedent: + +If the ``snapshot.by.interval`` option is set to zero, the checkpoint daemon is disabled. +If the ``snapshot.count`` option is set to zero, the checkpoint daemon does not delete old snapshots. + .. _configuration_persistence_snapshot_dir: Specify a directory for snapshot files @@ -50,35 +79,6 @@ For example, you can place snapshots and write-ahead logs on different hard driv wal: dir: '/media/drive2/wals' -.. _configuration_persistence_checkpoint_daemon: - -Configure the automatic snapshot creation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In Tarantool, it is possible to set automatic :ref:`snapshot creation `. -To enable it, the :ref:`snapshot.by.interval ` option is used. -The option sets up the :ref:`checkpoint daemon ` (snapshot daemon), which is -a constantly running :ref:`fiber `. -The checkpoint daemon takes new snapshots every ``snapshot.by.interval`` seconds. -When the number of snapshots reaches the limit of :ref:`snapshot.count ` size, -the daemon activates Tarantool garbage collector after the new snapshot is taken. -Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. - -The :ref:`snapshot.by.wal_size ` option defines the maximum size in bytes -for of all WAL files created since the last snapshot taken. -Once this size is exceeded, the checkpoint daemon takes a snapshot and deletes the old WAL files. - -The configuration of the checkpoint daemon might look as follows: - -.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml - :language: yaml - :start-at: count: - :end-at: 60 - :dedent: - -If the ``snapshot.by.interval`` option is set to zero, the checkpoint daemon is disabled. -If the ``snapshot.count`` option is set to zero, the checkpoint daemon does not delete old snapshots. - .. _configuration_persistence_wal: Configure the write-ahead log @@ -157,7 +157,7 @@ size. The configuration for this option might look as follows: Set a delay for the garbage collector ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In Tarantool, the :ref:`checkpoint daemon ` (snapshot daemon) +In Tarantool, the :ref:`checkpoint daemon ` takes new snapshots at the given interval (see :ref:`snapshot.by.interval `). After an instance restart, the daemon activates the Tarantool garbage collector that deletes the old WAL files. @@ -184,3 +184,50 @@ In Tarantool Enterprise, you can store an old and new tuple for each crud operat The detailed description and examples of the WAL extensions are provided in the :ref:`WAL extensions ` section. See also: :ref:`wal.ext.* ` configuration options. + +.. _configuration_persistence_checkpoint_daemon: + +Checkpoint daemon +----------------- + +The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. +If the checkpoint daemon is enabled, it takes new :ref:`snapshot (.snap) files ` at the +:ref:`given interval ` automatically. +If necessary, the checkpoint daemon also activates the Tarantool garbage collector that deletes old snapshot and WAL files. + +.. NOTE:: + + This garbage collector is distinct from the `Lua garbage collector `_ + which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. + +This garbage collector is called as follows: + +* When the number of snapshots reaches the limit of :ref:`snapshot.count ` size. + After a new snapshot is taken, Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. + +* When the size all WAL files created since the last snapshot reaches the limit of :ref:`snapshot.by.wal_size `. + Once this size is exceeded, the checkpoint daemon takes a snapshot, then the garbage collector deletes the old WAL files. + +If the checkpoint daemon deletes an old snapshot file, the Tarantool garbage collector also deletes +any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: + +* The WAL files are older than the snapshot file. +* The WAL files contain information present in the snapshot file. + +Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. + +The checkpoint daemon and the Tarantool garbage collector don't delete a file in the following cases: + +* A **backup** is running, and the file has not been backed up + (see :ref:`"Hot backup" `). + +* **Replication** is running, and the file has not been relayed to a replica + (see :ref:`"Replication architecture" `), + +* A replica is connecting. + +* A replica has fallen behind. + The progress of each replica is tracked; if a replica's position is far + from being up to date, then the server stops to give it a chance to catch up. + If an administrator concludes that a replica is permanently down, then the + correct procedure is to restart the server, or (preferably) :ref:`remove the replica from the cluster `. diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index eaa373ba1e..421656d59a 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2404,9 +2404,9 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence ` - :ref:`snapshot.snap_io_rate_limit ` -- :ref:`checkpoint daemon ` +- :ref:`snapshot.count ` +- :ref:`snapshot.by.* ` - - :ref:`snapshot.count ` - :ref:`snapshot.by.interval ` - :ref:`snapshot.by.wal_size ` @@ -2442,52 +2442,13 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence `. -At intervals, it may make new :ref:`snapshot (.snap) files ` and then delete old snapshot files. -The checkpoint daemon may activate the Tarantool garbage collector that deletes old files -- snapshot and WAL files. - -.. NOTE:: - - This garbage collector is distinct from the `Lua garbage collector `_ - which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. - -If the checkpoint daemon deletes an old snapshot file, the Tarantool garbage collector also deletes -any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: - -* The WAL files are older than the snapshot file. -* The WAL files contain information present in the snapshot file. - -Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. - -The checkpoint daemon and the Tarantool garbage collector don't delete a file in the following cases: - -* A **backup** is running, and the file has not been backed up - (see :ref:`"Hot backup" `). - -* **Replication** is running, and the file has not been relayed to a replica - (see :ref:`"Replication architecture" `), - -* A replica is connecting. - -* A replica has fallen behind. - The progress of each replica is tracked; if a replica's position is far - from being up to date, then the server stops to give it a chance to catch up. - If an administrator concludes that a replica is permanently down, then the - correct procedure is to restart the server, or (preferably) :ref:`remove the replica from the cluster `. - - .. _configuration_reference_snapshot_count: .. confval:: snapshot.count The maximum number of snapshots that may exist on the :ref:`snapshot.dir ` directory - before the checkpoint daemon deletes old snapshots. + before the :ref:`checkpoint daemon ` deletes old snapshots. If ``snapshot.count`` equals zero, then the checkpoint daemon does not delete old snapshots. @@ -2514,11 +2475,16 @@ The checkpoint daemon and the Tarantool garbage collector don't delete a file in | Default: 2 | Environment variable: TT_SNAPSHOT_COUNT +.. _configuration_reference_snapshot_by: + +snapshot.by.* +~~~~~~~~~~~~~ + .. _configuration_reference_snapshot_by_interval: .. confval:: snapshot.by.interval - The interval in seconds between actions by the checkpoint daemon. + The interval in seconds between actions by the :ref:`checkpoint daemon `. If the option is set to a value greater than zero, and there is activity which causes change to a database, then the checkpoint daemon calls :doc:`box.snapshot() ` every ``snapshot.by.interval`` @@ -2546,7 +2512,7 @@ The checkpoint daemon and the Tarantool garbage collector don't delete a file in The threshold for the total size in bytes of all WAL files created since the last snapshot taken. Once the configured threshold is exceeded, the WAL thread notifies the - checkpoint daemon that it must make a new snapshot and delete old WAL files. + :ref:`checkpoint daemon ` that it must make a new snapshot and delete old WAL files. | | Type: integer @@ -2581,7 +2547,7 @@ To learn more about the WAL configuration, check the :ref:`Persistence ` + The delay in seconds used to prevent the :ref:`checkpoint daemon ` from immediately removing :ref:`write-ahead log ` files after a node restart. This delay eliminates possible erroneous situations when the master deletes WALs needed by :ref:`replicas ` after restart. diff --git a/doc/reference/reference_lua/box_snapshot.rst b/doc/reference/reference_lua/box_snapshot.rst index c1106d3179..9bcf3d9dd1 100644 --- a/doc/reference/reference_lua/box_snapshot.rst +++ b/doc/reference/reference_lua/box_snapshot.rst @@ -46,7 +46,7 @@ Although ``box.snapshot()`` does not cause a fork, there is a separate fiber which may produce snapshots at regular intervals -- see the discussion of - the :ref:`checkpoint daemon `. + the :ref:`checkpoint daemon `. **Example:** From 739e35269240f0fce834f3370bdbce25c6bef716 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Mon, 19 Feb 2024 11:58:50 +0300 Subject: [PATCH 04/10] persistence: apply review suggestions --- .../persistence_snapshot/config.yaml | 5 +- .../configuration_persistence.rst | 107 ++++++++++++------ doc/dev_guide/internals/file_formats.rst | 38 ++++--- doc/reference/configuration/cfg_basic.rst | 20 ++-- .../cfg_binary_logging_snapshots.rst | 7 +- .../configuration/cfg_snapshot_daemon.rst | 39 ++++--- .../configuration/configuration_reference.rst | 47 ++++---- 7 files changed, 160 insertions(+), 103 deletions(-) diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml index 08f46c1c52..ac71476864 100644 --- a/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml @@ -6,9 +6,10 @@ groups: instance001: snapshot: dir: 'var/lib/{{ instance_name }}/snapshots' - count: 10 + count: 3 by: - interval: 60 + interval: 7200 + wal_size: 1000000000000000000 iproto: listen: - uri: '127.0.0.1:3301' \ No newline at end of file diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst index 5d13a51cd1..5489f13120 100644 --- a/doc/concepts/configuration/configuration_persistence.rst +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -26,32 +26,32 @@ This section describes how to define snapshot settings in the :ref:`snapshot `. -To enable it, the :ref:`snapshot.by.interval ` option is used. -The option sets up the :ref:`checkpoint daemon ` that takes new snapshots -every ``snapshot.by.interval`` seconds. -When the number of snapshots reaches the limit of :ref:`snapshot.count ` size, -the daemon activates Tarantool garbage collector after the new snapshot is taken. -Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. +In Tarantool, it is possible to automate the :ref:`snapshot creation `. +Automatic creation is enabled by default and can be configured in two ways: -The :ref:`snapshot.by.wal_size ` option defines the maximum size in bytes -for of all WAL files created since the last snapshot taken. -Once this size is exceeded, the checkpoint daemon takes a snapshot and deletes the old WAL files. +* A new snapshot is taken once in a given period (see :ref:`snapshot.by.interval `). +* A new snapshot is taken once the size of all WAL files created since the last snapshot exceeds a given limit + (see :ref:`snapshot.by.wal_size `). -The configuration of the checkpoint daemon might look as follows: +The ``snapshot.by.interval`` option sets up the :ref:`checkpoint daemon ` +that takes a new snapshot every ``snapshot.by.interval`` seconds. +If the ``snapshot.by.interval`` option is set to zero, the checkpoint daemon is disabled. + +The ``snapshot.by.wal_size`` option defines the maximum size in bytes for of all WAL files created since the last snapshot taken. +Once this size is exceeded, the checkpoint daemon takes a snapshot. Then, :ref:`Tarantool garbage collector ` +deletes the old WAL files. + +The example shows how to specify the ``snapshot.by.interval`` and the ``snapshot.by.wal_size`` options: .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml :language: yaml - :start-at: count: - :end-at: 60 + :start-at: by: + :end-at: 1000000000000000000 :dedent: -If the ``snapshot.by.interval`` option is set to zero, the checkpoint daemon is disabled. -If the ``snapshot.count`` option is set to zero, the checkpoint daemon does not delete old snapshots. - .. _configuration_persistence_snapshot_dir: Specify a directory for snapshot files @@ -79,6 +79,28 @@ For example, you can place snapshots and write-ahead logs on different hard driv wal: dir: '/media/drive2/wals' +.. _configuration_persistence_snapshot_count: + +Configure a maximum number of stored snapshots +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can set a limit on the number of snapshots stored in the :ref:`snapshot.dir ` +directory using the :ref:`snapshot.count ` option. +Once the number of snapshots reaches the given limit, :ref:`Tarantool garbage collector ` +deletes the oldest snapshot file and any associated WAL files after the new snapshot is taken. + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml + :language: yaml + :start-at: count: + :end-at: 7200 + :dedent: + +In the example, the snapshot is created every two hours (every 7200 seconds) until there are three snapshots in the +``snapshot.dir`` directory. +After creating a new snapshot (the fourth one), the oldest snapshot and the corresponding WALs are deleted. + +If the ``snapshot.count`` option is set to zero, the garbage collector does not delete old snapshots. + .. _configuration_persistence_wal: Configure the write-ahead log @@ -93,8 +115,15 @@ This section describes how to define WAL settings in the :ref:`wal ` configuration option to ``write`` or ``fsync``. +The recording to the write-ahead log is enabled by default. +It means that if an instance restart occurs, the data will be recovered. +The recording to the WAL can be configured using the :ref:`wal.mode ` configuration option. + +There are two modes that enable writing to the WAL: + +* ``write`` (default) -- enable WAL and write the data without waiting the data to be flushed to the storage device. +* ``fsync`` -- enable WAL and ensure that the record is written to the storage device. + The example below shows how to specify the ``write`` WAL mode for ``instance001``: .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml @@ -103,9 +132,6 @@ The example below shows how to specify the ``write`` WAL mode for ``instance001` :end-at: 'write' :dedent: -The ``write`` mode enables WAL and writes the data without waiting the data to be flushed to the storage device. -The ``fsync`` mode enables WAL and ensures that the record is written to the storage device. - To turn the WAL writer off, set the ``wal.mode`` option to ``none``. .. _configuration_persistence_wal_dir: @@ -159,7 +185,7 @@ Set a delay for the garbage collector In Tarantool, the :ref:`checkpoint daemon ` takes new snapshots at the given interval (see :ref:`snapshot.by.interval `). -After an instance restart, the daemon activates the Tarantool garbage collector that deletes the old WAL files. +After an instance restart, the Tarantool garbage collector deletes the old WAL files. To delay the immediate deletion of WAL files, use the :ref:`wal.cleanup_delay ` configuration option. The delay eliminates possible erroneous situations when the master deletes WALs @@ -180,7 +206,7 @@ In the example, the delay is set to 5 hours (18000 seconds): Specify the WAL extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~ -In Tarantool Enterprise, you can store an old and new tuple for each crud operation performed. +In Tarantool Enterprise, you can store an old and new tuple for each CRUD operation performed. The detailed description and examples of the WAL extensions are provided in the :ref:`WAL extensions ` section. See also: :ref:`wal.ext.* ` configuration options. @@ -191,13 +217,30 @@ Checkpoint daemon ----------------- The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. -If the checkpoint daemon is enabled, it takes new :ref:`snapshot (.snap) files ` at the -:ref:`given interval ` automatically. -If necessary, the checkpoint daemon also activates the Tarantool garbage collector that deletes old snapshot and WAL files. +The checkpoint daemon creates a schedule for the periodic snapshot creation based on +the :ref:`configuration options `and the speed of file size growth. +If enabled, the daemon makes new snapshots (``.snap``) files according to this schedule. + +The work of checkpoint daemon is based on the following configuration options: + +* :ref:`snapshot.by.interval ` -- a new snapshot is taken once in a given period. +* :ref:`snapshot.by.wal_size ` -- a new snapshot is taken once the size + of all WAL files created since the last snapshot exceeds a given limit. + +If necessary, the checkpoint daemon also activates the :ref:`Tarantool garbage collector ` that deletes old snapshot and WAL files. + +.. _configuration_persistence_garbage_collector: + +Tarantool garbage collector +--------------------------- + +Tarantool garbage collector can be activated by the :ref:`checkpoint daemon `. +The garbage collector tracks the snapshots that are to be :ref:`relayed to a replica ` or needed +by other consumers. When the files are no longer needed, Tarantool garbage collector deletes them. .. NOTE:: - This garbage collector is distinct from the `Lua garbage collector `_ + The garbage collector called by the checkpoint daemon, is distinct from the `Lua garbage collector `_ which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. This garbage collector is called as follows: @@ -205,10 +248,10 @@ This garbage collector is called as follows: * When the number of snapshots reaches the limit of :ref:`snapshot.count ` size. After a new snapshot is taken, Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. -* When the size all WAL files created since the last snapshot reaches the limit of :ref:`snapshot.by.wal_size `. +* When the size of all WAL files created since the last snapshot reaches the limit of :ref:`snapshot.by.wal_size `. Once this size is exceeded, the checkpoint daemon takes a snapshot, then the garbage collector deletes the old WAL files. -If the checkpoint daemon deletes an old snapshot file, the Tarantool garbage collector also deletes +If an old snapshot filen is deleted, the Tarantool garbage collector also deletes any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: * The WAL files are older than the snapshot file. @@ -216,7 +259,7 @@ any :ref:`write-ahead log (.xlog) ` files that meet the following Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. -The checkpoint daemon and the Tarantool garbage collector don't delete a file in the following cases: +Tarantool garbage collector doesn't delete a file in the following cases: * A **backup** is running, and the file has not been backed up (see :ref:`"Hot backup" `). diff --git a/doc/dev_guide/internals/file_formats.rst b/doc/dev_guide/internals/file_formats.rst index 128fa9b99e..edf7b55c42 100644 --- a/doc/dev_guide/internals/file_formats.rst +++ b/doc/dev_guide/internals/file_formats.rst @@ -5,22 +5,26 @@ File formats .. _internals-wal: -Data persistence and the WAL file format ----------------------------------------- +The WAL file format +------------------- -To maintain data persistence, Tarantool writes each data change request (insert, -update, delete, replace, upsert) into a write-ahead log (WAL) file in the +To maintain :ref:`data persistence `, Tarantool writes each data change request (insert, +update, delete, replace, upsert) to a write-ahead log (WAL) file in the :ref:`wal.dir ` directory. -Each data change request gets assigned a continuously growing 64-bit log sequence +Each data change request is assigned a continuously growing 64-bit log sequence number. The name of the WAL file is based on the log sequence number of the first record in the file, plus an extension ``.xlog``. A new WAL file is created when the current one reaches the :ref:`wal_max_size ` size. -Apart from a log sequence number and the data change request (formatted as in -:ref:`Tarantool's binary protocol `), -each WAL record contains a header, some metadata, and then the data formatted -according to `msgpack `_ rules. +Each WAL record contains: + +* a log sequence number +* a data change request (formatted as in :ref:`Tarantool's binary protocol `) + +* a header +* some metadata +* the data formatted according to `msgpack `_ rules. To see the hexadecimal bytes of the given WAL file, use the ``hexdump`` command: @@ -102,7 +106,7 @@ It is possible to turn the write-ahead log completely off, by setting the ``wal_ Even without the write-ahead log it's still possible to take a persistent copy of the entire data set with the :ref:`box.snapshot() ` request. -An .xlog file always contains changes based on the primary key. +An ``.xlog`` file always contains changes based on the primary key. Even if the client requested an update or delete using a secondary key, the record in the .xlog file contains the primary key. @@ -111,12 +115,14 @@ a secondary key, the record in the .xlog file contains the primary key. The snapshot file format ------------------------ -The format of a snapshot .snap file is similar to the format of a WAL .xlog file, except for the header and content. -The snapshot header contains the instance's global unique identifier -and the snapshot file's position in history, relative to earlier snapshot files. -Also, the content differs: an .xlog file may contain records for any data-change -requests (inserts, updates, upserts, and deletes), a .snap file may only contain records -of inserts to memtx spaces. +The format of a snapshot (``.snap``) file is the following: + +* The snapshot header contains the instance's global unique identifier + and the snapshot file's position in history, relative to earlier snapshot files. + +* The snapshot content contains the records of inserts to memtx spaces. + That differs from the content of an ``.xlog`` file that may contain records for any data-change requests + (inserts, updates, upserts, and deletes). Primarily, the records in the snapshot file have the following order: diff --git a/doc/reference/configuration/cfg_basic.rst b/doc/reference/configuration/cfg_basic.rst index 495b7f1eb2..8d289f95c0 100644 --- a/doc/reference/configuration/cfg_basic.rst +++ b/doc/reference/configuration/cfg_basic.rst @@ -100,9 +100,12 @@ Since version 1.7.4. - A directory where memtx stores snapshot (.snap) files. Can be relative to - :ref:`work_dir `. If not specified, defaults to - ``work_dir``. See also :ref:`wal_dir `. + A directory where memtx stores snapshot (.snap) files. + A relative path in this option is interpreted as relative to :ref:`work_dir `. + + By default, snapshots and WAL files are stored in the same directory. + However, you can set different values for the ``memtx_dir`` and :ref:`wal_dir ` options + to store them on different physical disks for performance matters. | Type: string | Default: "." @@ -230,11 +233,12 @@ Since version 1.6.2. - A directory where write-ahead log (.xlog) files are stored. Can be relative - to :ref:`work_dir `. Sometimes ``wal_dir`` and - :ref:`memtx_dir ` are specified with different values, so - that write-ahead log files and snapshot files can be stored on different - disks. If not specified, defaults to ``work_dir``. + A directory where write-ahead log (.xlog) files are stored. + A relative path in this option is interpreted as relative to :ref:`work_dir `. + + By default, WAL files and snapshots are stored in the same directory. + However, you can set different values for the ``wal_dir`` and :ref:`memtx_dir ` options + to store them on different physical disks for performance matters. | Type: string | Default: "." diff --git a/doc/reference/configuration/cfg_binary_logging_snapshots.rst b/doc/reference/configuration/cfg_binary_logging_snapshots.rst index 0da363eb45..4fc7c87537 100644 --- a/doc/reference/configuration/cfg_binary_logging_snapshots.rst +++ b/doc/reference/configuration/cfg_binary_logging_snapshots.rst @@ -130,15 +130,14 @@ Since version :doc:`2.6.3 `. - The delay (in seconds) used to prevent the :ref:`Tarantool garbage collector ` + The delay in seconds used to prevent the :ref:`Tarantool garbage collector ` from immediately removing :ref:`write-ahead log ` files after a node restart. This delay eliminates possible erroneous situations when the master deletes WALs needed by :ref:`replicas ` after restart. As a consequence, replicas sync with the master faster after its restart and don't need to download all the data again. - - Once all the nodes in the replica set are up and running, - automatic cleanup is started again even if ``wal_cleanup_delay`` has not expired. + Once all the nodes in the replica set are up and running, a scheduled garbage collection is started again + even if ``wal_cleanup_delay`` has not expired. .. NOTE:: diff --git a/doc/reference/configuration/cfg_snapshot_daemon.rst b/doc/reference/configuration/cfg_snapshot_daemon.rst index bff683a573..57ab8dd70f 100644 --- a/doc/reference/configuration/cfg_snapshot_daemon.rst +++ b/doc/reference/configuration/cfg_snapshot_daemon.rst @@ -52,21 +52,18 @@ The checkpoint daemon and the Tarantool garbage collector will not delete a file Since version 1.7.4. - The interval between actions by the checkpoint daemon, in seconds. If + The interval in seconds between actions by the checkpoint daemon. If ``checkpoint_interval`` is set to a value greater than zero, and there is - activity which causes change to a database, then the checkpoint daemon will - call :doc:`box.snapshot() ` every ``checkpoint_interval`` + activity which causes change to a database, then the checkpoint daemon + calls :doc:`box.snapshot() ` every ``checkpoint_interval`` seconds, creating a new snapshot file each time. If ``checkpoint_interval`` - is set to zero, then the checkpoint daemon is disabled. - - For example: + is set to zero, the checkpoint daemon is disabled. .. code-block:: lua - box.cfg{checkpoint_interval=60} + box.cfg{ checkpoint_interval = 7200 } - will cause the checkpoint daemon to create a new database snapshot once - per minute, if there is activity. + In the example, the checkpoint daemon creates a new database snapshot every two hours, if there is activity. | Type: integer | Default: 3600 (one hour) @@ -79,26 +76,28 @@ The checkpoint daemon and the Tarantool garbage collector will not delete a file Since version 1.7.4. - The maximum number of snapshots that may exist on the + The maximum number of snapshots that are stored in the :ref:`memtx_dir ` directory - before the checkpoint daemon will delete old snapshots. - If ``checkpoint_count`` equals zero, then the checkpoint daemon - does not delete old snapshots. For example: + before the checkpoint daemon deletes old snapshots. + If ``checkpoint_count`` is set to zero, the checkpoint daemon + does not delete old snapshots. .. code-block:: lua box.cfg{ - checkpoint_interval = 3600, - checkpoint_count = 10 + checkpoint_interval = 7200, + checkpoint_count = 3 } - will cause the checkpoint daemon to create a new snapshot each hour until - it has created ten snapshots. After that, it will delete the oldest snapshot + In the example, the checkpoint daemon creates a new snapshot every two hours until + it has created three snapshots. After that, it deletes the oldest snapshot (and any associated write-ahead-log files) after creating a new one. - Remember that, as noted earlier, snapshots will not be deleted if - replication is ongoing and the file has not been relayed to a replica. - Therefore ``checkpoint_count`` has no effect unless all replicas are alive. + .. NOTE:: + + Snapshots will not be deleted if replication is ongoing and the file has not been relayed to a replica. + Therefore, ``snapshot.count`` has no effect unless all replicas are alive. + | Type: integer | Default: 2 diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index 421656d59a..48a263024e 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2414,9 +2414,12 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence `. + A directory where memtx stores snapshot (.snap) files. + A relative path in this option is interpreted as relative to ``process.work_dir``. + + By default, snapshots and WAL files are stored in the same directory. + However, you can set different values for the ``snapshot.dir`` and :ref:`wal.dir ` options + to store them on different physical disks for performance matters. | | Type: string @@ -2446,10 +2449,10 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence ` directory - before the :ref:`checkpoint daemon ` deletes old snapshots. - If ``snapshot.count`` equals zero, then the checkpoint daemon + before the :ref:`Tarantool garbage collector ` deletes old snapshots. + If ``snapshot.count`` is set to zero zero, the garbage collector does not delete old snapshots. Example: @@ -2458,12 +2461,12 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence ` + The delay in seconds used to prevent the :ref:`Tarantool garbage collector ` from immediately removing :ref:`write-ahead log ` files after a node restart. This delay eliminates possible erroneous situations when the master deletes WALs needed by :ref:`replicas ` after restart. As a consequence, replicas sync with the master faster after its restart and don't need to download all the data again. + Once all the nodes in the replica set are up and running, a scheduled garbage collection is started again + even if ``wal_cleanup_delay`` has not expired. - Once all the nodes in the replica set are up and running, - automatic cleanup is started again even if ``wal.cleanup_delay`` has not expired. .. NOTE:: @@ -2571,11 +2574,12 @@ To learn more about the WAL configuration, check the :ref:`Persistence ` are specified with different values, so - that write-ahead log files and snapshot files can be stored on different - disks. If not specified, defaults to ``process.work_dir``. + A directory where write-ahead log (.xlog) files are stored. + A relative path in this option is interpreted as relative to ``process.work_dir``. + + By default, WAL files and snapshots are stored in the same directory. + However, you can set different values for the ``wal.dir`` and :ref:`snapshot.dir ` options + to store them on different physical disks for performance matters. | | Type: string @@ -2635,6 +2639,7 @@ To learn more about the WAL configuration, check the :ref:`Persistence ` to submit new transactions to a :ref:`write-ahead log ` (WAL). This option helps limit the rate at which a replica submits transactions to the WAL. + Limiting the queue size might be useful when a replica is trying to sync with a master and reads new transactions faster than writing them to the WAL. From bbb7cd79d9315bb950df11e3d75104f31b96dffc Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Tue, 20 Feb 2024 13:56:13 +0300 Subject: [PATCH 05/10] persistence: add minor fixes --- doc/concepts/configuration/configuration_persistence.rst | 4 ++-- doc/enterprise/wal_extensions.rst | 2 +- doc/reference/configuration/configuration_reference.rst | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst index 5489f13120..5a6b4da12e 100644 --- a/doc/concepts/configuration/configuration_persistence.rst +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -218,7 +218,7 @@ Checkpoint daemon The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. The checkpoint daemon creates a schedule for the periodic snapshot creation based on -the :ref:`configuration options `and the speed of file size growth. +the :ref:`configuration options ` and the speed of file size growth. If enabled, the daemon makes new snapshots (``.snap``) files according to this schedule. The work of checkpoint daemon is based on the following configuration options: @@ -240,7 +240,7 @@ by other consumers. When the files are no longer needed, Tarantool garbage colle .. NOTE:: - The garbage collector called by the checkpoint daemon, is distinct from the `Lua garbage collector `_ + The garbage collector called by the checkpoint daemon is distinct from the `Lua garbage collector `_ which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. This garbage collector is called as follows: diff --git a/doc/enterprise/wal_extensions.rst b/doc/enterprise/wal_extensions.rst index 143a014113..55c41b43c2 100644 --- a/doc/enterprise/wal_extensions.rst +++ b/doc/enterprise/wal_extensions.rst @@ -20,7 +20,7 @@ To configure them, use the :ref:`wal.ext.* ` co Inside the ``wal.ext`` block, you can enable storing old and new tuples as follows: * To store old and new tuples in a write-ahead log for all spaces, set the - :ref:`wal.ext.old ` and `wal.ext.new ` + :ref:`wal.ext.old ` and :ref:`wal.ext.new ` options to ``true``: .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence/config.yaml diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index 48a263024e..dca80d6194 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2452,7 +2452,7 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence ` directory before the :ref:`Tarantool garbage collector ` deletes old snapshots. - If ``snapshot.count`` is set to zero zero, the garbage collector + If ``snapshot.count`` is set to zero, the garbage collector does not delete old snapshots. Example: @@ -2466,7 +2466,7 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence Date: Tue, 20 Feb 2024 14:21:03 +0300 Subject: [PATCH 06/10] persistence: minor fixes --- .../persistence_wal/config.yaml | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml index 6350ad3b21..522bd7752c 100644 --- a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml @@ -1,21 +1,22 @@ +wal: + mode: 'write' + dir: 'var/lib/{{ instance_name }}/wals' + dir_rescan_delay: 3 + cleanup_delay: 18000 + max_size: 268435456 + ext: + new: true + old: true + spaces: + bands: + new: false + groups: group001: replicasets: replicaset001: instances: instance001: - wal: - mode: 'write' - dir: 'var/lib/{{ instance_name }}/wals' - dir_rescan_delay: 3 - cleanup_delay: 18000 - max_size: 268435456 - ext: - new: true - old: true - spaces: - bands: - new: false iproto: listen: - uri: '127.0.0.1:3301' \ No newline at end of file From 4c1ca4b2319c4584f3f6815452722aa177e11137 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Tue, 20 Feb 2024 21:04:40 +0300 Subject: [PATCH 07/10] persistence: apply review suggestions --- .../persistence_wal/config.yaml | 25 ++-- .../configuration_persistence.rst | 42 +++--- doc/enterprise/wal_extensions.rst | 2 +- doc/reference/configuration/cfg_basic.rst | 4 +- .../cfg_binary_logging_snapshots.rst | 6 +- .../configuration/cfg_snapshot_daemon.rst | 124 ++++++++++-------- .../configuration/configuration_reference.rst | 29 ++-- 7 files changed, 128 insertions(+), 104 deletions(-) diff --git a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml index 522bd7752c..a073a3f2e2 100644 --- a/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml +++ b/doc/code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml @@ -1,22 +1,21 @@ -wal: - mode: 'write' - dir: 'var/lib/{{ instance_name }}/wals' - dir_rescan_delay: 3 - cleanup_delay: 18000 - max_size: 268435456 - ext: - new: true - old: true - spaces: - bands: - new: false - groups: group001: replicasets: replicaset001: instances: instance001: + wal: + dir: 'var/lib/{{ instance_name }}/wals' + mode: 'write' + dir_rescan_delay: 3 + cleanup_delay: 18000 + max_size: 268435456 + ext: + new: true + old: true + spaces: + bands: + old: false iproto: listen: - uri: '127.0.0.1:3301' \ No newline at end of file diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst index 5a6b4da12e..540967cf3e 100644 --- a/doc/concepts/configuration/configuration_persistence.rst +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -6,11 +6,15 @@ Persistence To ensure data persistence, Tarantool provides the abilities to: * Record each data change request into a :ref:`write-ahead log ` (WAL) file (``.xlog`` files). -* Take :ref:`snapshots ` that contain on-disk copy of the entire data set for a given moment +* Take :ref:`snapshots ` that contain an on-disk copy of the entire data set for a given moment (``.snap`` files). It is possible to set automatic snapshot creation using the :ref:`checkpoint daemon `. During the recovery process, Tarantool can load the latest snapshot file and then read the requests from the WAL files, produced after this snapshot was made. +This topic describes how to configure: + +* the snapshot creation in the :ref:`snapshot ` section of a :ref:`YAML configuration `. +* the recording to the write-ahead log in the :ref:`wal ` section of a YAML configuration. To learn more about the persistence mechanism in Tarantool, see the :ref:`Persistence ` section. The formats of WAL and snapshot files are described in detail in the :ref:`File formats ` section. @@ -40,7 +44,7 @@ The ``snapshot.by.interval`` option sets up the :ref:`checkpoint daemon ` deletes the old WAL files. @@ -52,6 +56,11 @@ The example shows how to specify the ``snapshot.by.interval`` and the ``snapshot :end-at: 1000000000000000000 :dedent: +In the example, a new snapshot is created in two cases: + +* every 2 hours (every 7200 seconds) +* when the size for all WAL files created since the last snapshot reaches the size of 1e18 (1000000000000000000) bytes. + .. _configuration_persistence_snapshot_dir: Specify a directory for snapshot files @@ -121,14 +130,14 @@ The recording to the WAL can be configured using the :ref:`wal.mode ` or :ref:`hot standby ` mode, +In case of :ref:`replication ` or :ref:`hot standby ` mode, Tarantool scans for changes in the WAL files every :ref:`wal.dir_rescan_delay ` seconds. The example below shows how to specify the interval between scans: @@ -207,7 +216,7 @@ Specify the WAL extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~ In Tarantool Enterprise, you can store an old and new tuple for each CRUD operation performed. -The detailed description and examples of the WAL extensions are provided in the :ref:`WAL extensions ` section. +A detailed description and examples of the WAL extensions are provided in the :ref:`WAL extensions ` section. See also: :ref:`wal.ext.* ` configuration options. @@ -219,15 +228,16 @@ Checkpoint daemon The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. The checkpoint daemon creates a schedule for the periodic snapshot creation based on the :ref:`configuration options ` and the speed of file size growth. -If enabled, the daemon makes new snapshots (``.snap``) files according to this schedule. +If enabled, the daemon makes new :ref:`snapshot (``.snap``) files ` according to this schedule. -The work of checkpoint daemon is based on the following configuration options: +The work of the checkpoint daemon is based on the following configuration options: * :ref:`snapshot.by.interval ` -- a new snapshot is taken once in a given period. * :ref:`snapshot.by.wal_size ` -- a new snapshot is taken once the size of all WAL files created since the last snapshot exceeds a given limit. -If necessary, the checkpoint daemon also activates the :ref:`Tarantool garbage collector ` that deletes old snapshot and WAL files. +If necessary, the checkpoint daemon also activates the :ref:`Tarantool garbage collector ` +that deletes old snapshots and WAL files. .. _configuration_persistence_garbage_collector: @@ -251,7 +261,7 @@ This garbage collector is called as follows: * When the size of all WAL files created since the last snapshot reaches the limit of :ref:`snapshot.by.wal_size `. Once this size is exceeded, the checkpoint daemon takes a snapshot, then the garbage collector deletes the old WAL files. -If an old snapshot filen is deleted, the Tarantool garbage collector also deletes +If an old snapshot file is deleted, the Tarantool garbage collector also deletes any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: * The WAL files are older than the snapshot file. @@ -261,11 +271,11 @@ Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. Tarantool garbage collector doesn't delete a file in the following cases: -* A **backup** is running, and the file has not been backed up - (see :ref:`"Hot backup" `). +* A backup is running, and the file has not been backed up + (see :ref:`Hot backup `). -* **Replication** is running, and the file has not been relayed to a replica - (see :ref:`"Replication architecture" `), +* Replication is running, and the file has not been relayed to a replica + (see :ref:`Replication architecture `), * A replica is connecting. diff --git a/doc/enterprise/wal_extensions.rst b/doc/enterprise/wal_extensions.rst index 55c41b43c2..5be4967530 100644 --- a/doc/enterprise/wal_extensions.rst +++ b/doc/enterprise/wal_extensions.rst @@ -50,7 +50,7 @@ Inside the ``wal.ext`` block, you can enable storing old and new tuples as follo Note that records with additional fields are :ref:`replicated ` as follows: * If a replica doesn't support the extended format configured on a master, auxiliary fields are skipped. -* If a replica and master have different configurations for WAL records, a master's configuration is ignored. +* If a replica and master have different configurations for WAL records, the master's configuration is ignored. .. _wal_extensions_example: diff --git a/doc/reference/configuration/cfg_basic.rst b/doc/reference/configuration/cfg_basic.rst index 8d289f95c0..aee6609527 100644 --- a/doc/reference/configuration/cfg_basic.rst +++ b/doc/reference/configuration/cfg_basic.rst @@ -100,7 +100,7 @@ Since version 1.7.4. - A directory where memtx stores snapshot (.snap) files. + A directory where memtx stores snapshot (``.snap``) files. A relative path in this option is interpreted as relative to :ref:`work_dir `. By default, snapshots and WAL files are stored in the same directory. @@ -233,7 +233,7 @@ Since version 1.6.2. - A directory where write-ahead log (.xlog) files are stored. + A directory where write-ahead log (``.xlog``) files are stored. A relative path in this option is interpreted as relative to :ref:`work_dir `. By default, WAL files and snapshots are stored in the same directory. diff --git a/doc/reference/configuration/cfg_binary_logging_snapshots.rst b/doc/reference/configuration/cfg_binary_logging_snapshots.rst index 4fc7c87537..a649e8663d 100644 --- a/doc/reference/configuration/cfg_binary_logging_snapshots.rst +++ b/doc/reference/configuration/cfg_binary_logging_snapshots.rst @@ -74,7 +74,7 @@ Specify fiber-WAL-disk synchronization mode as: * ``none``: write-ahead log is not maintained. - A node with ``wal_mode = none`` can't be replication master. + A node with ``wal_mode`` set to ``none`` can't be a replication master. * ``write``: :ref:`fibers ` wait for their data to be written to the write-ahead log (no :manpage:`fsync(2)`). @@ -93,7 +93,7 @@ Since version 1.6.2. - Number of seconds between periodic scans of the write-ahead-log + The time interval in seconds between periodic scans of the write-ahead-log file directory, when checking for changes to write-ahead-log files for the sake of :ref:`replication ` or :ref:`hot standby `. @@ -191,7 +191,7 @@ Note that records with additional fields are :ref:`replicated ` as follows: * If a replica doesn't support the extended format configured on a master, auxiliary fields are skipped. - * If a replica and master have different configurations for WAL records, a master's configuration is ignored. + * If a replica and master have different configurations for WAL records, the master's configuration is ignored. | Type: map | Default: nil diff --git a/doc/reference/configuration/cfg_snapshot_daemon.rst b/doc/reference/configuration/cfg_snapshot_daemon.rst index 57ab8dd70f..2e84883f8c 100644 --- a/doc/reference/configuration/cfg_snapshot_daemon.rst +++ b/doc/reference/configuration/cfg_snapshot_daemon.rst @@ -2,64 +2,83 @@ * :ref:`checkpoint_interval ` * :ref:`checkpoint_wal_threshold ` -The checkpoint daemon is a fiber which is constantly running. At intervals, -it may make new :ref:`snapshot (.snap) files ` and then -may delete old snapshot files. +.. _cfg_checkpoint_daemon: -The :ref:`checkpoint_interval ` and -:ref:`checkpoint_count ` configuration -settings determine how long the intervals are, and how many snapshots should -exist before deletions occur. +**Checkpoint daemon** -.. _cfg_checkpoint_daemon-garbage-collector: +The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. +The checkpoint daemon creates a schedule for the periodic snapshot creation based on +the configuration options and the speed of file size growth. +If enabled, the daemon makes new :ref:`snapshot (``.snap``) files ` according to this schedule. + +The work of the checkpoint daemon is based on the following configuration options: + +* :ref:`checkpoint_interval ` -- a new snapshot is taken once in a given period. +* :ref:`cfg_checkpoint_daemon-checkpoint_wal_threshold>` -- a new snapshot is taken once the size + of all WAL files created since the last snapshot exceeds a given limit. + +If necessary, the checkpoint daemon also activates the :ref:`Tarantool garbage collector ` +that deletes old snapshots and WAL files. + +.. _cfg_checkpoint_daemon-garbage-collector: **Tarantool garbage collector** -The checkpoint daemon may activate the Tarantool garbage collector -which deletes old files. This garbage collector is distinct from the -`Lua garbage collector `_ -which is for Lua objects, and distinct from a -Tarantool garbage collector which specializes in -:ref:`handling shard buckets `. +Tarantool garbage collector can be activated by the :ref:`checkpoint daemon `. +The garbage collector tracks the snapshots that are to be :ref:`relayed to a replica ` or needed +by other consumers. When the files are no longer needed, Tarantool garbage collector deletes them. + +.. NOTE:: + + The garbage collector called by the checkpoint daemon is distinct from the `Lua garbage collector `_ + which is for Lua objects, and distinct from the Tarantool garbage collector that specializes in :ref:`handling shard buckets `. + +This garbage collector is called as follows: + +* When the number of snapshots reaches the limit of :ref:`checkpoint_count ` size. + After a new snapshot is taken, Tarantool garbage collector deletes the oldest snapshot file and any associated WAL files. + +* When the size of all WAL files created since the last snapshot reaches the limit of :ref:`checkpoint_wal_threshold `. + Once this size is exceeded, the checkpoint daemon takes a snapshot, then the garbage collector deletes the old WAL files. + +If an old snapshot file is deleted, the Tarantool garbage collector also deletes +any :ref:`write-ahead log (.xlog) ` files that meet the following conditions: + +* The WAL files are older than the snapshot file. +* The WAL files contain information present in the snapshot file. -If the checkpoint daemon deletes an old snapshot file, then the -Tarantool garbage collector will also delete -any :ref:`write-ahead log (.xlog) ` files which are older than -the snapshot file and which contain information that is present in the snapshot -file. It will also delete obsolete vinyl ``.run`` files. +Tarantool garbage collector also deletes obsolete vinyl ``.run`` files. -The checkpoint daemon and the Tarantool garbage collector will not delete a file if: +Tarantool garbage collector doesn't delete a file in the following cases: -* a **backup** is ongoing and the file has not been backed up - (see :ref:`"Hot backup" `), or +* A backup is running, and the file has not been backed up + (see :ref:`Hot backup `). -* **replication** is ongoing and the file has not been relayed to a replica - (see :ref:`"Replication architecture" `), +* Replication is running, and the file has not been relayed to a replica + (see :ref:`Replication architecture `), -* a replica is connecting, or +* A replica is connecting. -* a replica has fallen behind. - The progress of each replica is tracked; if a replica's position is far - from being up to date, then the server stops to give it a chance to - catch up. - If an administrator concludes that a replica is permanently down, then the - correct procedure is to restart the server, or (preferably) - :ref:`remove the replica from the cluster `. +* A replica has fallen behind. + The progress of each replica is tracked; if a replica's position is far + from being up to date, then the server stops to give it a chance to catch up. + If an administrator concludes that a replica is permanently down, then the + correct procedure is to restart the server, or (preferably) :ref:`remove the replica from the cluster `. -.. _cfg_checkpoint_daemon-checkpoint_interval: +.. _cfg_checkpoint_daemon-checkpoint_interval: -.. confval:: checkpoint_interval +.. confval:: checkpoint_interval Since version 1.7.4. - The interval in seconds between actions by the checkpoint daemon. If - ``checkpoint_interval`` is set to a value greater than zero, and there is - activity which causes change to a database, then the checkpoint daemon + The interval in seconds between actions by the :ref:`checkpoint daemon `. + If the option is set to a value greater than zero, and there is + activity that causes change to a database, then the checkpoint daemon calls :doc:`box.snapshot() ` every ``checkpoint_interval`` - seconds, creating a new snapshot file each time. If ``checkpoint_interval`` + seconds, creating a new snapshot file each time. If the option is set to zero, the checkpoint daemon is disabled. - .. code-block:: lua + .. code-block:: lua box.cfg{ checkpoint_interval = 7200 } @@ -70,19 +89,20 @@ The checkpoint daemon and the Tarantool garbage collector will not delete a file | Environment variable: TT_CHECKPOINT_INTERVAL | Dynamic: yes -.. _cfg_checkpoint_daemon-checkpoint_count: +.. _cfg_checkpoint_daemon-checkpoint_count: -.. confval:: checkpoint_count +.. confval:: checkpoint_count Since version 1.7.4. The maximum number of snapshots that are stored in the - :ref:`memtx_dir ` directory - before the checkpoint daemon deletes old snapshots. - If ``checkpoint_count`` is set to zero, the checkpoint daemon + :ref:`memtx_dir ` directory. + If the number of snapshots after creating a new one exceeds this value, + the :ref:`Tarantool garbage collector ` deletes old snapshots. + If the option is set to zero, the garbage collector does not delete old snapshots. - .. code-block:: lua + .. code-block:: lua box.cfg{ checkpoint_interval = 7200, @@ -90,13 +110,13 @@ The checkpoint daemon and the Tarantool garbage collector will not delete a file } In the example, the checkpoint daemon creates a new snapshot every two hours until - it has created three snapshots. After that, it deletes the oldest snapshot - (and any associated write-ahead-log files) after creating a new one. + it has created three snapshots. After creating a new snapshot (the fourth one), the oldest snapshot + and any associated write-ahead-log files are deleted. .. NOTE:: Snapshots will not be deleted if replication is ongoing and the file has not been relayed to a replica. - Therefore, ``snapshot.count`` has no effect unless all replicas are alive. + Therefore, ``checkpoint_count`` has no effect unless all replicas are alive. | Type: integer @@ -104,15 +124,15 @@ The checkpoint daemon and the Tarantool garbage collector will not delete a file | Environment variable: TT_CHECKPOINT_COUNT | Dynamic: yes -.. _cfg_checkpoint_daemon-checkpoint_wal_threshold: +.. _cfg_checkpoint_daemon-checkpoint_wal_threshold: -.. confval:: checkpoint_wal_threshold +.. confval:: checkpoint_wal_threshold Since version 2.1.2. - The threshold for the total size in bytes of all WAL files created since the last checkpoint. + The threshold for the total size in bytes for all WAL files created since the last checkpoint. Once the configured threshold is exceeded, the WAL thread notifies the - checkpoint daemon that it must make a new checkpoint and delete old WAL files. + :ref:`checkpoint daemon ` that it must make a new checkpoint and delete old WAL files. This parameter enables administrators to handle a problem that could occur with calculating how much disk space to allocate for a partition containing diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index dca80d6194..d5684734c7 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2450,8 +2450,9 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence ` directory - before the :ref:`Tarantool garbage collector ` deletes old snapshots. + :ref:`snapshot.dir ` directory. + If the number of snapshots after creating a new one exceeds this value, + the :ref:`Tarantool garbage collector ` deletes old snapshots. If ``snapshot.count`` is set to zero, the garbage collector does not delete old snapshots. @@ -2489,7 +2490,7 @@ snapshot.by.* The interval in seconds between actions by the :ref:`checkpoint daemon `. If the option is set to a value greater than zero, and there is - activity which causes change to a database, then the checkpoint daemon calls + activity that causes change to a database, then the checkpoint daemon calls :doc:`box.snapshot() ` every ``snapshot.by.interval`` seconds, creating a new snapshot file each time. If the option is set to zero, the checkpoint daemon is disabled. @@ -2513,7 +2514,7 @@ snapshot.by.* .. confval:: snapshot.by.wal_size - The threshold for the total size in bytes of all WAL files created since the last snapshot taken. + The threshold for the total size in bytes for all WAL files created since the last snapshot taken. Once the configured threshold is exceeded, the WAL thread notifies the :ref:`checkpoint daemon ` that it must make a new snapshot and delete old WAL files. @@ -2557,7 +2558,7 @@ To learn more about the WAL configuration, check the :ref:`Persistence ` or :ref:`hot standby `. + files for the sake of :ref:`replication ` or :ref:`hot standby `. | | Type: number @@ -2619,7 +2620,7 @@ To learn more about the WAL configuration, check the :ref:`Persistence ` wait for their data to be written to the write-ahead log (no ``fsync(2)``). @@ -2664,12 +2665,6 @@ wal.ext.* Configuring ``wal.ext.*`` parameters is available in the `Enterprise Edition `_ only. This section describes options related to :ref:`WAL extensions `. -The options allow you to add auxiliary information to each :ref:`write-ahead log ` record. - -Note that records with additional fields are :ref:`replicated ` as follows: - -* If a replica doesn't support the extended format configured on a master, auxiliary fields are skipped. -* If a replica and master have different configurations for WAL records, a master's configuration is ignored. .. _configuration_reference_wal_ext_new: @@ -2677,7 +2672,7 @@ Note that records with additional fields are :ref:`replicated ` operation performed. The option is in effect for all spaces. - To adjust the option for specific spaces, use the :ref:`wal.ext.spaces <_configuration_reference_wal_ext_spaces>` + To adjust the option for specific spaces, use the :ref:`wal.ext.spaces ` option. | @@ -2691,7 +2686,7 @@ Note that records with additional fields are :ref:`replicated ` operation performed. The option is in effect for all spaces. - To adjust the option for specific spaces, use the :ref:`wal.ext.spaces <_configuration_reference_wal_ext_spaces>` + To adjust the option for specific spaces, use the :ref:`wal.ext.spaces ` option. | From b3e2db09631e20c1ee02e664e5fff08dcd828a92 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Tue, 20 Feb 2024 21:12:01 +0300 Subject: [PATCH 08/10] persistence: fix link --- doc/concepts/configuration/configuration_persistence.rst | 2 +- doc/reference/configuration/cfg_snapshot_daemon.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst index 540967cf3e..e9a3d053b2 100644 --- a/doc/concepts/configuration/configuration_persistence.rst +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -228,7 +228,7 @@ Checkpoint daemon The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. The checkpoint daemon creates a schedule for the periodic snapshot creation based on the :ref:`configuration options ` and the speed of file size growth. -If enabled, the daemon makes new :ref:`snapshot (``.snap``) files ` according to this schedule. +If enabled, the daemon makes new :ref:`snapshot ` (``.snap``) files according to this schedule. The work of the checkpoint daemon is based on the following configuration options: diff --git a/doc/reference/configuration/cfg_snapshot_daemon.rst b/doc/reference/configuration/cfg_snapshot_daemon.rst index 2e84883f8c..dc63a89b63 100644 --- a/doc/reference/configuration/cfg_snapshot_daemon.rst +++ b/doc/reference/configuration/cfg_snapshot_daemon.rst @@ -9,7 +9,7 @@ The checkpoint daemon (snapshot daemon) is a constantly running :ref:`fiber `. The checkpoint daemon creates a schedule for the periodic snapshot creation based on the configuration options and the speed of file size growth. -If enabled, the daemon makes new :ref:`snapshot (``.snap``) files ` according to this schedule. +If enabled, the daemon makes new :ref:`snapshot ` (``.snap``) files according to this schedule. The work of the checkpoint daemon is based on the following configuration options: From a965b0e2b29fae7ec7c37105fa60c568789cc4ae Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Wed, 21 Feb 2024 09:50:15 +0300 Subject: [PATCH 09/10] persistence: fix link --- doc/reference/configuration/cfg_snapshot_daemon.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/reference/configuration/cfg_snapshot_daemon.rst b/doc/reference/configuration/cfg_snapshot_daemon.rst index dc63a89b63..f72b37c367 100644 --- a/doc/reference/configuration/cfg_snapshot_daemon.rst +++ b/doc/reference/configuration/cfg_snapshot_daemon.rst @@ -14,7 +14,7 @@ If enabled, the daemon makes new :ref:`snapshot ` -- a new snapshot is taken once in a given period. -* :ref:`cfg_checkpoint_daemon-checkpoint_wal_threshold>` -- a new snapshot is taken once the size +* :ref:`checkpoint_wal_threshold ` -- a new snapshot is taken once the size of all WAL files created since the last snapshot exceeds a given limit. If necessary, the checkpoint daemon also activates the :ref:`Tarantool garbage collector ` From 2e2ee25ff0a3180bdf07196c32f0c14d586a55a8 Mon Sep 17 00:00:00 2001 From: Kseniia Antonova Date: Wed, 21 Feb 2024 10:58:39 +0300 Subject: [PATCH 10/10] persistence: add minor updates --- .../configuration_persistence.rst | 10 ++++----- .../configuration/cfg_snapshot_daemon.rst | 4 ++++ .../configuration/configuration_reference.rst | 21 +++++++++---------- 3 files changed, 18 insertions(+), 17 deletions(-) diff --git a/doc/concepts/configuration/configuration_persistence.rst b/doc/concepts/configuration/configuration_persistence.rst index e9a3d053b2..07a080aa72 100644 --- a/doc/concepts/configuration/configuration_persistence.rst +++ b/doc/concepts/configuration/configuration_persistence.rst @@ -98,18 +98,16 @@ directory using the :ref:`snapshot.count ` deletes the oldest snapshot file and any associated WAL files after the new snapshot is taken. +In the example below, the snapshot is created every two hours (every 7200 seconds) until there are three snapshots in the +``snapshot.dir`` directory. +After creating a new snapshot (the fourth one), the oldest snapshot and the corresponding WALs are deleted. + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_snapshot/config.yaml :language: yaml :start-at: count: :end-at: 7200 :dedent: -In the example, the snapshot is created every two hours (every 7200 seconds) until there are three snapshots in the -``snapshot.dir`` directory. -After creating a new snapshot (the fourth one), the oldest snapshot and the corresponding WALs are deleted. - -If the ``snapshot.count`` option is set to zero, the garbage collector does not delete old snapshots. - .. _configuration_persistence_wal: Configure the write-ahead log diff --git a/doc/reference/configuration/cfg_snapshot_daemon.rst b/doc/reference/configuration/cfg_snapshot_daemon.rst index f72b37c367..adfa864771 100644 --- a/doc/reference/configuration/cfg_snapshot_daemon.rst +++ b/doc/reference/configuration/cfg_snapshot_daemon.rst @@ -78,6 +78,8 @@ Tarantool garbage collector doesn't delete a file in the following cases: seconds, creating a new snapshot file each time. If the option is set to zero, the checkpoint daemon is disabled. + **Example** + .. code-block:: lua box.cfg{ checkpoint_interval = 7200 } @@ -102,6 +104,8 @@ Tarantool garbage collector doesn't delete a file in the following cases: If the option is set to zero, the garbage collector does not delete old snapshots. + **Example** + .. code-block:: lua box.cfg{ diff --git a/doc/reference/configuration/configuration_reference.rst b/doc/reference/configuration/configuration_reference.rst index d5684734c7..08ea8b8c8b 100644 --- a/doc/reference/configuration/configuration_reference.rst +++ b/doc/reference/configuration/configuration_reference.rst @@ -2456,7 +2456,11 @@ To learn more about the snapshots' configuration, check the :ref:`Persistence ` * The value is a table that includes two optional boolean options: ``old`` and ``new``. The format and the default value of these options are described in ``wal.ext.old`` and ``wal.ext.new``. + **Example** - Example: + In the example, only new tuples are added to the log for the ``bands`` space. .. literalinclude:: /code_snippets/snippets/config/instances.enabled/persistence_wal/config.yaml :language: yaml @@ -2720,8 +2721,6 @@ This section describes options related to :ref:`WAL extensions ` :end-at: old: false :dedent: - In the example, only new tuples are added to the log for the ``bands`` space. - | | Type: map | Default: nil