Improve Snapshot Repository Scalability #74350
Description
Snapshot repositories do not scale well with large numbers of snapshots and indices in them. With the introduction of the cold and frozen tier compelling use cases for repositories containing tens of thousands of snapshots have emerged. This issue tracks the work required to enable larger snapshot repositories.
- switch back to almost 7.x snapshot listing API responses Flatten Get Snapshots Response #74451
- enable efficient writing of large repository metadata blobs Save Memory on Large Repository Metadata Blob Writes #74313
- enable efficient reading of large repository metadata blobs Deserialize BlobStore Metadata Files in a Streaming Manner #73149
- add parameter to filter by SLM policy Add Filtering by SLM Policy to Get Snapshots API #77321
- add the ability to set an offset by concrete sort column value Implement from_sort_value Parameter in Get Snapshots API #77618
- add exclude pattern support to snapshot and repository name path parts Implement Exclude Patterns for Snapshot- and Repository Names in Get Snapshots API #77308
- implement
nextfield for pagination requests Introduce Next Field in Paginated GetSnapshots Response #74236 - implement pagination parameters for get snapshots request Pagination and Sorting for Get Snapshots API #73952
- adjust get snapshot-info API to allow for bulk snapshot-info fetches Add Bulk Fetch SnapshotInfo API to Repository #73570
- add new thread-pool for snapshot metadata and parallelize get-snapshot requests Introduce SNAPSHOT_META Threadpool for Fetching Repository Metadata #73172
- add overall snapshot count to response and a numeric offset parameter to requests to allow for clean pagination in Kibana Return Total Result Count and Remaining Count in Get Snapshots Response #76150
- add query parameter to paginate by numeric offset so that Kibana can jump to page
nefficiently Implement Numeric Offset Parameter in Get Snapshots API #76233 - add sorting snapshots by shard count and failed shard count Add Sort By Shard Count and Failed Shard Count to Get Snapshots API #77011
- add sorting snapshots by repository name Implement Sort By Repository Name in Get Snapshots API #77049
- adjust docs that warn about excessive snapshot counts (e.g. [1]) [DOCS] Remove soft limit for snapshot repositories #80745
- adjust snapshot count limit mentioned in SLM UI (at least, open an issue and give guidance to the UI folks on the changes needed) Remove soft limit for snapshot repositories kibana#118634