Metadata caching (#2626)

* Cache metafiles and iteration results when scanning bucket for blocks.

Signed-off-by: Peter Štibraný <[email protected]>

* Update Thanos to latest master (with fix for metrics and data race)

Signed-off-by: Peter Štibraný <[email protected]>

* Replace storecache.MemcachedBucketCacheProvider with own constant

Signed-off-by: Peter Štibraný <[email protected]>

* Added CHANGELOG.md entry.

Signed-off-by: Peter Štibraný <[email protected]>

* Documentation for metadata cache.

Signed-off-by: Peter Štibraný <[email protected]>

* Documentation for metadata cache.

Signed-off-by: Peter Štibraný <[email protected]>

* Fix json tag, add inline for consistency.

Signed-off-by: Peter Štibraný <[email protected]>

* Fix documentation.

Signed-off-by: Peter Štibraný <[email protected]>

* Use snappy-compression for all iter results.

Signed-off-by: Peter Štibraný <[email protected]>

* Rename ObjectSizeTTL to AttributesTTL.

Signed-off-by: Peter Štibraný <[email protected]>

* Fix documentation.

Signed-off-by: Peter Štibraný <[email protected]>

* Fix help.

Signed-off-by: Peter Štibraný <[email protected]>

* Fix documentation, again. 🤦

Signed-off-by: Peter Štibraný <[email protected]>

Author: pstibrany

CHANGELOG.md

@@ -40,6 +40,7 @@
   * `cortex_compactor_block_cleanup_completed_total`
   * `cortex_compactor_block_cleanup_failed_total`
   * `cortex_compactor_block_cleanup_last_successful_run_timestamp_seconds`
+* [ENHANCEMENT] Experimental TSDB: Use shared cache for metadata. This is especially useful when running multiple querier and store-gateway components to reduce number of object store API calls. #2626
 * [BUGFIX] Ruler: Ensure temporary rule files with special characters are properly mapped and cleaned up. #2506
 * [BUGFIX] Fixes #2411, Ensure requests are properly routed to the prometheus api embedded in the query if `-server.path-prefix` is set. #2372
 * [BUGFIX] Experimental TSDB: fixed chunk data corruption when querying back series using the experimental blocks storage. #2400

docs/configuration/config-file-reference.md

@@ -2807,14 +2807,89 @@ bucket_store:
     # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.max-get-range-requests
     [max_get_range_requests: <int> | default = 3]
 
-    # TTL for caching object size for chunks.
-    # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.object-size-ttl
-    [object_size_ttl: <duration> | default = 24h]
+    # TTL for caching object attributes for chunks.
+    # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.attributes-ttl
+    [attributes_ttl: <duration> | default = 24h]
 
     # TTL for caching individual chunks subranges.
     # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.subrange-ttl
     [subrange_ttl: <duration> | default = 24h]
 
+  metadata_cache:
+    # Backend for metadata cache, if not empty. Supported values: memcached.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.backend
+    [backend: <string> | default = ""]
+
+    memcached:
+      # Comma separated list of memcached addresses. Supported prefixes are:
+      # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query,
+      # dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after
+      # that).
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.addresses
+      [addresses: <string> | default = ""]
+
+      # The socket read/write timeout.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.timeout
+      [timeout: <duration> | default = 100ms]
+
+      # The maximum number of idle connections that will be maintained per
+      # address.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-idle-connections
+      [max_idle_connections: <int> | default = 16]
+
+      # The maximum number of concurrent asynchronous operations can occur.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-async-concurrency
+      [max_async_concurrency: <int> | default = 50]
+
+      # The maximum number of enqueued asynchronous operations allowed.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-async-buffer-size
+      [max_async_buffer_size: <int> | default = 10000]
+
+      # The maximum number of concurrent connections running get operations. If
+      # set to 0, concurrency is unlimited.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-get-multi-concurrency
+      [max_get_multi_concurrency: <int> | default = 100]
+
+      # The maximum number of keys a single underlying get operation should run.
+      # If more keys are specified, internally keys are splitted into multiple
+      # batches and fetched concurrently, honoring the max concurrency. If set
+      # to 0, the max batch size is unlimited.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-get-multi-batch-size
+      [max_get_multi_batch_size: <int> | default = 0]
+
+      # The maximum size of an item stored in memcached. Bigger items are not
+      # stored. If set to 0, no maximum size is enforced.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-item-size
+      [max_item_size: <int> | default = 1048576]
+
+    # How long to cache list of tenants in the bucket.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.tenants-list-ttl
+    [tenants_list_ttl: <duration> | default = 15m]
+
+    # How long to cache list of blocks for each tenant.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.tenant-blocks-list-ttl
+    [tenant_blocks_list_ttl: <duration> | default = 15m]
+
+    # How long to cache list of chunks for a block.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.chunks-list-ttl
+    [chunks_list_ttl: <duration> | default = 24h]
+
+    # How long to cache information that block metafile exists.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-exists-ttl
+    [metafile_exists_ttl: <duration> | default = 2h]
+
+    # How long to cache information that block metafile doesn't exist.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-doesnt-exist-ttl
+    [metafile_doesnt_exist_ttl: <duration> | default = 15m]
+
+    # How long to cache content of the metafile.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-content-ttl
+    [metafile_content_ttl: <duration> | default = 24h]
+
+    # Maximum size of metafile content to cache in bytes.
+    # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-max-size-bytes
+    [metafile_max_size_bytes: <int> | default = 1048576]
+
   # Duration after which the blocks marked for deletion will be filtered out
   # while fetching blocks. The idea of ignore-deletion-marks-delay is to ignore
   # blocks that are marked for deletion with some delay. This ensures store can

docs/operations/blocks-storage.md

@@ -111,6 +111,14 @@ To enable chunks cache, please set `-experimental.tsdb.bucket-store.chunks-cache
 
 There are additional low-level options for configuring chunks cache. Please refer to other flags with `experimental.tsdb.bucket-store.chunks-cache` prefix.
 
+## Metadata cache
+
+Store-gateway and querier can use memcached for storing metadata: list of users, list of blocks per user, meta.json files and deletion mark files. Using the cache can reduce number of API calls to object storage significantly.
+
+To enable metadata cache, please set `-experimental.tsdb.bucket-store.metadata-cache.backend`. Only `memcached` backend is supported currently. Memcached client has additional configuration available via flags with `-experimental.tsdb.bucket-store.metadata-cache.memcached` prefix.
+
+Additional options for configuring metadata cache have `-experimental.tsdb.bucket-store.metadata-cache.` prefix. By configuring TTL to zero or negative value, caching of given item type is disabled.
+
 ## Configuration
 
 The general [configuration documentation](../configuration/_index.md) also applied to a Cortex cluster running the blocks storage, with few differences:
@@ -328,14 +336,89 @@ tsdb:
       # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.max-get-range-requests
       [max_get_range_requests: <int> | default = 3]
 
-      # TTL for caching object size for chunks.
-      # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.object-size-ttl
-      [object_size_ttl: <duration> | default = 24h]
+      # TTL for caching object attributes for chunks.
+      # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.attributes-ttl
+      [attributes_ttl: <duration> | default = 24h]
 
       # TTL for caching individual chunks subranges.
       # CLI flag: -experimental.tsdb.bucket-store.chunks-cache.subrange-ttl
       [subrange_ttl: <duration> | default = 24h]
 
+    metadata_cache:
+      # Backend for metadata cache, if not empty. Supported values: memcached.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.backend
+      [backend: <string> | default = ""]
+
+      memcached:
+        # Comma separated list of memcached addresses. Supported prefixes are:
+        # dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV
+        # query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup
+        # made after that).
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.addresses
+        [addresses: <string> | default = ""]
+
+        # The socket read/write timeout.
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.timeout
+        [timeout: <duration> | default = 100ms]
+
+        # The maximum number of idle connections that will be maintained per
+        # address.
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-idle-connections
+        [max_idle_connections: <int> | default = 16]
+
+        # The maximum number of concurrent asynchronous operations can occur.
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-async-concurrency
+        [max_async_concurrency: <int> | default = 50]
+
+        # The maximum number of enqueued asynchronous operations allowed.
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-async-buffer-size
+        [max_async_buffer_size: <int> | default = 10000]
+
+        # The maximum number of concurrent connections running get operations.
+        # If set to 0, concurrency is unlimited.
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-get-multi-concurrency
+        [max_get_multi_concurrency: <int> | default = 100]
+
+        # The maximum number of keys a single underlying get operation should
+        # run. If more keys are specified, internally keys are splitted into
+        # multiple batches and fetched concurrently, honoring the max
+        # concurrency. If set to 0, the max batch size is unlimited.
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-get-multi-batch-size
+        [max_get_multi_batch_size: <int> | default = 0]
+
+        # The maximum size of an item stored in memcached. Bigger items are not
+        # stored. If set to 0, no maximum size is enforced.
+        # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.memcached.max-item-size
+        [max_item_size: <int> | default = 1048576]
+
+      # How long to cache list of tenants in the bucket.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.tenants-list-ttl
+      [tenants_list_ttl: <duration> | default = 15m]
+
+      # How long to cache list of blocks for each tenant.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.tenant-blocks-list-ttl
+      [tenant_blocks_list_ttl: <duration> | default = 15m]
+
+      # How long to cache list of chunks for a block.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.chunks-list-ttl
+      [chunks_list_ttl: <duration> | default = 24h]
+
+      # How long to cache information that block metafile exists.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-exists-ttl
+      [metafile_exists_ttl: <duration> | default = 2h]
+
+      # How long to cache information that block metafile doesn't exist.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-doesnt-exist-ttl
+      [metafile_doesnt_exist_ttl: <duration> | default = 15m]
+
+      # How long to cache content of the metafile.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-content-ttl
+      [metafile_content_ttl: <duration> | default = 24h]
+
+      # Maximum size of metafile content to cache in bytes.
+      # CLI flag: -experimental.tsdb.bucket-store.metadata-cache.metafile-max-size-bytes
+      [metafile_max_size_bytes: <int> | default = 1048576]
+
     # Duration after which the blocks marked for deletion will be filtered out
     # while fetching blocks. The idea of ignore-deletion-marks-delay is to
     # ignore blocks that are marked for deletion with some delay. This ensures

docs/operations/blocks-storage.template

@@ -111,6 +111,14 @@ To enable chunks cache, please set `-experimental.tsdb.bucket-store.chunks-cache
 
 There are additional low-level options for configuring chunks cache. Please refer to other flags with `experimental.tsdb.bucket-store.chunks-cache` prefix.
 
+## Metadata cache
+
+Store-gateway and querier can use memcached for storing metadata: list of users, list of blocks per user, meta.json files and deletion mark files. Using the cache can reduce number of API calls to object storage significantly.
+
+To enable metadata cache, please set `-experimental.tsdb.bucket-store.metadata-cache.backend`. Only `memcached` backend is supported currently. Memcached client has additional configuration available via flags with `-experimental.tsdb.bucket-store.metadata-cache.memcached` prefix.
+
+Additional options for configuring metadata cache have `-experimental.tsdb.bucket-store.metadata-cache.` prefix. By configuring TTL to zero or negative value, caching of given item type is disabled.
+
 ## Configuration
 
 The general [configuration documentation](../configuration/_index.md) also applied to a Cortex cluster running the blocks storage, with few differences:

go.mod

@@ -50,7 +50,7 @@ require (
 	github.com/segmentio/fasthash v0.0.0-20180216231524-a72b379d632e
 	github.com/spf13/afero v1.2.2
 	github.com/stretchr/testify v1.4.0
-	github.com/thanos-io/thanos v0.12.3-0.20200520075802-806479182a6b
+	github.com/thanos-io/thanos v0.12.3-0.20200524133339-079ad4274d5b
 	github.com/uber/jaeger-client-go v2.20.1+incompatible
 	github.com/weaveworks/common v0.0.0-20200512154658-384f10054ec5
 	go.etcd.io/bbolt v1.3.4

go.sum

@@ -919,8 +919,8 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
 github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/thanos-io/thanos v0.8.1-0.20200109203923-552ffa4c1a0d/go.mod h1:usT/TxtJQ7DzinTt+G9kinDQmRS5sxwu0unVKZ9vdcw=
-github.com/thanos-io/thanos v0.12.3-0.20200520075802-806479182a6b h1:BQPyohMtq8uV+6eoVywL+3DW14454uWZzBtY0aMpNqw=
-github.com/thanos-io/thanos v0.12.3-0.20200520075802-806479182a6b/go.mod h1:I/DAcQA84wLoyYvz5g/jpStEf/dm6nF0KXlDHvv7xQ8=
+github.com/thanos-io/thanos v0.12.3-0.20200524133339-079ad4274d5b h1:ne+i2dLDoxsnKauoX2itJSod4julDLwXhykaXDn/BTE=
+github.com/thanos-io/thanos v0.12.3-0.20200524133339-079ad4274d5b/go.mod h1:I/DAcQA84wLoyYvz5g/jpStEf/dm6nF0KXlDHvv7xQ8=
 github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
 github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4=
 github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=

pkg/storage/tsdb/bucket_client_mock.go

@@ -8,6 +8,7 @@ import (
 	"io/ioutil"
 
 	"github.com/stretchr/testify/mock"
+	"github.com/thanos-io/thanos/pkg/objstore"
 )
 
 var errObjectDoesNotExist = errors.New("object does not exist")
@@ -112,10 +113,10 @@ func (m *BucketClientMock) IsObjNotFoundErr(err error) bool {
 	return err == errObjectDoesNotExist
 }
 
-// ObjectSize mocks objstore.Bucket.ObjectSize()
-func (m *BucketClientMock) ObjectSize(ctx context.Context, name string) (uint64, error) {
+// ObjectSize mocks objstore.Bucket.Attributes()
+func (m *BucketClientMock) Attributes(ctx context.Context, name string) (objstore.ObjectAttributes, error) {
 	args := m.Called(ctx, name)
-	return args.Get(0).(uint64), args.Error(1)
+	return args.Get(0).(objstore.ObjectAttributes), args.Error(1)
 }
 
 // Close mocks objstore.Bucket.Close()