Implement Metering for Blobstore Repository API Requests

[original-brownbear]

We need to be able to inspect the number of API requests made to cloud backed snapshot repositories (S3, GCS, Azure).
This entails creating implementations of org.elasticsearch.common.blobstore.BlobStore#stats for GCS and Azure as well, analogous to what we have for S3 already and completing the S3 implementation to cover all types of requests (at the time of writing this it only covers GET and LIST requests).

The existing code for metering S3 requests was implemented in #55051.

• Implement the remaining request types (PUT and POST) in the all repositories (Track multipart/resumable uploads GCS API calls #56821, Track upload requests on S3 repositories #56826, Track PUT/PUT_BLOCK operations on AzureBlobStore. #56936)
• Implement counting of requests by type for GCS Track GET/LIST GoogleCloudStorage API calls #56585
• Implement counting of requests by type for Azure Track GET/LIST Azure Storage API calls  #56773
• Expose request counts via API or logging

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[original-brownbear]

cc @suyograo

[yoavderazon]

thanks for the issue, @original-brownbear .

I am assuming that the classification of the type of request (verb) to the matching billing dimension will be in the cloud billing side, and be tailored for each cloud provider.

question: will we also provide metadata that indicates the context in which an API request was made ? for example, as part of r "snapshot restore" vs. "searchable snapshot" ?

[original-brownbear]

np @yoavderazon

I am assuming that the classification of the type of request (verb) to the matching billing dimension will be in the cloud billing side, and be tailored for each cloud provider.

Yes exactly. We'll tailor this exactly to the kind of request that are billable on each provider and will count at least at a resolution that allows you to determine the exact cost. Or put differently, the resolution across various request types will be at least as good (better regarding some details of multi-part requests than what you currently get from the estimator).

ll we also provide metadata that indicates the context in which an API request was made ? for example, as part of r "snapshot restore" vs. "searchable snapshot" ?

This was not part of the initial plan. There are definitely challenges around implementing knowledge of the context. We can discuss this on another channel maybe, but if you see a use-case for having this the cost of building it wouldn't be prohibitive. The initial plan here was to simply meter the physical request regardless of context though.

[yoavderazon]

Thanks Armin.

The reason im asking about the scope is to see if we have any flexibility to apply a different price for an API call if its in the context of searchable snapshot, vs. snapshot-restore.
I'm NOT saying we decided to do so, but just checking if we would have the option.

As a general design principle, i think it could be helpful to log the context of a call, now that we have more than one context beyond restore.

If customers will use searchable snapshot extensively for reads - the API snapshot billing will cease being an "ancillary" billing dimension and will become a more signficant portion of the bill. in such case, it would be helpful from an analytics perspective as well as billing details, to separate the records used for searching from the ones used for restore.

Is providing such context a significant scope increase ?

[original-brownbear]

Is providing such context a significant scope increase ?

It depends on what we need/want here. If we simply want to meter all search-related operations separately that may be doable relatively quickly. But detailed context on whether other requests were part of a restore/snapshot/status-api-call, that would be quite complex.
Would separately identifying search-related work be enough ?

[jasontedor]

Let's keep the scope focused on what we actually need. I've heard in other conversations that we do not plan to bill differently on the context of the API calls, so let's keep the scope focused on adding metering, the right way to expose it in Elasticsearch, etc. If in the future we have an actual concrete need for providing the context, then we can consider that when the need arises.

[yoavderazon]

@jasontedor - this is not just for billing purposes. we will also benefit from calling out # of API calls for searchable snapshots vs. total # of API calls for snapshot backup/restore for analytics purposes. in fact - the number of API calls for searchable snapshots may be a leading KPI for the adoption of the searchable snapshot feature.

I agree that we want to control a scope creep, but it also sounds that @original-brownbear proposal of {"calls made from searchable snapshots"; "all other calls"} may be a reasonable compromise that will still give us some additional level of detail. That is if i understood the proposal correctly.

@zuketo , @zanbel - do you have an issue for how we will measure the feature's adoption and what are the KPIs ?
cc: @linyaru

[fcofdez]

I was wondering what would be the expected output for Expose request counts via configurable (periodic) logging. Should we log it into a specific file? should we log it into an index? For now I'll implement the exporting as a pluggable class so we can decide later.

[original-brownbear]

@fcofdez if we just use a standard named log4j logger, then we can configure where the output goes via log4j and consumers of this can decide what to do with it. No need to worry about this on our end.
That said, the discussion on this is incoming elsewhere shortly as discussed on Tue :) sorry for the delay in getting this going.