-
Notifications
You must be signed in to change notification settings - Fork 85
Description
Is your feature request related to a problem? Please describe.
I have a use case that necessitates using the Storage/Bucket write method variants with InputStream arguments. The inputs for my use case to store in Cloud Storage are:
- predominantly already compressed,
- have a wide range of file sizes, from a few KB to GB+,
- we don't always know the size in advance.
Describe the solution you'd like
Given that our content is already compressed, I would prefer to avoid spending the CPU time on compressing the content again en route to the Bucket.
Describe alternatives you've considered
We have used the byte[] variants, with BlobTargetOption.disableGzipContent and the Compose request. This is suitable but leaves us with a tuning challenge:
- Buffering these streams into memory (
byte[]chunks) requires additional heap space be available - Compose is limited to only 32 chunks. If the file content we intend to store is far larger than
32 * bufferSize, we will upload 31 small sized chunks and 1 large chunk that we have to use theInputStreamvariant for anyways, and pay the additional overhead of gzip compression.
Additional context
I have done some exploration and it appears that the values on the BlobWriteOption (used on InputStream variants) and BlobTargetOption (used on byte[] variants) enums both are translated into StorageRpc.Option. I have a small contribution to offer in the form of a pull request to follow.