bmc-objectstorage: Is there a way to upload large file, when inputstream doesn't support read in parallel.

[sudison]

Just want to know the reason behind the code:
In StreamChunkCreator.java, canDuplicate is set to true only when uploadmanager sets isAllowParallelUploads to true.
If canDuplicate is set to false, then DuplicatedSubRangeInputStream is not created in StreamChunkCreator->next function.
This creates a headache for me to upload a large file, my inputstream doesn't support read in parallel, thus I can't set allowparalleluploads in uploadmanager.
then likely, I'll get Out-of-memory exception during the upload process.

[priteshkp]

Hi Edison,
I'm a little unclear on the issue here. Are you asking how to support large files without causing OutOfMemory? By default, the client will not buffer your stream into memory, it will stream it directly to the server, so you should not hit OutOfMemory (regardless of whether or not parallel reads are used).

What type of InputStream are you supplying? If your file is large enough, it can still use multi-part uploads, it will just be serial uploads of small chunks, and each chunk can be retried if it fails (so you don't have to restart a large upload from start if it fails at any point).

[sudison]

If enforceMd5 is true, allowParallelUploads is false, then uploadmanager will read the inputstream into memory to calculate md5. If it's called by multiple threads, then multiple threads will try to read multiple inputstream into memory to calculate md5, thus causing out-of-memory easily.

Uploadmanager even has the log in the code:

  LOG.info("About to copy object into memory to calculate MD5, may lead to OutOfMemory exceptions");

[priteshkp]

Ah, ok, I got it. Even if you provide a Duplicable stream, it ends up buffering if you enable MD5. Yes, that's not intentional, its a bug. I saw the PR you just sent, I'll add feedback there. Thanks!

[jodoglevy]

This is now fixed in v1.2.13. Closing.