Overly expensive resource utilization via object-storage (and related) clients

[snazy]

Describe the bug

Instances of object-storage (e.g. S3Client) and credential-vending clients (e.g. StsClient) are "fully" created for every client request. This means, that also implicit resources like HTTP connection pools and thread pools are created for every client request. This is too expensive.

Related to #563

[flyrain]

Thanks for reporting this. It'd be a nice improvement. I will mark it as a 1.0.0 milestone, while removing it as a 1.0 blocker.

[snazy]

@flyrain re-creating HTTP connections and thread pools is a completely unnecessary waste of system resources, without guarantees that these system resources are properly released, risking system stability. So the labels you've removed are IMHO justified, so I'm re-adding those.

[nssalian]

@snazy, can you help find instances where you see this? I'd like to put in a fix.

[flyrain]

Hi @nssalian, here is the place creating client supplier,

polaris/service/common/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java

Line 62 in 1daf749

default Supplier<StsClient> stsClientSupplier() {

. We could make it a singleton.