Storage and url encoding

[soujiro32167]

Environment details

1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
   General, Core, and Other are also allowed as types

• Storage

2. OS type and version:

• MacOS Catalina

3. Java version:

• 11.0.4
• Scala 2.13.1

4. google-cloud-java version(s):

• 1.103.1

Steps to reproduce

1. upload a file named 1+1 2 to my-bucket
2. list files in my-bucket
3. for each file, download

Code example

storage.list(bucket).iterateAll().asScala.foreach(blob => blob.downloadTo(blob.getName()))

Stack trace

com.google.cloud.RetryHelper$RetryHelperException: com.google.cloud.storage.StorageException: 404 Not Found
No such object: eli-test/1+1+2
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:233)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:217)
	at Main$.$anonfun$download$1(Main.scala:73)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at zio.blocking.Blocking$Service.$anonfun$effectBlocking$5(Blocking.scala:134)
	at zio.internal.FiberContext.evaluateNow(FiberContext.scala:386)
	at zio.internal.FiberContext.$anonfun$fork$2(FiberContext.scala:655)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.google.cloud.storage.StorageException: 404 Not Found
No such object: eli-test/1+1+2
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:675)
	at com.google.cloud.storage.Blob$2.run(Blob.java:238)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	... 10 more
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
No such object: eli-test/1+1+2
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
	at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:255)
	at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:185)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:685)
	at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:6996)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:671)
	... 15 more

External references such as API reference guides used

• ?

Any additional information below

If I try to encode the blob name using blob.toBuilder.setBlodId(...).build, I see double encoding: 1%2B1%202 => 1%252B1%25202
Also, using blob.signUrl produces a valid url that allows download

[elharo]

Can you reproduce this without scala?

[soujiro32167]

With a single file names 1 1

import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

import java.nio.file.Path;

public class Yava {
  public static void main(String[] args) {
    Storage storage = StorageOptions.getDefaultInstance().getService();
    storage.list("eli-test").iterateAll().forEach(blob -> blob.downloadTo(Path.of("target", blob.getName())));
  }
}

Exception in thread "main" com.google.cloud.RetryHelper$RetryHelperException: com.google.cloud.storage.StorageException: 404 Not Found
No such object: eli-test/1+1
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:233)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:217)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:260)
	at Yava.lambda$main$0(Yava.java:9)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at Yava.main(Yava.java:9)
Caused by: com.google.cloud.storage.StorageException: 404 Not Found
No such object: eli-test/1+1
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:675)
	at com.google.cloud.storage.Blob$2.run(Blob.java:238)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	... 6 more
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
No such object: eli-test/1+1
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
	at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:255)
	at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:185)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:685)
	at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:6996)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:671)
	... 11 more

[elharo]

Do you have a pom.xml or build.gradle for this?

[soujiro32167]

I have sbt: "com.google.cloud" % "google-cloud-storage" % "1.103.1"

[soujiro32167]

the only thing needed is the gcloud storage dependency

[athakor]

@soujiro32167 Thanks for pointing out this bug, but it's related to google-http-java-client.

FYI, A string of characters that do not need to be encoded when used in URI query strings, as
specified in RFC 3986. Note that some of these characters do need to be escaped when used in
other parts of the URI and space is one of them which needs to be escape.

you can check list of rules for an encoding a string which is parts of the URI.

Rules :
https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/util/escape/PercentEscaper.java#L32

Also look at the below links which replacse the space with {'+'}

https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/util/escape/PercentEscaper.java#L218

https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/util/escape/PercentEscaper.java#L93