Maven plugin build-image creator step fails due to “Invalid content length provided” in Bitbucket pipelines

[michalrames]

Running mvn spring-boot:build-image in bitbucket pipeline (maven:3.6.3-openjdk-11 image with default docker service) fails in the "Running creator" step.

From the docker log it appears that "authorization (was) denied by plugin pipelines: Invalid content length provided". This could be a red herring as bitbucket documentation indicates the docker commands are restricted https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/

Unfortunately, there isn't enough debug logging to troubleshoot the createContainer method further.

Can you please indicate whether this is a known limitation or if there is some configuration that can be included to work around the bitbucket restrictions?

Bitbucket docker log:

time="2020-08-09T11:28:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/_ping
time="2020-08-09T11:28:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri=/v1.39/version
time="2020-08-09T11:29:47Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=gcr.io%2Fpaketo-buildpacks%2Fbuilder%3Alatest"
time="2020-08-09T11:30:11Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/gcr.io/paketo-buildpacks/builder@sha256:c4feeff9bb38c56cc615129c65162e5e8f6bfd3cd7ffae6e2e673c57ac6b9d5b/json"
time="2020-08-09T11:30:11Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=docker.io%2Fpaketobuildpacks%2Frun%3Afull-cnb-cf"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/docker.io/paketobuildpacks/run@sha256:66128f7a3675b8fd8cd454ed2b266910a04f6473983801bb0250e1acd80f3560/json"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.24/images/load
**time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/v1.24/containers/create
time="2020-08-09T11:30:12.607341026Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: Invalid content length provided"**
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-layers-qktednfhbn?force=1"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-app-olvjnashcf?force=1"
time="2020-08-09T11:30:12Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/images/pack.local/builder/ssmtjetwcz:latest?force=1"

Maven stack trace:

[INFO] --- spring-boot-maven-plugin:2.3.2.RELEASE:build-image (default-cli) @ xyz-services ---
[INFO] Building image 'docker.io/library/xyz-services:2.1.0-SNAPSHOT'
[INFO] 
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 3%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 10%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 10%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 11%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 12%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 14%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 33%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 48%
[INFO]  > Pulling builder image 'gcr.io/paketo-buildpacks/builder:latest' 100%
[INFO]  > Pulled builder image 'gcr.io/paketo-buildpacks/builder@sha256:c4feeff9bb38c56cc615129c65162e5e8f6bfd3cd7ffae6e2e673c57ac6b9d5b'
[INFO]  > Pulling run image 'docker.io/paketobuildpacks/run:full-cnb-cf' 100%
[INFO]  > Pulled run image 'paketobuildpacks/run@sha256:66128f7a3675b8fd8cd454ed2b266910a04f6473983801bb0250e1acd80f3560'
[INFO]  > Executing lifecycle version v0.8.1
[INFO]  > Using build cache volume 'pack-cache-900830d40c15.build'
[INFO] 
[INFO]  > Running creator
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  52.776 s
[INFO] Finished at: 2020-08-09T11:30:12Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image (default-cli) on project xyz-services: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden" -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image (default-cli) on project xyz-services: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.3.2.RELEASE:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.springframework.boot.buildpack.platform.docker.transport.DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
    at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute (HttpClientTransport.java:136)
    at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute (HttpClientTransport.java:123)
    at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.post (HttpClientTransport.java:94)
    at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.createContainer (DockerApi.java:234)
    at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.create (DockerApi.java:225)
    at org.springframework.boot.buildpack.platform.build.Lifecycle.createContainer (Lifecycle.java:166)
    at org.springframework.boot.buildpack.platform.build.Lifecycle.run (Lifecycle.java:146)
    at org.springframework.boot.buildpack.platform.build.Lifecycle.execute (Lifecycle.java:113)
    at org.springframework.boot.buildpack.platform.build.Builder.executeLifecycle (Builder.java:122)
    at org.springframework.boot.buildpack.platform.build.Builder.build (Builder.java:71)
    at org.springframework.boot.maven.BuildImageMojo.buildImage (BuildImageMojo.java:144)
    at org.springframework.boot.maven.BuildImageMojo.execute (BuildImageMojo.java:136)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

[wilkinsona]

Thanks for the report.

The problem appears to be that BitBucket considers the POST request that is creating a container to have an invalid Content-Length:

time="2020-08-09T11:30:12.607341026Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: Invalid content length provided"

As far as I can tell, BitBucket's problem is that the POST request is sent using Transfer-Encoding: chunked as the content length isn't known. There is, therefore, no Content-Length header in the request. This is what I assume causes BitBucket to consider the Content-Length to be invalid but we can't be certain as, AFAIK, BitBucket Pipelines isn't open source.

In this specific case, we could supply a Content-Length with the container creation POST request. However, it would then fail for the subsequent PUT that uploads the container's content.

I think this will have to be raised with Atlassian. Assuming my analysis above is correct, ideally they would support chunked transfer encoding as Docker usually does. Failing that, if my analysis is incorrect or they're unable to support chunked transfer encoding and they can provide detailed information about the Docker-related restrictions, we can re-open this and take another look.

[goatfryed]

Thanks for providing feedback about this issue.

For anyone else coming here, I'd like to add some more info.
There is a related public bug issue that can be found here: https://jira.atlassian.com/browse/BCLOUD-15844

For now, I added jib gradle plugin and replaced bootBuildImage in my pipeline with jibDockerBuild which can create the desired image. Note that this image is roughly 20% smaller compared to the image generated by bootBuildImage and I haven't figured out yet why, but the tests are passing, so it seems to be an alternative..

[goatfryed]

Today the bitbucket team made an update explaining why they can't support Transfer-Encoding Chunked see here

In short, this is not an issue with bitbucket in particular, but with how dockers authorization plugins work and they can't do much about it currently, although they would appreciate an update by docker, since other projects are experience similar problems.
Other environments with dockers authZ plugin configured are affected as well.

In this specific case, we could supply a Content-Length with the container creation POST request. However, it would then fail for the subsequent PUT that uploads the container's content.

The plugin only applies to content-type application/json. The subsequent PUT should be content-type application/x-tar, so the plugin wouldn't intercept and reject a chunked transfer there.

Could you reopen this issue and update the POST request, since a change to dockers authZ plugin seems unlikely currently?

From the linked duplicate

It's not clear to me if this is an RFC-style should where it's something that's recommended but optional or if it's a requirement a must in RFC terms). It would appear that the local Docker daemon treats it as optional (but perhaps recommended) whereas BitBucket's pipelines treat it as a requirement. Either way, it's probably worth us complying with the API's recommendations if we can.

I guess the answer is, it is a requirement that applies as soon as docker plugin infrastructure is involved 😄