Strange compilation problem in Docker container

[zou000]

I am trying to build PR #199, following the instructions in README.md, but met a strange problem. Here is a simplified repro:

docker run -it --rm -v $(pwd):/v -w /v tfio-dev
./configure.sh
bazel build -s --verbose_failures @libaprutil1//...

The compilation failed with the following error message:

Use --sandbox_debug to see verbose messages from the sandbox: gcc failed: error executing command
  (cd /root/.cache/bazel/_bazel_root/a7ecd8237744645c5d189c197108d6d2/execroot/org_tensorflow_io && \
  exec env - \
    PATH=/miniconda/envs/tfio-dev/bin:/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
    TF_HEADER_DIR=/miniconda/envs/tfio-dev/lib/python3.6/site-packages/tensorflow/include \
    TF_SHARED_LIBRARY_DIR=/miniconda/envs/tfio-dev/lib/python3.6/site-packages/tensorflow \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF bazel-out/k8-fastbuild/bin/external/libaprutil1/_objs/libaprutil1/apr_dbm_sdbm.pic.d -fPIC -D_GNU_SOURCE -DHAVE_CONFIG_H -DLINUX -DREENTRANT -iquote external/libaprutil1 -iquote bazel-out/k8-fastbuild/genfiles/external/libaprutil1 -iquote bazel-out/k8-fastbuild/bin/external/libaprutil1 -iquote external/libapr1 -iquote bazel-out/k8-fastbuild/genfiles/external/libapr1 -iquote bazel-out/k8-fastbuild/bin/external/libapr1 -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/genfiles/external/bazel_tools -iquote bazel-out/k8-fastbuild/bin/external/bazel_tools -Ibazel-out/k8-fastbuild/bin/external/libaprutil1/_virtual_includes/libaprutil1 -Ibazel-out/k8-fastbuild/bin/external/libapr1/_virtual_includes/libapr1 -isystem external/libaprutil1/include -isystem bazel-out/k8-fastbuild/genfiles/external/libaprutil1/include -isystem bazel-out/k8-fastbuild/bin/external/libaprutil1/include -isystem external/libaprutil1/include/private -isystem bazel-out/k8-fastbuild/genfiles/external/libaprutil1/include/private -isystem bazel-out/k8-fastbuild/bin/external/libaprutil1/include/private -isystem external/libapr1/include -isystem bazel-out/k8-fastbuild/genfiles/external/libapr1/include -isystem bazel-out/k8-fastbuild/bin/external/libapr1/include -isystem external/libapr1/include/arch/unix -isystem bazel-out/k8-fastbuild/genfiles/external/libapr1/include/arch/unix -isystem bazel-out/k8-fastbuild/bin/external/libapr1/include/arch/unix -isystem external/libapr1/include/private -isystem bazel-out/k8-fastbuild/genfiles/external/libapr1/include/private -isystem bazel-out/k8-fastbuild/bin/external/libapr1/include/private '-D_GLIBCXX_USE_CXX11_ABI=0' -pthread -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/libaprutil1/dbm/apr_dbm_sdbm.c -o bazel-out/k8-fastbuild/bin/external/libaprutil1/_objs/libaprutil1/apr_dbm_sdbm.pic.o)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox
external/libaprutil1/dbm/apr_dbm_sdbm.c:22:10: fatal error: apu_config.h: No such file or directory
 #include "apu_config.h"
          ^~~~~~~~~~~~~~
compilation terminated.

However, if I manually run the gcc command given by the message, it can find apu_config.h and can compile successfully.

I was using custom-op image with up-to-date bazel before, and this still works:

docker run -it --rm -v $(pwd):/v -w /v tensorflow/tensorflow:custom-op
# update bazel to required version
curl -sL https://github.com/bazelbuild/bazel/releases/download/0.20.0/bazel-0.20.0-installer-linux-x86_64.sh -o bazel-install.sh &&     bash -x bazel-install.sh && rm bazel-install.sh
./configure.sh
bazel build -s --verbose_failures @libaprutil1//...

I suspect this could be related the different base-images used. It is ubuntu 18.04 for tfio-dev and ubuntu 14.04 for custom-op.

The Docker host I am using is Docker Desktop Community v 2.0.0.3 on a macbook Pro.

[zou000]

@wangkuiyi @terrytangyuan

[yongtang]

Is apu_config.h included in the bazel file? Depending on the bazel version, it may explicitly requires that a header file declared in bazel BUILD file.

[zou000]

I was using the same bazel version (0.20.0) in both docker images. Actually, I had to update the one in custom-op image, as the bazel came with it cannot parse some of the new BUILD files.

…

On Wed, Apr 17, 2019, 16:53 Yong Tang ***@***.***> wrote: Is apu_config.h included in the bazel file? Depending on the bazel version, it may explicitly requires that a header file declared in bazel BUILD file. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#200 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADV3AM4Z4D5HMINK37JGHLPQ6Z6PANCNFSM4HGXDAKA> .

[zou000]

I found the problem by comparing custom-op's Dockerfile with ours.

there are some pointers in the .bazelrc file copied to the image.

After PR #203 our docker image is based on custom-op so the problem is gone. BUT, the CI doesn't have this .bazlerc file, so the CI is failing for the same reason. See for example: https://travis-ci.org/tensorflow/io/jobs/526285448#L475

[zou000]

@yongtang suggested another way to circumvent the bazel-in-docker bug.

To summarize, there are 2 ways to fix this:

• Add following to .bazelrc or /etc/bazelrc

# Running bazel inside a `docker build` command causes trouble, cf:
#   https://github.com/bazelbuild/bazel/issues/134
# The easiest solution is to set up a bazelrc file forcing --batch.
startup --batch

# Similarly, we need to workaround sandboxing issues:
#   https://github.com/bazelbuild/bazel/issues/418
build  --verbose_failures --spawn_strategy=standalone --strategy=Genrule=standalone
test --spawn_strategy=standalone

• explicitly add the files generated by bazel to build src file list, etc. like this.