Jetson AGX/TX2/Nano - Build release from pip3 install

[alexis-gruet-deel]

Describe the feature and the current behavior/state.
Doing a pip3 install tensorflow-addons from an Nvidia TX2 produce No matching distribution found for tensorflow-addons. My current TensorFlow is 2.1.0 from Jetpack 4.3.

[bhack]

The main issue Is that we need tensorflow/build#9

[alexis-gruet-deel]

I tried to build from src the r0.8.3 ; However, I'm not able to build w/ bazel. Pb from the build to find cuda libs ; then pb w/ crosstool:toolchain' does not contain a toolchain for cpu 'aarch64 My question ; How you guys compile tfa on tx2 with jetpack 4.3 (TF-2.1 / Cuda 10.0 / Cudnn 7) ?

[bhack]

Currently we don't release/build tfa on arm64.
As I told you we need to have Tensorflow build infra on that arch available. Check again tensorflow/build#9.

[alexis-gruet-deel]

Thanks @bhack :
I read twice your message and tensorflow/build#9 with limited knowledge on this field thus ; limited understanding on my side. I'm just surprised to see that I can compile for the CPU ; but not able to make it work for the GPU. I understand you are not providing release nor build ;

Can you confirm w/o tensorflow/build#9 there is no way to build tfa from sources on the tx2 for the GPU ? (while apparently make it work from src for the CPU)

[bhack]

What I mean is that as in tfa we have custom-ops we need custom-ops infra for arm64.
Nvidia packages like the one you are using from https://developer.download.nvidia.com/compute/redist/jp/v44/tensorflow/ are supported by Nvidia and not officially supported by Tensorflow/SIGs.

So if you want to go ahead with Nvidia packages I suggest you to post in https://forums.developer.nvidia.com/c/agx-autonomous-machines/jetson-embedded-systems/70 cause they know their packages receipts and probably they could add one prepared package also for tfa.
The official (Tensorflow) way to support ARM64 is solve and upvote tensorflow/build#9.

[alexis-gruet-deel]

Ok make total sense.
I finally make it works on the GPU by editing the external/local_config_cuda/crosstool/BUILD file and defining "aarch64": ":cc-compiler-local" under cc_toolchain_suite > toolchains { I've tested and everything is ok.

[bhack]

@MI-LA01 If you have solved on your local setup you can talk with @bzhaoopenstack to see if you could add a zuul job for tfa at https://github.com/theopenlab/openlab-zuul-jobs other then Tensorflow.

[alexis-gruet-deel]

Sure, pls note my version of Jetpack was 4.3 so I was only able to compile the tag <= v0.9.1

[Tetsujinfr]

Ok make total sense.
I finally make it works on the GPU by editing the external/local_config_cuda/crosstool/BUILD file and defining "aarch64": ":cc-compiler-local" under cc_toolchain_suite > toolchains { I've tested and everything is ok.

hi
how did you get tfa to work on Jetson pls?
I got the tf2.2 distrib from NVidia using the Jetson GPU and I would like to avoid building it from source. Did you manage to install tfa on top of the NVidia tf distribution?

I have built Bazel from src successfully, and now I am truing to build tfa from source with GPU support.

Here is my external/local_config_cuda/crosstool/BUILD file, I do not see what I did not do correctly, please help me:

licenses(["restricted"])

package(default_visibility = ["//visibility:public"])

load(":cc_toolchain_config.bzl", "cc_toolchain_config")

toolchain(
name = "toolchain-linux-aarch64",
exec_compatible_with = [
"@bazel_tools//platforms:linux",
"@bazel_tools//platforms:aarch64",
],
target_compatible_with = [
"@bazel_tools//platforms:linux",
"@bazel_tools//platforms:aarch64",
],
toolchain = ":cc-compiler-local",
toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

cc_toolchain_suite(
name = "toolchain",
toolchains = {
"local|compiler": ":cc-compiler-local",
"k8": ":cc-compiler-local",
"ppc": ":cc-compiler-local",
},
)

cc_toolchain(
name = "cc-compiler-local",
all_files = ":crosstool_wrapper_driver_is_not_gcc",
compiler_files = ":empty",
dwp_files = ":empty",
linker_files = ":crosstool_wrapper_driver_is_not_gcc",
objcopy_files = ":empty",
strip_files = ":empty",
# To support linker flags that need to go to the start of command line
# we need the toolchain to support parameter files. Parameter files are
# last on the command line and contain all shared libraries to link, so all
# regular options will be left of them.
supports_param_files = 1,
toolchain_config = ":cc-compiler-local-config",
toolchain_identifier = "local_linux",
)

cc_toolchain_config(
name = "cc-compiler-local-config",
cpu = "local",
builtin_include_directories = "/usr/include/c++/7,/usr/include/aarch64-linux-gnu/c++/7,/usr/include/c++/7/backward,/usr/lib/gcc/aarch64-linux-gnu/7/include,/usr/local/include,/usr/lib/gcc/aarch64-linux-gnu/7/include-fixed,/usr/include/aarch64-linux-gnu,/usr/include,/usr/local/cuda/targets/aarch64-linux/include,/usr/local/cuda/include,/usr/local/cuda/include,/usr/include".split(","),
extra_no_canonical_prefixes_flags = ["-fno-canonical-system-headers"],
host_compiler_path = "clang/bin/crosstool_wrapper_driver_is_not_gcc",
host_compiler_prefix = "/usr/bin",
host_compiler_warnings = [],
host_unfiltered_compile_flags = [],
linker_bin_path = "/usr/bin",
)

filegroup(
name = "empty",
srcs = [],
)

filegroup(
name = "crosstool_wrapper_driver_is_not_gcc",
srcs = ["clang/bin/crosstool_wrapper_driver_is_not_gcc"],
)

Error msg I still have:

external/local_config_cuda/crosstool/BUILD:22:1: in cc_toolchain_suite rule @local_config_cuda//crosstool:toolchain: cc_toolchain_suite '@local_config_cuda//crosstool:toolchain' does not contain a toolchain for cpu 'aarch64'

thanks for your help

[Tetsujinfr]

Ok so I misread you initial post and did not edit the BUILD file at the right place.
So for sake of ref, I have edited the BUILD.tpl file under addons/build_deps/toolchains/gpu/crosstool/ at the right place and the build did work fine.

Modified part of BUILD.tpl:

cc_toolchain_suite(
name = "toolchain",
toolchains = {
"local|compiler": ":cc-compiler-local",
"k8": ":cc-compiler-local",
"ppc": ":cc-compiler-local",
"aarch64": ":cc-compiler-local",
},
)

Thanks or your solution on this, brilliant!