Trying to use autoaugment on TPU

[InfiniteLife]

Im trying to use augmix on TPU v3-8 with installed tensorflow 2.2, tensorflow_addons and tfa-nigthly.

I am recieving following error:

tensorflow.python.framework.errors_impl.NotFoundError: in user code:

    /usr/local/lib/python3.7/dist-packages/tensorflow_addons/image/translate_ops.py:104 translate  *
        translations_to_projective_transforms(translations),
    /usr/local/lib/python3.7/dist-packages/tensorflow_addons/image/transform_ops.py:111 transform  *
        output = _image_so.ops.addons_image_projective_transform_v2(
    /usr/local/lib/python3.7/dist-packages/tensorflow_addons/utils/resource_loader.py:64 ops
        self._ops = tf.load_op_library(get_path_to_datafile(self.relative_path))
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/load_library.py:58 load_op_library
        lib_handle = py_tf.TF_LoadLibrary(library_filename)

    NotFoundError: /usr/local/lib/python3.7/dist-packages/tensorflow_addons/custom_ops/image/_image_ops.so: undefined symbol: __cudaPushCallConfiguration

Is there a way to use augmix on TPU? Seems like it looks for cuda

[Ely-S]

did you pass --strategy=tpu to the training command?

[InfiniteLife]

did you pass --strategy=tpu to the training command?

Of course, training on TPU itself is working. It do not works with autoaugment.

[InfiniteLife]

I tried to built tensorflow-addons from source, it still gives same error

[InfiniteLife]

I tried to install them both with tf 2.1 and 2.2, gives errors. In case of 2.2 it now writes this:

  File "/mars/automl/efficientdet/dataloader.py", line 311, in _dataset_parser
    from aug import autoaugment  # pylint: disable=g-import-not-at-top
  File "/mars/automl/efficientdet/aug/autoaugment.py", line 39, in <module>
    from tensorflow_addons import image as image_ops  # pylint: disable=g-import-not-at-top
  File "/mars/automl/efficientdet/addons/tensorflow_addons/__init__.py", line 21, in <module>
    from tensorflow_addons import activations
  File "/mars/automl/efficientdet/addons/tensorflow_addons/activations/__init__.py", line 22, in <module>
    from tensorflow_addons.activations.rrelu import rrelu
  File "/mars/automl/efficientdet/addons/tensorflow_addons/activations/rrelu.py", line 28, in <module>
    rng: Optional[tf.random.Generator] = None,
AttributeError: module 'tensorflow_core._api.v2.random' has no attribute 'Generator'

Is there a way to install a tensorflow.contrib with 2.2 version? May be that will work

[mingxingtan]

IIRC, augmix may not work well for now. @fsx950223

[fsx950223]

Custom ops don't support tpu. Use tf1.15 instead.

[InfiniteLife]

Custom ops don't support tpu. Use tf1.15 instead.

Hello! But this repository works with 2.+ version, how can I use it with 1.15?

[Koteswara-ML]

Same issue i am also facing using V2-8 on efficientDet D2 using auto augmentation. Did that get resolved.

[fsx950223]

You could solve it by downgrade to tf1.15 or install tfa-nightly.

[PurnaKoteswaraRaoMallepaddi]

By doing that wouldn't it be causing any affect on training on TPU..?

[Koteswara-ML]

Yeah, I am actually getting this error now Paddings must be non-negative: -37 0