Modules fail with compiled tensorflow

[baluyotraf]

I compiled tensorflow from source. When using the arrow module of the library I get the error below.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ds/miniconda3/envs/test/lib/python3.6/site-packages/tensorflow_io/arrow/__init__.py", line 26, in <module>
    from tensorflow_io.arrow.python.ops.arrow_dataset_ops import ArrowDataset
  File "/home/ds/miniconda3/envs/test/lib/python3.6/site-packages/tensorflow_io/arrow/python/ops/arrow_dataset_ops.py", line 27, in <module>
    arrow_ops = _load_library('_arrow_ops.so')
  File "/home/ds/miniconda3/envs/test/lib/python3.6/site-packages/tensorflow_io/__init__.py", line 56, in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
NotImplementedError: unable to open file: _arrow_ops.so, from paths: ['/home/ds/miniconda3/envs/test/lib/python3.6/site-packages/tensorflow_io/arrow/python/ops/_arrow_ops.so']
caused by: ['/home/ds/miniconda3/envs/test/lib/python3.6/site-packages/tensorflow_io/arrow/python/ops/_arrow_ops.so: undefined symbol: _ZN10tensorflow6Status12empty_stringEv']

[yongtang]

@baluyotraf The release version of tensorflow-io is compiled with tensorflow 1.13.1. It may not be compatible with earlier or later versions of tensorflow, as tesorflow's C++ API experienced some changes recently (approaching TF2.0 release).

You may need to recompile tensorflow-io as well in order to make it work with your compiled tensorflow.

Can you shred's which version of tensorflow source you are building?

/cc @BryanCutler as well for arrow.

[baluyotraf]

I compiled the r1.13 branch of the tensorflow repository with gpu support. The created install has a 1.13.1 on its name so I think it's also the correct version. I tried to build the tensorflow_io code with my tensorflow but I got the same error. After running the tests, I get the same error from different .so files so I guess this is a build problem.

Pytest Result:

___________________________________________________________________________________________________ ERROR collecting tests/test_arrow.py ____________________________________________________________________________________________________
tests/test_arrow.py:37: in <module>
    import tensorflow_io.arrow as arrow_io # pylint: disable=wrong-import-position
tensorflow_io/arrow/__init__.py:26: in <module>
    from tensorflow_io.arrow.python.ops.arrow_dataset_ops import ArrowDataset
tensorflow_io/arrow/python/ops/arrow_dataset_ops.py:27: in <module>
    arrow_ops = _load_library('_arrow_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _arrow_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/arrow/python/ops/_arrow_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/arrow/python/ops/_arrow_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/arrow/python/ops/_arrow_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/arrow/python/ops/_arrow_ops.so: undefined symbol: _ZN10tensorflow6Status12empty_stringEv']
___________________________________________________________________________________________________ ERROR collecting tests/test_hadoop.py ___________________________________________________________________________________________________
tests/test_hadoop.py:30: in <module>
    import tensorflow_io.hadoop as hadoop_io         # pylint: disable=wrong-import-position
tensorflow_io/hadoop/__init__.py:24: in <module>
    from tensorflow_io.hadoop.python.ops.hadoop_dataset_ops import SequenceFileDataset
tensorflow_io/hadoop/python/ops/hadoop_dataset_ops.py:24: in <module>
    hadoop_ops = _load_library('_hadoop_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _hadoop_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/hadoop/python/ops/_hadoop_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/hadoop/python/ops/_hadoop_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/hadoop/python/ops/_hadoop_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/hadoop/python/ops/_hadoop_ops.so: undefined symbol: _ZN10tensorflow7strings6StrCatERKNS0_8AlphaNumES3_']
___________________________________________________________________________________________________ ERROR collecting tests/test_ignite.py ___________________________________________________________________________________________________
tests/test_ignite.py:31: in <module>
    import tensorflow_io.ignite as ignite_io # pylint: disable=wrong-import-position
tensorflow_io/ignite/__init__.py:35: in <module>
    from tensorflow_io.ignite.python.ops.ignite_dataset_ops import IgniteDataset
tensorflow_io/ignite/python/ops/ignite_dataset_ops.py:32: in <module>
    ignite_ops = _load_library("_ignite_ops.so")
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _ignite_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/ignite/python/ops/_ignite_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/ignite/python/ops/_ignite_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/ignite/python/ops/_ignite_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/ignite/python/ops/_ignite_ops.so: undefined symbol: _ZN10tensorflow10FileSystem11IsDirectoryERKSs']
___________________________________________________________________________________________________ ERROR collecting tests/test_image.py ____________________________________________________________________________________________________
tests/test_image.py:30: in <module>
    import tensorflow_io.image as image_io # pylint: disable=wrong-import-position
tensorflow_io/image/__init__.py:27: in <module>
    from tensorflow_io.image.python.ops.image_dataset_ops import WebPDataset
tensorflow_io/image/python/ops/image_dataset_ops.py:24: in <module>
    image_ops = _load_library('_image_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _image_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/image/python/ops/_image_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/image/python/ops/_image_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/image/python/ops/_image_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/image/python/ops/_image_ops.so: undefined symbol: _ZN10tensorflow7strings6StrCatERKNS0_8AlphaNumES3_']
___________________________________________________________________________________________________ ERROR collecting tests/test_kafka.py ____________________________________________________________________________________________________
tests/test_kafka.py:33: in <module>
    import tensorflow_io.kafka as kafka_io # pylint: disable=wrong-import-position
tensorflow_io/kafka/__init__.py:25: in <module>
    from tensorflow_io.kafka.python.ops.kafka_dataset_ops import KafkaDataset
tensorflow_io/kafka/python/ops/kafka_dataset_ops.py:24: in <module>
    kafka_ops = _load_library('_kafka_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _kafka_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/kafka/python/ops/_kafka_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/kafka/python/ops/_kafka_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/kafka/python/ops/_kafka_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/kafka/python/ops/_kafka_ops.so: undefined symbol: _ZN10tensorflow6Status12empty_stringEv']
__________________________________________________________________________________________________ ERROR collecting tests/test_kinesis.py ___________________________________________________________________________________________________
tests/test_kinesis.py:43: in <module>
    import tensorflow_io.kinesis as kinesis_io # pylint: disable=wrong-import-position
tensorflow_io/kinesis/__init__.py:24: in <module>
    from tensorflow_io.kinesis.python.ops.kinesis_dataset_ops import KinesisDataset
tensorflow_io/kinesis/python/ops/kinesis_dataset_ops.py:25: in <module>
    kinesis_ops = _load_library('_kinesis_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _kinesis_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/kinesis/python/ops/_kinesis_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/kinesis/python/ops/_kinesis_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/kinesis/python/ops/_kinesis_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/kinesis/python/ops/_kinesis_ops.so: undefined symbol: _ZN10tensorflow8str_util9LowercaseEN4absl11string_viewE']
___________________________________________________________________________________________________ ERROR collecting tests/test_libsvm.py ___________________________________________________________________________________________________
tests/test_libsvm.py:27: in <module>
    import tensorflow_io.libsvm as libsvm_io
tensorflow_io/libsvm/__init__.py:25: in <module>
    from tensorflow_io.libsvm.python.ops.libsvm_dataset_ops import make_libsvm_dataset
tensorflow_io/libsvm/python/ops/libsvm_dataset_ops.py:8: in <module>
    gen_libsvm_ops = _load_library('_libsvm_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _libsvm_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/libsvm/python/ops/_libsvm_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/libsvm/python/ops/_libsvm_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/libsvm/python/ops/_libsvm_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/libsvm/python/ops/_libsvm_ops.so: undefined symbol: _ZN10tensorflow7strings8internal9CatPiecesESt16initializer_listIN4absl11string_viewEE']
____________________________________________________________________________________________________ ERROR collecting tests/test_lmdb.py ____________________________________________________________________________________________________
tests/test_lmdb.py:28: in <module>
    import tensorflow_io.lmdb as lmdb_io # pylint: disable=wrong-import-position
tensorflow_io/lmdb/__init__.py:24: in <module>
    from tensorflow_io.lmdb.python.ops.lmdb_dataset_ops import LMDBDataset
tensorflow_io/lmdb/python/ops/lmdb_dataset_ops.py:24: in <module>
    lmdb_ops = _load_library('_lmdb_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _lmdb_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/lmdb/python/ops/_lmdb_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/lmdb/python/ops/_lmdb_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/lmdb/python/ops/_lmdb_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/lmdb/python/ops/_lmdb_ops.so: undefined symbol: _ZN10tensorflow7strings6StrCatERKNS0_8AlphaNumES3_']
___________________________________________________________________________________________________ ERROR collecting tests/test_mnist.py ____________________________________________________________________________________________________
tests/test_mnist.py:31: in <module>
    from tensorflow_io.mnist.python.ops import mnist_dataset_ops # pylint: disable=wrong-import-position
tensorflow_io/mnist/__init__.py:26: in <module>
    from tensorflow_io.mnist.python.ops.mnist_dataset_ops import MNISTDataset
tensorflow_io/mnist/python/ops/mnist_dataset_ops.py:24: in <module>
    mnist_ops = _load_library('_mnist_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _mnist_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/mnist/python/ops/_mnist_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/mnist/python/ops/_mnist_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/mnist/python/ops/_mnist_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/mnist/python/ops/_mnist_ops.so: undefined symbol: _ZN10tensorflow7strings6StrCatERKNS0_8AlphaNumES3_']
__________________________________________________________________________________________________ ERROR collecting tests/test_parquet.py ___________________________________________________________________________________________________
tests/test_parquet.py:31: in <module>
    import tensorflow_io.parquet as parquet_io # pylint: disable=wrong-import-position
tensorflow_io/parquet/__init__.py:24: in <module>
    from tensorflow_io.parquet.python.ops.parquet_dataset_ops import ParquetDataset
tensorflow_io/parquet/python/ops/parquet_dataset_ops.py:24: in <module>
    parquet_ops = _load_library('_parquet_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _parquet_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/parquet/python/ops/_parquet_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/parquet/python/ops/_parquet_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/parquet/python/ops/_parquet_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/parquet/python/ops/_parquet_ops.so: undefined symbol: _ZN10tensorflow6Status12empty_stringEv']
___________________________________________________________________________________________________ ERROR collecting tests/test_pubsub.py ___________________________________________________________________________________________________
ImportError while importing test module '/home/ds/tensorflow_io/tests/test_pubsub.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_pubsub.py:24: in <module>
    from google.cloud import pubsub_v1
E   ModuleNotFoundError: No module named 'google.cloud'
____________________________________________________________________________________________________ ERROR collecting tests/test_text.py ____________________________________________________________________________________________________
tests/test_text.py:27: in <module>
    import tensorflow_io.text as text_io
tensorflow_io/text/__init__.py:24: in <module>
    from tensorflow_io.text.python.ops.text_ops import TextOutputSequence
tensorflow_io/text/python/ops/text_ops.py:21: in <module>
    text_ops = _load_library('_text_ops.so')
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _text_ops.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/text/python/ops/_text_ops.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/text/python/ops/_text_ops.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/text/python/ops/_text_ops.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/text/python/ops/_text_ops.so: undefined symbol: _ZN10tensorflow7strings6StrCatERKNS0_8AlphaNumE']
___________________________________________________________________________________________________ ERROR collecting tests/test_video.py ____________________________________________________________________________________________________
tests/test_video.py:35: in <module>
    import tensorflow_io.video as video_io  # pylint: disable=wrong-import-position
tensorflow_io/video/__init__.py:24: in <module>
    from tensorflow_io.video.python.ops.video_dataset_ops import VideoDataset
tensorflow_io/video/python/ops/video_dataset_ops.py:67: in <module>
    "libswscale.so.2",
tensorflow_io/video/python/ops/video_dataset_ops.py:43: in load_dependency_and_library
    return _load_library(library)
tensorflow_io/__init__.py:59: in _load_library
    "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))
E   NotImplementedError: unable to open file: _video_ops_ffmpeg_3.4.so, from paths: ['/home/ds/tensorflow_io/tensorflow_io/video/python/ops/_video_ops_ffmpeg_3.4.so', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/video/python/ops/_video_ops_ffmpeg_3.4.so']
E   caused by: ['/home/ds/tensorflow_io/tensorflow_io/video/python/ops/_video_ops_ffmpeg_3.4.so: cannot open shared object file: No such file or directory', '/home/ds/tensorflow_io/bazel-bin/tensorflow_io/video/python/ops/_video_ops_ffmpeg_3.4.so: undefined symbol: _ZN10tensorflow7strings6StrCatERKNS0_8AlphaNumE']
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 13 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================================================= 13 error in 1.36 seconds ==========================================================================================================

[yongtang]

Haven't check to deep into the issue, though a common issue that surface frequently is the -D_GLIBCXX_USE_CXX11_ABI=0 flag. Are you using this flag for tensorflow build?

[baluyotraf]

Not yet. I'll try to rebuild with that flag. Thanks.

[baluyotraf]

That did the trick. Thanks.

Just in case someone will read this thread in the future, I also rebuilt the tensorflow_io package. The tensorflow_io should be compiled with the compiled tensorflow. Not doing so resulted in seg faults.

[aayushARM]

I get a similar error about _image_ops.so:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-47-55aea3a665c7> in <module>
----> 1 from tensorflow_io.image.python.ops.image_dataset_ops import TIFFDataset
      2 import glob2

~/anaconda3/lib/python3.7/site-packages/tensorflow_io/image/python/ops/image_dataset_ops.py in <module>
     22 from tensorflow.compat.v1 import data
     23 from tensorflow_io import _load_library
---> 24 image_ops = _load_library('_image_ops.so')
     25 
     26 

~/anaconda3/lib/python3.7/site-packages/tensorflow_io/__init__.py in _load_library(filename, lib)
     57   raise NotImplementedError(
     58       "unable to open file: " +
---> 59       "{}, from paths: {}\ncaused by: {}".format(filename, filenames, errs))

NotImplementedError: unable to open file: _image_ops.so, from paths: ['/home/aayush/anaconda3/lib/python3.7/site-packages/tensorflow_io/image/python/ops/_image_ops.so']
caused by: ['/home/aayush/anaconda3/lib/python3.7/site-packages/tensorflow_io/image/python/ops/_image_ops.so: undefined symbol: _ZN10tensorflow7strings6StrCatERKNS0_8AlphaNumES3_']

This is in spite of the fact that my TF version is 1.13.1, and I am using the available pip whl. The .so file does exist in the directory. I am using python3.7 though, I changed cp36 in whl name to cp37, can this be the reason? @yongtang

[icyblade]

Same issue occurred for conda python==3.6.8.

[yongtang]

@aayushARM Python 3.7 is not supported by tensorflow-io yet. As tensorflow-io use Ubuntu 14.04 for python 2.7, 3.4-3.6 build which is the same as tensorflow. However, tensorflow python 3.7 seems to be built with Ubuntu 16.04 so it may not be binary compatible. See some background information in #173 (comment)

[zh794390558]

same problem when build custom ops.

undefined symbol: _ZN10tensorflow8str_util9LowercaseEN4absl11string_viewE

[zh794390558]

#203