Convert file system implementations to use C file system API in tensorflow

[yongtang]

As C file system API in tensorflow is available (see tensorflow/community#101), we should start migrate the file system implementations in tensorflow-io to use the C file system API.

This issue track the progresses for the following:

• Azure Storage (PR Convert AZFS to use C modular file system API #1143)
• OSS Storage
• HTTP FS (PR Convert HTTP file system to use C modular file system API #1173)

[yongtang]

Before we can migrate to modular file system, additional work on tensorflow side is needed. The following is the progress of related PRs:

• PR Expose modular file system header in pip install package tensorflow#43209
• PR Update to include filesystem:filesystem_interface in libtensorflow_framework.so tensorflow#43424

[yongtang]

Additional fixes that need to happen on tensorflow/tensorflow:

• PR Avoid using TF file system before TF is initialized with plugins (avoid circular dependency) tensorflow#43563

[yongtang]

One more fix (remove duplication of Env::Default() instance) that is needed in tensorflow/tensorflow:

• PR Split env library so that there is only one instance of Env::Default() tensorflow#43594
• PR Expose TF_RegisterFilesystemPlugin C API tensorflow#43680

[terrytangyuan]

cc @zouxu09 Would anyone in your team be interested in helping out with OSS storage support?

[yongtang]

@terrytangyuan One remaining issue in AZFS is that we commented out logging temporarily https://github.com/tensorflow/io/pull/1143/files#r517351641

This issue can be addressed by using C logging API in TensorFlow. A PR in tensorflow repo has been created to expose the header files in pip package:

• Expose Logging C API in pip package tensorflow#44690
• [CherryPick:r2.4] Expose Logging C API in pip package tensorflow#44771

[yongtang]

With AZFS and HTTP file system converted to modular file system C API we can thinking about maintain forward-compatibility now. One thing we can do, is to lazy load the libtensorflow_io.so (non-forward-compatible) and only imperatively load libtensorflow_io_plugins.so (forward-compatible). In this way it will be possible to allow user to use file systems in future TF versions when they only touch the following:

import tensorflow_io as tfio

I have created a PR #1208 for that purpose.

The only outstanding issue is that, if we lazy-load libtensorflow_io.so, then OSS file system (still inside libtensorflow_io.so) will not be immediately available with import tensorflow_io as tfio. We can still have some other ways, e.g., expose import tensorflow_io.experimental.oss as oss to load libtensorflow_io.so for oss. Though we may also want to think about the timeline.

/cc @terrytangyuan do you know if there is any timeline with respect to oss moving to modular file system C API?

[terrytangyuan]

@zouxu09 is working on the OSS part and it's tracked in #1180. I'll defer this to @zouxu09 to comment on the timeline.