Support decoding video from byte stream for tensorflow-io

[wookayin]

Video (ffmpeg) support (#30) is awesome. I was wondering if we can improve the video dataset API by having a counterpart of tf.contrib.ffmpeg.decode_video, which decodes the video from the contents bytestream (tf.string) rather than a filename.

This can be useful in some use cases where videos are encoded and stored in a TFRecord dataset. A side note: it seems that a couple of exemplar video datasets stores a sequence of frame images (encoded as PNG) so I'm not 100% sure if this feature would be worthwhile.

I took a quick look at the C++ kernel implementation but the implementation seems quite tricky. One way is, as in tf.contrib.ffmpeg.decode_video, we could write the buffer content into a temporary file but I am doubtful this is the right way. Pipe or reading from stdin could work, but needs some study.

[yongtang]

@wookayin The appropriate way to achieve it, I remember, is to set callback open/read functions in AVFormatContext , so that avformat_open_input will use the open/read callback function instead (and bypass filename open).

In case a memory buffer is passed by, the open/read callback will simply copy the buffer around so that FFmpeg could use the copied buffer (than read buffer from file).

[wookayin]

Hi @yongtang, thanks for the guidance. I will look into that soon to see if I can implement this functionality.

[yongtang]

@wookayin Fantastic! Let me know if you encounter any issues.

[yongtang]

@wookayin Added a PR #295 to support reading video from streams.

[SudeepDasari]

Any documentation on how this works? I'd be very interested in a replacement for tf.,contrib.ffmpeg.decode_video which doesn't seem to work reliably at the moment.

For context, I can load byte strings from a tf.Dataset object (using either a custom format or TFRecords) and would like to use a Tensorflow op to decode the strings into mp4 videos.

[yongtang]

For context, I can load byte strings from a tf.Dataset object (using either a custom format or TFRecords) and would like to use a Tensorflow op to decode the strings into mp4 videos.

@SudeepDasari Are you referring to encode strings (raw stream) to mp4 videos, or decode videos?

[SudeepDasari]

I meant decoding the (mp4) videos from a string or byte buffer which I loaded myself through some other source (e.g TFRecords, HDF5, etc.). The following ImageIO code would provide the same functionality:

images = [img for t, img in enumerate(imageio.get_reader(io.BytesIO(buf), format='mp4'))

Context - I would like to load entire video sequences and batch them together (into [N, T, H, W, C] tensors) for some video understanding and prediction work.

[yongtang]

@SudeepDasari VideoDataset supports reading a filename (of video) and decode the video into a list of frames. What you are asking for is to read a file in memory and decode it into list of frames. This is not possible yet but should not be very easy to add this feature. Can you create another issue to track this feature request?

[SudeepDasari]

Done! Check #455