Support live audio for tensorflow-io

[yongtang]

One interesting usage of tensorflow from the community is to process live audio stream. For that I think it would be great to support live audio stream in tensorflow-io so that more applications could benefit from it directly.

Note this issue might be different from #49 (audio file format, not live audio).

[yongtang]

FYI @Alireza89

[Alireza89]

Thanks @yongtang for raising this issue.

Recently I've written two c/c++ programs on top of speech_commands/test_streaming_accuracy.cc which use JACK2 audio server daemon in Linux to capture the sound and perform real-time audio recognition with little latency (~30ms).

Here is a demonstration of the working code in Ubuntu 18.04:
https://youtu.be/2CoRbuRRKbw

And these are links to the source code:

1. test_live_stream.cc :
   https://github.com/Alireza89/tensorflow/blob/master/tensorflow/examples/speech_commands/test_live_stream.cc

2. jack_capture_tensorflow.c :
   https://github.com/Alireza89/tensorflow/blob/master/tensorflow/examples/speech_commands/jack_capture_tensorflow.c

3. Updated BUILD file for Bazel :
   https://github.com/Alireza89/tensorflow/blob/master/tensorflow/examples/speech_commands/BUILD

Hope the TensorFlow community find this useful.

[ivelin]

I am looking at wiring tensorflow models into live SIP/RTP streams for VoIP and WebRTC call scenarios such as live virtual agents. Still researching design and implementation options. Please let me know if anyone is working on this and interested to coordinate effort.

@Alireza89 thank you for sharing your work. I think its relevant to my subject and will study it.

[yongtang]

@ivelin As far as I know no one is actively working on this issue at the moment. The support for live audio may not be very difficult, if there is a reliable and actively maintained C/C++ audio library available (not realistic for tensorflow-io to start from scratch). I could help the implementation if a C/C++ audio library available.

The requirement is that:

• Library is in C/C++
• Actively maintained with good quality
• Compatible license
  • Prefer licenses without restrictions (Apache/MIT/BSD)
  • tensorflow-io is Apache License so we could not take GPL license
  • If the library is LGPL license, then it needs to be part of a major Linux distribution. For example, tensorflow-io includes ffmpeg with dynamic linking. However, we only link against Ubuntu 14.04/16.04's system installed ffmpeg with very fixed versions. We could not afford to support a wide range of dynamic linking with random version of ffmpeg.

[ivelin]

@yongtang Thank you for the guidance.

There is a plethora of open source SIP and RTP libraries out there in C, Java and other languages. My team leads one of the more popular open source Java telephony middleware stacks - Restcomm.

The search criteria here is a library that satisfies your guidelines as well as:

• Allows convenient attachment to VoIP calls for bi-directional TF processing.
• Allows production deployment at scale to millions of concurrent calls.
• Allows two or more participants in a conversation. Conference calls are an interesting area of exploration for ML.
• Allows long running bi-directional conversations connected to LSTM RNN or other types of contextual recurrent NN models. Mainstream production deployments seem to be voice command and responses via Echo type of devices, which simulate more traditional IVR menus, but would be great to make TF easier for testing more natural contextual dialogs.
• Allows experimentation with models that go beyond Dialogflow type of tools, which would not require explicit feature engineering for context labels and explicit call path design between each request/response pair.

On this note, does anyone know if the Google Assistant team has released (or is planning to release) under ASL their voice streaming connectors to TF? That could be a good place to start.

I looked at the AWS Lex REST API as a potential approach but I don't think its a great fit here, because its designed for a discrete series of request / response pairs over HTTP, which is fine for voice commands, but is not exactly the way natural phone conversations take place.

I will do some more research and get back with suggestions.

[yongtang]

@ivelin Thanks for the details. Thinking again I realized even C/C++ is not necessarily needed. We actually could use libraries written in other languages with various ways:

1. For RTP/SIP written in golang (or even maybe swift), we could build golang and expose cdecl api so that tensorflow-io could call cdecl API directly. This should be fairly straightforward.
2. For RTP/SIP written in Java, the easiest way is to expose gRPC endpoint so that tensorflow-io could call gRPC endpoint and access stream. The gRPC is used by Google Cloud as the default API format. And gRPC automatically generate the server/client side stub code for you. It is very easy as well. Could be done in a matter of a couple of days in most cases.

So overall, we will need a library:

• Actively maintained with good quality
• Compatible license
• Could be C/C++/Golang/Java

[ivelin]

@yongtang noted. Allow me a few days to dig deeper in the options and get back to you.

[ivelin]

@yongtang I've made a bit of progress educating myself on this topic. I see potential to split Issue #50 in two related but separate issues:

1. Capturing audio stream on a local operating system port (hardware or TCP/IP) and feeding it to TF for training and predictions. The code provided by @Alireza89 is a great start. He's already written a TF class that can continuously read audio input from a local memory file into a TF session. It would be straightforward to use a library such as pcapsipdum to log into a local pcap file the audio packets coming to a local network interface over SIP/RTP and from there to audio_data_tensor.

2. Cloud scale processing of many thousands of simultaneous SIP/RTP calls. I am still researching this problem. So far I am hitting a rock with request/response based solutions like REST and gRPC. Some of the key problems I see:

• If a TF model is involved in some way in a real phone call with expectations that the application can process natural language and engage with a caller (or callers in a conference call), the latency for response cannot be more than 100ms. Longer than that makes the interaction feel artificial. The caller doesn't quite know if the synthetic model received and understood the input. Some sort of immediate minimal feedback has to be provided as soon as the caller stops talking. Normally in telephony networks we look for latency under 30ms. When its much more than that callers get confused whether to keep talking or listen. You start hearing callers talking over each other although they don't mean to.

• Using request/response API to transform streaming UDP packets into TF input and then potentially again on the way back introduces several challenges I'd like us to solve:
  -- Latency due to buffering of packets to fill a data array for a gRPC/HTTP request.
  -- Latency due to repackaging of data from its raw wire format to a different data array format for the gRPC/HTTP request and then again for the TF input format.
  -- Loss of UDP RTP packets during transformation to TCP interface. Since we are working with real time speech, if the gRPC interface has a network delay, we can't just continue buffering the incoming RTP as if it was a song or a movie and play it later when the gRPC interface is available again. It will throw off the rhythm of the conversation as described above. If we throw away the packets to allow processing of fresh packets, then the voice will sound interrupted.

Please feel free to correct me if I am missing something. Otherwise I will keep looking for options and share feedback here when I find a suitable answer.

Beyond the issues of interfacing efficiently SIP/RTP and TF, there is also the problem with performance and scalability due to TF processing time and potentially delays due to integration with other data sources. When we get to this bridge, we may think about crossing it by splitting TF models into several categories, ones that are less computationally intensive, live closer to the edge of the network and allow instant speech feedback to callers. For example detecting silence and reacting with something simple like "Ok, let me see". Then running the more computationally intensive models for a full response. Although this type of problem probably doesn't belong in the SIG IO group. I suppose only to the extend of finding the upper limit of latency for (IO + TF model processing) < 30ms.

[yongtang]

Thanks @ivelin, I created a new issue #241 to cover the local os live audio capture part.

For cloud scale processing of many thousands simultaneous SIP/RTP calls, there are several potential issues though I think it is still might be feasible.

1. For grpc serialization cost, I haven't done performance for quite some time, but 30ms seems to be within the range.
2. For model processing, one thing to note is that, if a user talks, model does not need to wait for the very last moment (last ms) to process. At least it might be possible for the model to build up the needed knowledge in order to have several possible paths to explore as user's talk continue. There might be still some challenges as the model or workflow might not be the same as traditional ways of processing.
3. Simultaneous SIP/RTP calls might be more about scale-out than scale-up. Scale-out typically are easier as in many cases the bottleneck is just about load balancing (could be a combination of IP-level LB and DNS).

Overall, I think those are good topics and a good chunk of them (not all of them) fall into SIG IO's focus. Things could be built-up gradually so that it could help the community.

[ivelin]

Thanks @yongtang for splitting the issue. To your points:

1. I will do some benchmarking to see what are the boundaries of grpc related transformation costs. As I'm still researching, I came across the 2019 Google I/O keynote announcing that the 100GB voice model was crunched to 500MB to fit on a phone. The primary reason cited was reduction in latency. This is good news for folks that can solve their problems with local OS IO that Issue Support live audio on local os for tensorflow-io #241 targets . It remains an open topic how to bring this low latency into VoIP nodes deeper in the SIP network.

2. Yes, I am aware that some models (e.g. RNN) allow continuous processing and output of predictions with partial input of samples, which is great. That's a task to tackle once we figure a way to feed high performance RTP streams in and out of TF models. :)

3. Yes, I agree that scale out is a separate issues that we can address separately. The main challenge right now is to find high performance processing solution of a single RTP stream in and out of TF with less than 30ms total delay.

I'll post again as soon as I find something tangible to share. Of course if someone else has the answer, please comment.

[ivelin]

@yongtang it seems that bi-directional gRPC streaming could be a way forward. Here is an example:
https://grpc.io/docs/tutorials/basic/java/#bidirectional-streaming-rpc

I found a related thread under TF Serving, where the notion of bi-directional gRPC streaming was acknowledged over a year ago but declared out of scope. Do you know why that might be the case and whether its in scope currently for Serving for IO?
tensorflow/serving#722

I also researched the Google Speech to Text gRPC APIs a bit more and found out that bi-directional streaming was introduced with client SDKs as Alpha. The latest github activity suggests that bi-directional streaming for speech to text is moving towards beta.

We've done some testing with my team against the previous version of the speech-to-text API which only supported requests via fixed byte array or URL of a file on google cloud bucket. Testing from multiple locations around the world showed that the latency is in the order of 2-5 seconds for speech recognition of a short phrase plus receiving a callback with the text words. That's significantly above the 100ms maximum (30ms ideally).

Hopefully we can find a way to enable bi-directional gRPC into TF Serving. Looking forward to comments.

[yongtang]

@ivelin Sorry for the late reply as I was out of office last week. For bi-directional gRPC streaming, the protocol itself is straightforward and easy to implement at the first place. Since TensorFlow is protobuf based, write a gRPC server is really trivial. A starter server could normally done in < 100 lines of implementation, as most of the code are machine-generated automatically. Go with TF serving probably is not necessary as TF serving has some extra layers in between the server, format, and the TensorFlow's graph.

I think I could help the implementation in this case.