Adding Module Models

[tomassosorio]

🚀 Feature

Similar to torchvision would be interesting to have the architecture of the most important models available, such as Wav2Letter, DeepSpeech, DeepSpeech2 between others...

Motivation

When trying to train a model it is already available most features needed such as Datasets, Data Transformations, however it is not available any architecture of a known model.

Pitch

Adding a module with the architecture of the most known models.

[vincentqb]

This is a good idea. Would you like to contribute some?

EDIT: We need a way of validating a model implementation before being able to add it in torchaudio:

• For wav2letter and wavernn, we have a training pipeline showing reasonable convergence.
• In Add MobileNetV3 Architecture in TorchVision vision#3182, we (also) provided pre-trained weights that worked well with the model.
• If this is taken from an existing pytorch implementation, the reference can be used to compare numerically against.

[tomassosorio]

Sure!

I will inspire into torchvision structure and add at first Wav2Letter.

At the time I cannot offer the model with the trained weights, but I can talk with my company to see if we could offer this to torchaudio in the future.

[vincentqb]

Let's start with just the model class, untrained. Hosting trained models (and datasets) is a separate discussion :)

[vincentqb]

@tomassosorio -- thanks again for merging wav2letter! Which other ones do you believe would be a good idea to request help to implement?

• Wav2Letter
• DeepSpeech
• DeepSpeech2

[tomassosorio]

Yes, I fell both of them would be good options as future implementations, such as the Wav2Letter++, in this link there is also some other models that could be interesting to implement :)

[koukyo1994]

How about PANNs? It has a rich collection of pre-trained audio tagging / sed models.

[limazix]

Hi guys, I'd like to help with this task. Where can I start?

[Edresson]

@vincentqb I think it would be interesting to implement Jasper too, as it reported state of the art on LibreSpeech :).

It is available on OpenSeq2Seq

What do you think about converting the checkpoints of these models to pytorch and using them in the initialization of model weights?

[vincentqb]

@vincentqb I think it would be interesting to implement Jasper too, as it reported state of the art on LibreSpeech :).

It is available on OpenSeq2Seq

Thanks for suggesting jasper here, and there :) Do ping me when you open a pull request for it :)

What do you think about converting the checkpoints of these models to pytorch and using them in the initialization of model weights?

We do plan on offering some pre-trained models for those we offer. Is that what you mean?