Request for examples: Seq2seq

[seanpmorgan]

I believe there are many seq2seq examples already published, but it makes sense that we at least have one or two in our examples section for user experience.

Related:
https://google.github.io/seq2seq/
https://github.com/tensorflow/addons/blob/master/examples/README.md

[kazemnejad]

I can pick this up if it is ok.

Some ideas to start the example:

• For machine translation:
  • On toy English-to-German dataset (I think German-to-English has better user experience since we can demonstrate the outputs better) from http://opennmt.net/OpenNMT-tf/quickstart.html which contains 10k tokenized sentences.
  • Trivial seq2seq LSTM architecture with Attention Mechanism
  • Use beam decoding to generate outputs
  • Maybe BLEU calculation

[seanpmorgan]

I can pick this up if it is ok.

Some ideas to start the example:

• For machine translation:

  • On toy English-to-German dataset (I think German-to-English has better user experience since we can demonstrate the outputs better) from http://opennmt.net/OpenNMT-tf/quickstart.html which contains 10k tokenized sentences.
  • Trivial seq2seq LSTM architecture with Attention Mechanism
  • Use beam decoding to generate outputs
  • Maybe BLEU calculation

That'd be great! Thanks and welcome to Addons!

[PyExtreme]

Hi @seanpmorgan @kazemnejad , Is it still an Open Issue?
Asking because I am interested to work on it.

[kazemnejad]

Hi @PyExtreme, Thank you for showing your interest.
Yes, actually I'm currently working on it. I was waiting for the #375 to be fixed and it got fixed 3 days ago in #503. I think I can submit a PR in the next few days. Thus you can work on that PR if you want.

[matthen]

+1 any progress on this?
I'd appreciate an example of how to create and train a keras decoder with attention. I can't figure out how I am supposed to set up AttentionWrapper in a model without yet knowing the memory tensor

[kazemnejad]

Hi @Mainak431, Sorry for the inconvenience.

Actually, the draft of this example is present at my fork. However, in the default Keras training mode (graph mode) + tf.data.dataset (where the default mode is eager), there is a bug related to caching of tensor dimension which I guess is from the AttentionWrapper. Unfortunately, I'm busy with my university's works at this moment and I could not work on that bug, so I would be happy if someone could find the source of that bug and help the progress.

[matthen]

thanks for sharing @kazemnejad! Where do you get the bug? I was getting an error at some point from the rnn cell being wrapped by AttentionWrapper that it was being passed a rank 1 tensor when it was expecting a rank 2

[zhedongzheng]

Thanks @kazemnejad for submitting the fix. Based on his contribution, I have successfully get the seq2seq code running for a semantic parsing task.

For anyone who is interested, the notebook is here