Multi Head Attention Layer

[grafael]

I think is a good idea start to think how to implement this sort of layer in Keras.
I know that is a really fresh algorithm, but I believe that's a new cutting edge tech in Deep Learning for the next years.

Paper: Attention is all you need (https://arxiv.org/abs/1706.03762)

Blog showing some results: Google Research Blog
Tensor2Tensor library tensor2tensor
Pytorch implementation pytorch-t2t

[andhus]

@grafael Please keep an eye on this #8296. I'll try to have a look at the specific case to see if it will be covered.

[grafael]

Thank @andhus I'll follow the updates

[soham97]

Yeah attention layer is defacto standard used in NLP problems to achieve state of art be it generative or classification. I have implemented the attention layer in keras, and have obtained good results from it. It could be be much better if the layer is added to keras, so public can directly use it. Should I share the implementation in the thread or how is the procedure ?

[utkarshshukla2912]

@soham97 it would be great if you could i was trying to implement it couldn't get the thing working

[lvapeab]

Hi,

I implemented this some time ago in a fork. It is somewhat dirty and lacks test suites, but it works (an NMT example of this).

Cheers.

[utkarshshukla2912]

@lvapeab thanks, man !!

[miniwa]

Whats the status on this?
Seeing OpenAI's success with them made me want to try it.

[dynamicwebpaige]

Closing, as there is now a Keras-friendly multi-head attention layer in TensorFlow Addons. Thanks for the feature request!