add t5 stack that can function as either the encoder or decoder of a t5 model

[pmabbo13]

Stack from ghstack (oldest at bottom):

• add t5 model that can function as both encodery-only or encoder-decoder model #1829
• -> add t5 stack that can function as either the encoder or decoder of a t5 model #1828
• add t5 layer module that can be used for both encoder or decoder stack #1827
• add layer norm module for t5 model #1826
• adding forward method for multihead attention #1833
• computing attention scores using relative attention bias #1832
• computing relative attention bias #1831
• compute relative position buckets for relative attention bias #1830
• create T5MultiheadAttention module #1825

[Nayef211]

Leave comment with differences if any from original implementation

[pmabbo13]

Leave comment with differences if any from original implementation

updated!

[pmabbo13]

Description

The T5Stack implementation is very similar to the nn.TransformerEncoder and nn.TransformerDecoder implementations. The main differences are that:

1. T5Stack generalizes to be used as either an encoder or decoder layer. This is dependent on the value passed in for is_decoder and will dictate whether the stack of layers are configured to be encoder layers or decoder layers.
2. T5Stack returns the output of the final layer, the output of every layer in the stack, the self-attention scores computed at every layer in the stack, and the cross-attention scores at every layer of stack (will be None if stack is a decoder).

[parmeet]

LGTM!