add t5 model that can function as both encodery-only or encoder-decoder model

[pmabbo13]

Stack from ghstack (oldest at bottom):

• -> add t5 model that can function as both encodery-only or encoder-decoder model #1829
• add t5 stack that can function as either the encoder or decoder of a t5 model #1828
• add t5 layer module that can be used for both encoder or decoder stack #1827
• add layer norm module for t5 model #1826
• adding forward method for multihead attention #1833
• computing attention scores using relative attention bias #1832
• computing relative attention bias #1831
• compute relative position buckets for relative attention bias #1830
• create T5MultiheadAttention module #1825

[Nayef211]

@parmeet I wonder if you think it makes sense to update our docs to include the T5Model class as well as the yet to be created T5Bundle. Similarly is there a reason why we don't include the RobertaModel class in the docstring if it's a public facing component?

[pmabbo13]

Description

The T5Model implementation is very similar to the nn.Transformer implementation, with some additional functionality

1. Takes a tokenized encoder input sequence and decoder input sequence and transforms them into word embeddings.
2. Computes the padding masks for the input sequences based on the padding_idx input argument when initializing the model.
3. Generates a causal mask for decoder self-attention unless one has already been provided via the decoder_mask input argument to the forward method.
4. Returns the output of the final layers to the encoder and decoder, the output at each layer of the encoder and decoder, the self-attentions scores of each layer of the encoder and decoder, the cross-attention scores of each layer of the decoder
5. Based on input parameter encoder_only when initializing the model, will just run the encoder portion and output its corresponding results.

[parmeet]

Overall LGTM!

[parmeet]

I wonder if you think it makes sense to update our docs to include the T5Model class as well as the yet to be created T5Bundle.

This is just a prototype feature. So it may not be necessary to include it in the docs. I don't see other domains doing this as well?

Similarly is there a reason why we don't include the RobertaModel class in the docstring if it's a public facing component?

hmm, that's a good catch. Not really. Looks like we missed including the docs for it, perhaps we were only focusing on the Roberta Bundler API that expose this model to users..