add t5 layer module that can be used for both encoder or decoder stack

[pmabbo13]

Stack from ghstack (oldest at bottom):

• add t5 model that can function as both encodery-only or encoder-decoder model #1829
• add t5 stack that can function as either the encoder or decoder of a t5 model #1828
• -> add t5 layer module that can be used for both encoder or decoder stack #1827
• add layer norm module for t5 model #1826
• adding forward method for multihead attention #1833
• computing attention scores using relative attention bias #1832
• computing relative attention bias #1831
• compute relative position buckets for relative attention bias #1830
• create T5MultiheadAttention module #1825

[Nayef211]

Let's highlight what the modifications are with the use of a PR comment

[pmabbo13]

Let's highlight what the modifications are with the use of a PR comment

updated!

[pmabbo13]

Description

The T5Layer implementation is very similar to the nn.TransformerEncoderLayer and nn.TransformerDecoderLayer implementations. The main differences are that:

1. T5Layer generalizes to be used as either an encoder or decoder layer. For the encoder we perform self-attention + feed-forward. For the decoder we perform self-attention + cross-attention + feed-forward. If the boolean parameter is_decoder = True, the forward method will perform cross-attention after performing self-attention. Otherwise, it will skip the cross-attention step and go directly to the feed-forward step.
2. T5Layer takes in parameters relative_attention_num_buckets, relative_attention_max_distance, compute_relative_attention and relative_attention_bias. This is information passed into the self-attention block so that it can include a relative attention bias term when computing attention scores.

[parmeet]

Overall LGTM!

[Nayef211]

LGTM