computing relative attention bias

[pmabbo13]

Stack from ghstack (oldest at bottom):

• add t5 model that can function as both encodery-only or encoder-decoder model #1829
• add t5 stack that can function as either the encoder or decoder of a t5 model #1828
• add t5 layer module that can be used for both encoder or decoder stack #1827
• add layer norm module for t5 model #1826
• adding forward method for multihead attention #1833
• computing attention scores using relative attention bias #1832
• -> computing relative attention bias #1831
• compute relative position buckets for relative attention bias #1830
• create T5MultiheadAttention module #1825

[Nayef211]

LGTM

[pmabbo13]

Description

This module computes the relative position embeddings. This is the bias term that will be used when computing the attention scores. Our implementation is taken from HF.

[parmeet]

LGTM!