Prepare T5 Model for Language Generation

[pmabbo13]

Description

Add a T5 model that can be used for language generation tasks.

Process

For language generation, the outputs of the decoder pass through a linear layer which projects the hidden states to the vocab. The boolean parameter linear_head was added to T5Conf, which allows the user to specify if they want the model to include this final linear layer and return its output. A bundler object, T5_BASE_GENERATION was also created to instantiate such a model using pre-trained weights under the base configuration. These weights have been uploaded to the AWS and Manifold buckets as t5.base.generation.pt.

Testing

Tests that were used to check the outputs of the T5_BASE and T5_BASE_ENCODER models are also used to test the outputs of T5_BASE_GENERATION. Additionally, the linear head in the generation models allows us to compute the cross entropy loss. Therefore, we can now also test the train functionality.

pytest test/prototype/integration_tests/test_models.py
pytest test/prototype/models/test_models.py

Follow-Up

• test/prototype/models/test_models.py::TestModels::test_t5_bundler_train appears to be failing in select environments (ex. for the most recent commit it failed in windows_py3.8 and macos_py3.10, though the exact environment(s) it fails in varies with each new, yet unrelated, commit. It's unclear what the source of this failure is, and why it doesn't consistantly fail in the same environments.

[parmeet]

Thanks for adding Language generation capability to T5 model, overall LGTM!

[Nayef211]

LGTM