Force GPU tests run in GPU CIs

[mthrok]

We use skipIfNoCuda decorator to skip CUDA tests when CUDA is not available. This is nice but in CI with GPUs we should raise an error if CUDA is not available. We should update the decorator, so that we can provide further option to fail.

Setup

Please refer to the CONTRIBUTING.md for setting up torchaudio development environment.

Steps

1. Update the decorator to something like the following:

def skipIfNoCuda(test_item):
    if torch.cuda.is_available():
        return test_item
    force_cuda_test = os.environ.get('TORCHAUDIO_TEST_FORCE_CUDA', '0')
    if force_cuda_test not in ['0', '1']:
        raise ValueError('"TORCHAUDIO_TEST_FORCE_CUDA" must be either "0" or "1".')
    if force_cuda_test == '1':
        raise RuntimeError('"TORCHAUDIO_TEST_FORCE_CUDA" is set but CUDA is not available.')
    return unittest.skip('CUDA is not available.')(test_item)

2. Update CI to device TORCHAUDIO_TEST_FORCE_CUDA environment variable
   1. Add the definition here and here
   2. Run .circleci/regenerate.py, so that .circleci/config.yml is updated.
   3. Use CircleCI CLI to verify that the config.yml is valid. (circleci config validate)

Test

1. local test
   This should work

CUDA_VISIBLE_DEVICES=0  TORCHAUDIO_TEST_FORCE_CUDA=0 pytest test/torchaudio_unittest/functional/torchscript_consistency_cuda_test.py

This should fail

CUDA_VISIBLE_DEVICES=0 TORCHAUDIO_TEST_FORCE_CUDA=1 pytest test/torchaudio_unittest/functional/torchscript_consistency_cuda_test.py

2. CI test
   Please make a draft PR and see if the GPU tests pass.

[salilsdesai]

I'd like to work on this.

[mthrok]

Hi @salilsdesai

Thanks. Let me know if you need help.