Problems when using GPU

[by2101]

When I use kaldi compliance lib, I found that if the wave is in GPU, much device mismatch exsits.
For example, when I use fbank function, it will report

Expected object of backend CUDA but got backend CPU for argument #2 'other'

It is caused by the functions in kaldi.py. Some newly constructed tensors are built on CPU.
I have fixed some. And we expect an officially fix.

Thank you.

[rfalcon100]

I am getting a similar error when using transforms.MelSpectrogram and running on GPU. Maybe this is related?. This is from the release version 0.3.

Traceback (most recent call last):
  File "/m/home/home7/77/falconr1/unix/.pycharm_helpers/pydev/pydevd.py", line 1741, in <module>
    main()
  File "/m/home/home7/77/falconr1/unix/.pycharm_helpers/pydev/pydevd.py", line 1735, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/m/home/home7/77/falconr1/unix/.pycharm_helpers/pydev/pydevd.py", line 1135, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/m/home/home7/77/falconr1/unix/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/l/code/audioTagging2019/poc/robustness/baseline.py", line 736, in <module>
    main(config)
  File "/l/code/audioTagging2019/poc/robustness/baseline.py", line 468, in main
    specs = do_time_frequency_transform(audio, transformer, config) # time-freq transform
  File "/l/code/audioTagging2019/poc/robustness/baseline.py", line 356, in do_time_frequency_transform
    tmpList.append(transformer(tmp))
  File "/m/home/home7/77/falconr1/unix/.conda/envs/audio/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/m/home/home7/77/falconr1/unix/.conda/envs/audio/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/m/home/home7/77/falconr1/unix/.conda/envs/audio/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #2 'mat2'
The above operation failed in interpreter, with the following stack trace:
at /m/home/home7/77/falconr1/unix/.conda/envs/audio/lib/python3.7/site-packages/torchaudio/transforms.py:154:24
        """
        if self.fb.numel() == 0:
            tmp_fb = F.create_fb_matrix(specgram.size(1), self.f_min, self.f_max, self.n_mels)
            # Attributes cannot be reassigned outside __init__ so workaround
            self.fb.resize_(tmp_fb.size())
            self.fb.copy_(tmp_fb)

        # (channel, frequency, time).transpose(...) dot (frequency, n_mels)
        # -> (channel, time, n_mels).transpose(...)
        mel_specgram = torch.matmul(specgram.transpose(1, 2), self.fb).transpose(1, 2)
                       ~~~~~~~~~~~~ <--- HERE
        return mel_specgram

And torchaudio version:

Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchaudio
>>> print(torchaudio.__version__)
0.3.0+faed00e
>>> 

[rfalcon100]

I think I found a solution, based on this and #263.

In short, I created a nn.Sequential model/block with a few transforms, including torchaudio.transforms.MelSpectrogram. But then, on my main code, I moved the input tensors to GPU but not this model.

To fix it, I added a to.("cuda")to the model, and now it works.

model = nn.Sequential(
                torchaudio.transforms.MelSpectrogram(sample_rate=config.resampling_rate,
                                            n_fft=config.n_fft,
                                            win_length=config.hop_length,
                                            hop_length=config.hop_length,
                                            f_min=float(config.fmin),
                                            f_max=float(config.fmax),
                                            pad=0,
                                            n_mels=config.n_mels
                                            ),
                make_module(myTransforms.RandomCrop)(config.max_length_frames),
                torchaudio.transforms.AmplitudeToDB(stype='power', top_db=80),
            )

And in the main loop:

...
for audio, labels in dataloader_train:
    audio = audio.to("cuda")  # (batch, channel, time)
    labels = sample.to("cuda") # (batch, tags)

    model = model.to("cuda")
    # torchaudio.transforms has no support for batched, using loops...
    tmpList = []
    for i in range(audio.shape[0]):
        tmpList.append(model(audio[i,:,:]))

    # Stack adds a dim, so the shape is (batch, channel, freq_bins, frames) after doing STFT
    spectograms = torch.stack(tmpList, dim=0)
...

[by2101]

I added some device checking codes in the .py files. For example,

log_energy = torch.max(strided_input.pow(2).sum(1), epsilon).log().to(strided_input.device)

I don't know whether pytorch optimize some functions on GPU, such as fft. If the codes are fast on GPU, I think it is better to put the model on GPU.

[vincentqb]

When I use kaldi compliance lib, I found that if the wave is in GPU, much device mismatch exsits.
For example, when I use fbank function, it will report

Expected object of backend CUDA but got backend CPU for argument #2 'other'

It is caused by the functions in kaldi.py. Some newly constructed tensors are built on CPU.
I have fixed some. And we expect an officially fix.

Thanks for reporting this! Can you provide a minimal example to show the issue you are running into?

[vincentqb]

I added some device checking codes in the .py files. For example,

log_energy = torch.max(strided_input.pow(2).sum(1), epsilon).log().to(strided_input.device)

I don't know whether pytorch optimize some functions on GPU, such as fft. If the codes are fast on GPU, I think it is better to put the model on GPU.

Feel free to open a pull request to suggest those changes. Thanks!

[vincentqb]

This issue should be resolved by #314. I will close this issue, but please feel free to re-open if this is not the case.