Some issues with Kaldi MFCCs features

Hi,
I'm trying to do some experiments with the kaldi-compliant MFCCs, but I run into some possible issues:

1- When I run the following code

 file='/home/mirco/datasets/TIMIT/test/dr5/fnlp0/si1308.wav'
 [signal,fs]=sf.read(file)
 signal=torch.from_numpy(signal).unsqueeze(0).float()
 fea=mfcc(signal)
 print(fea)

The mfccs are different every time I run the script:

run 1

tensor([[ 29.2496, -32.6150,  -7.1791,  ...,  -6.2034,  -5.8100,   3.5894],
       [ 28.1680, -35.9921,  -8.5621,  ..., -13.5980,  -4.2804,  -8.8075],
       [ 29.2831, -31.8580,  -8.8565,  ...,  -5.8166,  -4.2538,   6.4913],
       ...,
       [ 27.5078, -36.1139, -12.1319,  ..., -11.6493,   0.2557,  -4.9566],
       [ 28.9667, -33.5803,  -6.6644,  ...,  -6.1208,   2.7111,   2.7867],
       [ 28.6988, -33.6590, -12.0312,  ...,  -3.0909,  -0.0643,  -4.1769]])

run 2

tensor([[ 27.8255, -33.2356,  -8.8006,  ..., -13.2640,   1.0311,   4.8004],
       [ 29.5605, -34.0147, -10.3465,  ...,  -4.0096,  -1.5156,  -3.2499],
       [ 29.3978, -31.3415,  -6.4141,  ...,  10.6100,   2.3651,   6.1324],
       ...,
       [ 29.4321, -33.0013, -11.9812,  ...,  -2.9076,   6.3498,   1.8854],
       [ 28.2726, -34.0620,  -9.5291,  ...,  -5.4033,   6.0385,  -0.1867],
       [ 29.5408, -33.7757,  -9.1063,  ...,   5.8796,  -6.1365,  -3.2730]])

This is due to the dithering, that is a type of noise. The problem can be easily solved by setting a manual_seed before executing the code. To avoid issues, the users should be aware of that. Maybe it could be great to provide an example in the documentation.

2- Even if I remove the dithering in both kaldi and torchaudio mfccs, the two vectors are very very different (the options in the two cases are exactly the same):t

torch.audio

> tensor([[-64.7435, -23.0893,   1.5796,  ...,  -4.9001,  -1.5039,  -2.7683],
>        [-61.5527, -17.7455,  -5.9670,  ...,   4.4663,   2.5523,  -0.9595],
>        [-58.9998, -21.4523, -10.7197,  ...,  10.2993,   9.5475,  -0.3667],
>        ...,
>        [-65.0258, -23.9535,   3.2329,  ...,   4.1740,   9.9711,  -1.2087],
>        [-65.4491, -23.4586,   3.0314,  ...,   3.4530,  -0.2666,  -3.1916],
>        [-65.9383, -23.1859,   3.6318,  ...,   2.9440,   3.5066,  -2.4232]])

kaldi

fnlp0_si1308  [
 33.93769 -26.93453 -4.314013 -9.108547 -2.538414 -7.403401 -7.393436 -19.1162 2.36114 -3.599539 -8.258158 -3.048464 -2.534939
 37.16539 -20.76378 -10.65134 -14.69143 -5.084549 -13.17811 -19.8767 -11.37231 0.9925694 3.125628 1.008414 0.7657758 -1.482104
....

3- When I put the tensor into the GPU (with .to('cuda')), I have the following error:
log_energy = torch.max(strided_input.pow(2).sum(1), epsilon).log() # size (m)RuntimeError: Expected object of backend CUDA but got backend CPU for argument #2 'other'

How can I compute the MFCCs using cuda?

4- A good thing is that the execution time (even on a single cpu only) is compatible with the kaldi one (around 15 second for the entire TIMIT dataset). I would expect a further speed up on the GPU.

5- I tried to feed the MFCCs coefficient inside a standard speech recognizer and the performance with the original kaldi coefficients is still much better. Do you have the same experience?

Thank you and thanks for developing this very useful toolkit!

[vincentqb]

Let's start with 1. I don't see the fluctuations that you mention. I'm using this file.

import soundfile as sf
import torch
import torchaudio

filename = "sample.wav"
waveform, sample_rate = sf.read(filename)
waveform = torch.from_numpy(waveform).mean(-1).unsqueeze(0).float()

mfcc = torchaudio.transforms.MFCC()

mfcc(waveform)
mfcc(waveform)
mfcc(waveform)
mfcc(waveform)
mfcc(waveform)

I get the same output after each call mfcc(waveform).

tensor([[[-7.3096e+02, -7.3096e+02, -7.1735e+02,  ..., -5.1697e+02,
          -5.3175e+02, -5.0290e+02],
         [ 5.2458e-06,  5.2458e-06,  1.3515e+01,  ...,  1.3600e+02,
           1.2202e+02,  1.3729e+02],
         [-7.4861e-05, -7.4861e-05,  1.3204e+01,  ...,  9.8696e+00,
           4.6198e+00,  7.2932e+00],
         ...,
         [ 8.0109e-05,  8.0109e-05, -3.1892e-01,  ..., -9.0881e+00,
          -1.0949e+01, -1.1669e+01],
         [-2.0412e-04, -2.0412e-04, -2.4082e+00,  ..., -9.9953e+00,
          -1.0371e+01, -7.6354e+00],
         [-1.8072e-04, -1.8072e-04,  1.3875e+00,  ..., -1.0247e+01,
          -1.3864e+01, -5.6211e+00]]], grad_fn=<DifferentiableGraphBackward>)

[vincentqb]

For 3, can you give a more complete code snippet? It looks like a variable (e.g. epsilon or strided_input) has not been moved to the GPU, see for example here and here:

strided_input = strided_input.to("cuda")
epsilon = epsilon.to("cuda")

[vincentqb]

For 5, I'm assuming you mean performance=runtime. Can you provide comparison/measurement? Maybe even code snippet for this?

[HsunGong]

Let's start with 1. I don't see the fluctuations that you mention. I'm using this file.

import soundfile as sf
import torch
import torchaudio

filename = "sample.wav"
waveform, sample_rate = sf.read(filename)
waveform = torch.from_numpy(waveform).mean(-1).unsqueeze(0).float()

mfcc = torchaudio.transforms.MFCC()

mfcc(waveform)
mfcc(waveform)
mfcc(waveform)
mfcc(waveform)
mfcc(waveform)

I get the same output after each call mfcc(waveform).

tensor([[[-7.3096e+02, -7.3096e+02, -7.1735e+02,  ..., -5.1697e+02,
          -5.3175e+02, -5.0290e+02],
         [ 5.2458e-06,  5.2458e-06,  1.3515e+01,  ...,  1.3600e+02,
           1.2202e+02,  1.3729e+02],
         [-7.4861e-05, -7.4861e-05,  1.3204e+01,  ...,  9.8696e+00,
           4.6198e+00,  7.2932e+00],
         ...,
         [ 8.0109e-05,  8.0109e-05, -3.1892e-01,  ..., -9.0881e+00,
          -1.0949e+01, -1.1669e+01],
         [-2.0412e-04, -2.0412e-04, -2.4082e+00,  ..., -9.9953e+00,
          -1.0371e+01, -7.6354e+00],
         [-1.8072e-04, -1.8072e-04,  1.3875e+00,  ..., -1.0247e+01,
          -1.3864e+01, -5.6211e+00]]], grad_fn=<DifferentiableGraphBackward>)

The results in torchaudio.transform remain the same, however, results in torchaudio.compliance.kaldi changes after different executions. The problem is due to #263 (comment)

[HsunGong]

#228 (comment) maybe have some problems.

[HsunGong]

tensor([[ 18.3451, -14.9228, -18.3650,  ...,  -6.3634,   1.8486,  -8.7933],
        [ 20.3241, -11.0113, -16.5513,  ..., -10.2232,  -2.2399, -13.0235],
        [ 22.7283,   7.1456, -32.8585,  ..., -14.6573, -22.6597,  -8.8397],
        ...,
        [ 15.3196, -18.4401,   3.7009,  ..., -20.2965,   7.2492,  -6.4407],
        [ 15.3898, -19.9645,  -5.6626,  ..., -12.0694,   5.0258, -15.5496],
        [ 14.6091, -23.0545,  -7.0441,  ..., -31.6500,  -7.9601,  -8.3047]]) torch.Size([7249, 13])
torch.Size([1, 580080]) 8000
tensor([[-18.8921, -14.9228, -18.3650,  ...,  -6.3634,   1.8486,  -8.7933],
        [ -8.8658, -11.0113, -16.5513,  ..., -10.2232,  -2.2399, -13.0235],
        [ -2.7176,   7.1456, -32.8585,  ..., -14.6574, -22.6597,  -8.8396],
        ...,
        [-33.5852, -18.4401,   3.7009,  ..., -20.2965,   7.2492,  -6.4408],
        [-33.6183, -19.9645,  -5.6626,  ..., -12.0694,   5.0258, -15.5496],
        [-36.0048, -23.0545,  -7.0441,  ..., -31.6500,  -7.9601,  -8.3048]]) torch.Size([7249, 13])
.

Find out: if set dither == 0., then the results fixed.

Comment closed!