Checking the data from failure mode 2; the source audio file and audio files processed with sox have the same checksum. This indicates that there is some un-deterministic behavior in torchaudio's implementation.

Success - torchaudio_unittest.backend.sox_io.save_test.SaveTest.test_save_flac_bytesio_16_0.tar.gz

Failure - torchaudio_unittest.backend.sox_io.save_test.SaveTest.test_save_flac_bytesio_16_0.tar.gz

Checksum of generated files.

Success

$ sha256sum torchaudio_unittest.backend.sox_io.save_test.SaveTest.test_save_flac_bytesio_16_0/*
1a43dd5202839020aa2bc8dbdf10bd5d4038359221f0bfd5a310233fa18f74a9  1.source.wav
94d012cd58ab008f214af795b95f36688a1fa6263df2bbfc8176f04f36280bf9  2.1.torchaudio.flac
7dee548df31d8e8f53d208568e5f559f6641ea8598bf464b5c1e0c23dd714462  2.2.result.wav
7471e954e0fd44a710b8ed75a42370ba2b709d63b47eb30789db13d02fc85033  3.1.sox.flac
7dee548df31d8e8f53d208568e5f559f6641ea8598bf464b5c1e0c23dd714462  3.2.ref.wav

Failure

$ sha256sum torchaudio_unittest.backend.sox_io.save_test.SaveTest.test_save_flac_bytesio_16_0/*
1a43dd5202839020aa2bc8dbdf10bd5d4038359221f0bfd5a310233fa18f74a9  1.source.wav
efa33771a51c6ccbb41615c2729335aa8f5a0782dc898e6599305493e2d5dd27  2.1.torchaudio.flac
d63b694fea46b766676589b1fb8ce720b495c22e3282dae0e830e5d9e6fe86ae  2.2.result.wav
7471e954e0fd44a710b8ed75a42370ba2b709d63b47eb30789db13d02fc85033  3.1.sox.flac
7dee548df31d8e8f53d208568e5f559f6641ea8598bf464b5c1e0c23dd714462  3.2.ref.wav

Where are they different?

Truncating the first 10K (12K total) gives the same check sum. This suggests that the un-deterministic behavior occurs at the end where the file is closed.

Success

$ truncate -s 10K  2.1.torchaudio.flac
$ sha256sum 2.1.torchaudio.flac
59cdc96b07b36031aa50ffed02ed2c5de1e8024c5e7d13015265d1569923ba1d  2.1.torchaudio.flac

Failure

$ truncate -s 10K  2.1.torchaudio.flac
$ sha256sum  2.1.torchaudio.flac
59cdc96b07b36031aa50ffed02ed2c5de1e8024c5e7d13015265d1569923ba1d  2.1.torchaudio.flac

How are they written?

Tapping into the number of bytes written each time write() is called, there is a slight difference in success case and failure case.

debug wrapper

class DebuggWrapper:
    def __init__(self, fileobj):
        self.fileobj = fileobj
        self.req_count = 0
        self.count = 0

    def read(self, size: int) -> bytes:
        bytes_ = self.fileobj.read(size)
        self.req_count += size
        self.count += len(bytes_)
        print(f'Read: {self.count} / {self.req_count}', file=sys.stderr)
        return bytes_

    def write(self, data: bytes):
        self.count += len(data)
        print(f'Write: {self.count}', file=sys.stderr)
        self.fileobj.write(data)

Success

Write: 1734
Write: 3899
Write: 5525
Write: 7690
Write: 9316
Write: 10940
Write: 11506

Failure

Write: 1734
Write: 3899
Write: 5525
Write: 7690
Write: 9316
Write: 10940
Write: 11048

Potentially problematic code?

The final write is called at

audio/torchaudio/csrc/sox/io.cpp

Lines 299 to 303 in 3488f31

    
           // Closing the sox_format_t is necessary for flushing the last chunk to the 
        
           // buffer 
        
           sf.close(); 
        
           fileobj.attr("write")(py::bytes(buffer.ptr, buffer.size));

The problem here is that, to get the final byte chunk, we need to close the sox_format_t structure. But doing so for the case of in-memory file object, we will lose the way to see how many bytes are written to the buffer. In the above code we use buffer.size, hoping that under the hood, the new buffer size is same as the number of data written, which could be wrong...

But where is the un-deterministic behavior coming from?

The above investigation still does not explain where is the origin. Is it open_memstream used by libsox? libsox implementation or flac encoder?

	// Closing the sox_format_t is necessary for flushing the last chunk to the
	// buffer
	sf.close();

	fileobj.attr("write")(py::bytes(buffer.ptr, buffer.size));

Tracking Issue for Flaky File-like obj CI failure #1229

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions