torchvision.io.decode_image: segmentation fault on cuda

[SomeoneSerge]

🐛 Describe the bug

from io import BytesIO

import numpy as np
import torch
import torchvision as tv
from PIL import Image


def test_image_decode(device: torch.device):
    f32 = np.clip(np.random.randn(1024, 1024, 3), -1.0, 1.0)
    u8 = (f32 * 127.5 + 127.5).astype(np.uint8)
    image = Image.fromarray(u8)

    with BytesIO() as f:
        image.save(f, format="jpeg")
        b = f.getvalue()

    decoded = tv.io.decode_image(torch.frombuffer(b, dtype=torch.uint8).to(device))
    return decoded

if __name__ == "__main__":
    print(f"{tv.__version__=}")

    print("Testing on CPU")
    test_image_decode(torch.device("cpu"))

    print("Testing on GPU")
    test_image_decode(torch.device("cuda"))

❯ python test-image-decode.py
tv.__version__='0.14.1a0'
Testing on CPU
test-image-decode.py:18: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /build/source/torch/csrc/utils/tensor_new.cpp:1563.)
  decoded = tv.io.decode_image(torch.frombuffer(b, dtype=torch.uint8).to(device))
Testing on GPU
Segmentation fault (core dumped)

Versions

Locally:

❯ python collect_env.py
Collecting environment information...
Traceback (most recent call last):
  File "/home/ss/Sources/unet-corresp/collect_env.py", line 602, in <module>
    main()
  File "/home/ss/Sources/unet-corresp/collect_env.py", line 585, in main
    output = get_pretty_env_info()
  File "/home/ss/Sources/unet-corresp/collect_env.py", line 580, in get_pretty_env_info
    return pretty_str(get_env_info())
  File "/home/ss/Sources/unet-corresp/collect_env.py", line 422, in get_env_info
    pip_version, pip_list_output = get_pip_packages(run_lambda)
  File "/home/ss/Sources/unet-corresp/collect_env.py", line 394, in get_pip_packages
    out = run_with_pip(sys.executable + ' -mpip')
  File "/home/ss/Sources/unet-corresp/collect_env.py", line 382, in run_with_pip
    for line in out.splitlines()
AttributeError: 'NoneType' object has no attribute 'splitlines'
❯ nvim collect_env.py
❯ python collect_env.py
Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: NixOS 23.05 (Stoat) (x86_64)
GCC version: (GCC) 12.2.0
Clang version: Could not collect
CMake version: version 3.25.1
Libc version: glibc-2.35

Python version: 3.10.10 (main, Feb  7 2023, 12:19:31) [GCC 12.2.0] (64-bit runtime)
Python platform: Linux-5.15.93-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 525.89.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          24
On-line CPU(s) list:             0-23
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen 9 5900X 12-Core Processor
CPU family:                      25
Model:                           33
Thread(s) per core:              2
Core(s) per socket:              12
Socket(s):                       1
Stepping:                        0
Frequency boost:                 enabled
CPU(s) scaling MHz:              75%
CPU max MHz:                     5619.1401
CPU min MHz:                     2200.0000
BogoMIPS:                        8399.46
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization:                  AMD-V
L1d cache:                       384 KiB (12 instances)
L1i cache:                       384 KiB (12 instances)
L2 cache:                        6 MiB (12 instances)
L3 cache:                        64 MiB (2 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-23
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] No relevant packages
[conda] Could not collect

On cluster:

$ python collect_env.py

Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:23:14) [GCC 10.4.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.83.1.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB
Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    1
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz
Stepping:              4
CPU MHz:               3649.804
CPU max MHz:           3700.0000
CPU min MHz:           1200.0000
BogoMIPS:              6400.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              25344K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.5
[pip3] numpydoc==1.5.0
[pip3] pytorch-lightning==1.9.2
[pip3] torch==1.13.1
[pip3] torch-cluster==1.6.0
[pip3] torch-geometric==2.2.0
[pip3] torch-scatter==2.1.0
[pip3] torch-sparse==0.6.16
[pip3] torchaudio==0.13.1
[pip3] torchmetrics==0.11.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl    conda-forge
[conda] cudatoolkit               11.7.0              hd8887f6_10    nvidia
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] mkl                       2022.2.1         h84fe81f_16997    conda-forge
[conda] numpy                     1.23.5          py310h53a5b5f_0    conda-forge
[conda] numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge
[conda] pyg                       2.2.0           py310_torch_1.13.0_cu117    pyg
[conda] pytorch                   1.13.1          py3.10_cuda11.7_cudnn8.5.0_0    pytorch
[conda] pytorch-cluster           1.6.0           py310_torch_1.13.0_cu117    pyg
[conda] pytorch-cuda              11.7                 h67b0de4_1    pytorch
[conda] pytorch-lightning         1.9.2              pyhd8ed1ab_0    conda-forge
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-scatter           2.1.0           py310_torch_1.13.0_cu117    pyg
[conda] pytorch-sparse            0.6.16          py310_torch_1.13.0_cu117    pyg
[conda] torchaudio                0.13.1              py310_cu117    pytorch
[conda] torchmetrics              0.11.1             pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.14.1              py310_cu117    pytorch

[malfet]

Hmm, I can reproduce it on latest trunk and in 2.0 release candidate. @NicolasHug shouldn't this be considered a release blocking? Though, this is not a regression, it crashed the same way in 1.13.1

Backtrace:

(lldb) bt
* thread #1, name = 'python', stop reason = signal SIGSEGV: address access protected (fault address: 0x7fff03200000)
  * frame #0: 0x00007fff3117d6b1 image.so`vision::image::decode_image(at::Tensor const&, long) + 97
    frame #1: 0x00007fff3118e7eb image.so`c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, long> >, true>::call(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*) + 59
    frame #2: 0x00007fff933e9b82 libtorch_cpu.so`c10::Dispatcher::callBoxed(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocator<c10::IValue> >*) const + 546
    frame #3: 0x00007fffa0765b63 libtorch_python.so`torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptr<torch::jit::Operator>, std::allocator<std::shared_ptr<torch::jit::Operator> > > const&, pybind11::args, pybind11::kwargs const&, c10::optional<c10::DispatchKey>) + 1155
    frame #4: 0x00007fffa0766452 libtorch_python.so`torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptr<torch::jit::Operator>, std::allocator<std::shared_ptr<torch::jit::Operator> > > const&, c10::Symbol, pybind11::args, pybind11::kwargs const&, bool, c10::optional<c10::DispatchKey>) + 1442
    frame #5: 0x00007fffa0658d22 libtorch_python.so`void pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::'lambda185'(std::string const&)::operator()(std::string const&) const::'lambda'(pybind11::args, pybind11::kwargs), pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::'lambda185'(std::string const&)::operator()(std::string const&) const::'lambda'(pybind11::args, pybind11::kwargs)&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) + 322
    frame #6: 0x00007fffa028d265 libtorch_python.so`pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 1557

[NicolasHug]

@malfet @SomeoneSerge I don't think decode_image was ever intended to work on CUDA tensors.

[NicolasHug]

The only GPU-decoding capabilities we have is decode_jpeg(input_tensor, device="cuda") but even then input_tensor needs to be a CPU tensor (and the GPU decoding is still Beta as it's known to have issues / leaks).

[malfet]

@malfet @SomeoneSerge I don't think decode_image was ever intended to work on CUDA tensors.

Yeah, just checked at the code. But crash is bad, at the very least we should return a structured error if Tensor is on GPU. Will propose a PR in a few

[SomeoneSerge]

I see. I assumed it was a wrapper around libnvjpeg. A graceful failure will do nicely, thanks!