RuntimeError: No such operator image::read_file on Docker

[moskomule]

🐛 Bug

In a Docker environment, torch.io.read_file causes RuntimeError: No such operator image::read_file.

To Reproduce

1. Setup a container

I used the following Docker file.

FROM nvidia/cuda:11.0-base-ubuntu20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt update -qq \
    && apt install -y -qq \
    apt-utils \
    bzip2 \
    build-essential \
    cmake \
    curl \
    git \
    libncurses5-dev \
    libncursesw5-dev \
    libboost-all-dev \
    locales \
    nasm \
    ruby \
    sudo \
    swig \
    unzip \
    wget \
    zsh \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

ENV CONDADIR ${HOME}/.miniconda
RUN cd ${HOME} \
    && wget -q https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh \
    && bash miniconda.sh -b -p ${CONDADIR} \
    && rm miniconda.sh

docker build -t test .
docker run --rm -it test
# in the launched container
export PATH=.miniconda/bin:$PATH
conda create -n test python=3.9 -y
source activate test
conda install -c pytorch -c nvidia -c conda-forge pytorch torchvision cudatoolkit=11.1 numpy -y # another issue, but numpy is not automatically installed as a dependency

Run the following

import torchvision
torchvision.io.read_file(".")
# RuntimeError: No such operator image::read_file

Expected behavior

Environment

collect_env.py collects the following information (Docker v20.10.7).

Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.5 (default, Jun  4 2021, 12:28:51)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-77-generic-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.0
[pip3] torch==1.9.0
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0                     9_mkl    conda-forge
[conda] libcblas                  3.9.0                     9_mkl    conda-forge
[conda] liblapack                 3.9.0                     9_mkl    conda-forge
[conda] mkl                       2021.2.0           h06a4308_296
[conda] numpy                     1.21.0           py39hdbf815f_0    conda-forge
[conda] pytorch                   1.9.0           py3.9_cuda11.1_cudnn8.0.5_0    pytorch
[conda] torchvision               0.10.0               py39_cu111    pytorch

Additional context

I could not reproduce this issue in a non-Docker environment. A nightly build resolved this issue.

[andfoy]

Hi @moskomule, thanks for reaching out, what happens if you run the python interpreter with LD_DEBUG=all python?

[moskomule]

Hi @andfoy!

LD_DEBUG=all python produces a log of 800MB, so I uploaded it to Google Drive.
https://drive.google.com/file/d/1ZYgHjxhY9iWMxjBm0tztsZQbE9e23-M0/view?usp=sharing
I hope it will help you to investigate the issue.

[h-vetinari]

Hitting this issue as well in conda-forge/allennlp-models-feedstock#12

[fmassa]

@andfoy did you have the time to investigate this issue?

[andfoy]

@fmassa, I'll reproduce the docker file on my local machine to see if there are missing dependencies, that could be the only explanation

[andfoy]

The problem is related to the jpeg version being installed, by looking at the ldd output over image.so it looks that the torchvision distribution is linking against libjpeg.so.8, whereas the conda solver is installing jpeg-9b-h024ee3a_2, which provides libjpeg.so.9

(test) root@a29d38ca2593:/.miniconda/envs/test/lib/python3.9/site-packages/torchvision# ldd image.so 
	linux-vdso.so.1 (0x00007ffd555c6000)
	libpng16.so.16 => /.miniconda/envs/test/lib/python3.9/site-packages/torchvision/./../../../libpng16.so.16 (0x00007f17902f9000)
	libjpeg.so.8 => not found
	...

(test) root@a29d38ca2593:/.miniconda/envs/test/lib/python3.9/site-packages/torchvision# ls /.miniconda/envs/test/lib/libjpeg.
libjpeg.a         libjpeg.la        libjpeg.so        libjpeg.so.9      libjpeg.so.9.2.0

This is an issue that requires a recompilation of torchvision to link against jpeg 9b and pinning this version in the recipe

vision/packaging/torchvision/meta.yaml

Line 29 in f3aff2f

- jpeg <=9b

[datumbox]

@andfoy FYI @zzk1st is about to remove the jpeg pinning. We got a working solution and it should be merged soon.