RuntimeError: No such operator torchvision::nms

[ppwwyyxx]

Similar to #1405, but I'm seeing this issue in the nightly builds.

What I did:

I download two nightly wheels from https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html, and installed them with:

pip install torch-1.4.0.dev20191017+cu100-cp37-cp37m-linux_x86_64.whl torchvision-0.5.0.dev20191017+cu100-cp37-cp37m-linux_x86_64.whl 

Then, I recompile detectron2; Running a simple detectron2 evaluation raises this error:

python tools/train_net.py --config-file configs/LVIS-InstanceSegmentation/mask_rcnn_R_101_FPN_1x.yaml --num-gpus 2 --eval-only DATALOADER.NUM_WORKERS 1 MODEL.WEIGHTS ~/data/D2models/LVIS-InstanceSegmentation/mask_rcnn_R_101_FPN_1x/144219035/model_final_824ab5.pkl

an example stack trace:

Traceback (most recent call last):
  File "tools/train_net.py", line 162, in <module>
    args=(args,),
  File "/private/home/yuxinwu/detectron2/detectron2/engine/launch.py", line 52, in launch
    main_func(*args)
  File "tools/train_net.py", line 133, in main
    res = Trainer.test(cfg, model)
  File "/private/home/yuxinwu/detectron2/detectron2/engine/defaults.py", line 413, in test
    results_i = inference_on_dataset(model, data_loader, evaluator)
  File "/private/home/yuxinwu/detectron2/detectron2/evaluation/evaluator.py", line 118, in inference_on_dataset
    outputs = model(inputs)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 66, in forward
    return self.inference(batched_inputs)
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 119, in inference
    proposals, _ = self.proposal_generator(images, features, None)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/proposal_generator/rpn.py", line 179, in forward
    self.training,
  File "/private/home/yuxinwu/detectron2/detectron2/modeling/proposal_generator/rpn_outputs.py", line 136, in find_top_rpn_proposals
    keep = batched_nms(boxes.tensor, scores_per_img, lvl, nms_thresh)
  File "/private/home/yuxinwu/detectron2/detectron2/layers/nms.py", line 17, in batched_nms
    return box_ops.batched_nms(boxes, scores, idxs, iou_threshold)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 70, in batched_nms
    keep = nms(boxes_for_nms, scores, iou_threshold)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 31, in nms
    return torch.ops.torchvision.nms(boxes, scores, iou_threshold)
  File "/private/home/yuxinwu/miniconda3/lib/python3.7/site-packages/torch/_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms

another stack trace: facebookresearch/detectron2#94 (comment)

[fmassa]

Thanks for the detailed report!

I think the issue is that since #1467 we now build torchvision nightlies against PyTorch 1.3, and not PyTorch master.

The reason why I changed it to PyTorch 1.3 was because I was planning on cutting a new release of torchvision in the next few days, so it should be cut against PyTorch 1.3 and we needed to test that CI is working with PyTorch 1.3

But I think there is no need that the nightlies to be built based on PyTorch 1.3

[tangorn]

building from source, torchvision latest master, pytorch from tag v1.3.0, see every test in test_ops.py fail with the similar error... Here is the tail of the log, the rest is similar

ERROR: test_roi_pool_gradient_cuda (__main__.RoIPoolTester)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_ops.py", line 249, in test_roi_pool_gradient_cuda
    y = layer(x, rois)
  File "/home/tangorn/src/pytorch/build/install/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tangorn/src/pytorch/build/install/lib64/python3.6/site-packages/torchvision-0.5.0a0+97b53f9-py3.6-linux-x86_64.egg/torchvision/ops/roi_pool.py", line 49, in forward
    return roi_pool(input, rois, self.output_size, self.spatial_scale)
  File "/home/tangorn/src/pytorch/build/install/lib64/python3.6/site-packages/torchvision-0.5.0a0+97b53f9-py3.6-linux-x86_64.egg/torchvision/ops/roi_pool.py", line 34, in roi_pool
    output, _ = torch.ops.torchvision.roi_pool(input, rois, spatial_scale,
  File "/home/tangorn/src/pytorch/build/install/lib64/python3.6/site-packages/torch/_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::roi_pool

----------------------------------------------------------------------
Ran 36 tests in 0.999s

FAILED (errors=36)

might be not related to the original issue, I am pretty new to pytorch, could have messed up something in my installation.

[fmassa]

Hi @tangorn

I think there was a problem with your compilation of torchvision.
Can you do a clean install (removing all previously installed version of torchvision that you had), doing

python setup.py clean
python setup.py install

and report back? This should fix your issues

[fmassa]

Also, @tangorn another option is to use PyTorch nightly and torchvision nightly, as we provide pre-built binaries for both.

[tangorn]

did full clean rebuild of both pytorch and vision. Same issue. Other tests like test_transforms etc run fine. test_models fails only for models that use nms op. I have a feeling that the ops somehow do not register properly. Working from binary is not an attractive option for me as i am planning to add a few ops of my own...

[tangorn]

rolling back to your 0.4.0 branch fixed it.

[fmassa]

@tangorn can you paste the build logs from TorchVision? I'm fairly confident that there has been a compilation issue somewhere, just need to find out where, and I'd love to get this fixed with master.

[tangorn]

Here you go:
torchvision-0.5.0-build.log
using python3 setup.py install, the beginning got cut due to excessive warnings, let me know if you need it too

[fmassa]

@tangorn thanks for the log.

One more question: did you use the same PyTorch install to both compile torchvision and to run it?
If you could go to the torchvision folder, and do

torch.ops.load_library('_C.so')

and print the error message, it will be very helpful as well.
There is probably an incompatibility between the PyTorch version used to compile torchvision and the one you are using during runtime

[tangorn]

OSError: /home/tangorn/src/torchvision-my/_C.so: cannot open shared object file: No such file or directory
which is understandable as

tangorn@linux-429s:~/src/torchvision-my> locate _C.so
/home/tangorn/src/torchvision-my/build/lib.linux-x86_64-3.6/torchvision/_C.so
tangorn@linux-429s:~/src/torchvision-my> 

it is built but not installed in the pytorch tree. But even copying it there by hand does not seem to work:

tangorn@linux-429s:~/src/torchvision-my> echo $LD_LIBRARY_PATH
/home/tangorn/src/pytorch/build/install/lib:/usr/local/cuda-10.0/lib64
tangorn@linux-429s:~/src/torchvision-my> /home/tangorn/src/pytorch/build/install/bin/python3 test.py
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    torch.ops.load_library('_C.so')
  File "/home/tangorn/src/pytorch/build/install/lib64/python3.6/site-packages/torch/_ops.py", line 106, in load_library
    ctypes.CDLL(path)
  File "/usr/lib64/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/tangorn/src/torchvision-my/_C.so: cannot open shared object file: No such file or directory
tangorn@linux-429s:~/src/torchvision-my> 

incompatibility I doubt as i run the test right after running setup.py install and i did not touch the pytorch since the clean rebuild yesterday

[fmassa]

@tangorn thanks for trying this out. The second error message still doesn't give us the info I was looking for, can you pass the full path to the _C.so lib?

[tangorn]

recreated the whole setup on my home box from scratch. Same result: 0.4.0 works, 0.5.0 does not. The only path I can find _C.so is in the build directory:

/home/tangorn/src/torchvision-my/build/lib.linux-x86_64-3.6/torchvision/_C.so

after running the test with 0.4.0 i can also see one in cache:

/home/tangorn/.cache/Python-Eggs/torchvision-0.4.0a0+d31eafa-py3.6-linux-x86_64.egg-tmp/torchvision/_C.so