Classes of torchvision\ops

[xhuan8]

Is there any plan to add classes of torchvision/ops?

roi_align and boxes are required, they are not implemented in TorchSharp

return torch.ops.torchvision.roi_align(
        input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
    )

[GeorgeS2019]

TORCHVISION.OPS

suggestion: Need to approach this ALSO from the perspective of Onnx contributed Ops
e.g. com.microsoft.vision

flowchart TD
    image-->|com.microsoft.vision\nops| TorchSharp\nmodel-->|com.microsoft.vision\nops| output\nimage

Loading

[GeorgeS2019]

@kaiidams

Why there are no Torchaudio.ops?

Are they in here?
https://github.com/pytorch/audio/tree/main/torchaudio/csrc

e.g. com.microsoft.audio

flowchart TD
    audio-->|com.microsoft.audio\nops| TorchSharp\nmodel-->|com.microsoft.audio\nops| output\naudio

Loading

[GeorgeS2019]

Most of the Torchtext.ops are here
https://github.com/pytorch/text/tree/main/torchtext/csrc

com.microsoft.nlp

flowchart TD
    text-->|com.microsoft.nlp\nops| TorchSharp\nmodel-->|com.microsoft.nlp\nops| output\ntext

Loading

[kaiidams]

@GeorgeS2019

Probably ONNX runtime approach is not related to this.

Why there are no Torchaudio.ops?

torchaudio has C code that uses Kaldi and Sox, FFMpeg, which is not implemented in TorchSharp. IMHO, Modern models don't depend on these except I/O, but still useful in some cases. torchaudio doesn't call them ops.

@NiklasGustafsson I don't know about what happened to torchvision.ops, if they are missing features, do you have plan to add them?

[NiklasGustafsson]

There's a small number of ops in torchvision.ops. Feel free to contribute more, if you have time. It would be good to have an issue to track them. I'll open something up.

[NiklasGustafsson]

BTW, it seems like FFMPEG is disabled by default in PyTorch, now:

https://pytorch.org/vision/stable/#torchvision.set_video_backend

You have to build from source to enable it, apparently.

[xhuan8]

seems there is no prebuild package for torchvision, what is the best way to add it after build from source?

[NiklasGustafsson]

seems there is no prebuild package for torchvision, what is the best way to add it after build from source?

TorchVision is now available on NuGet: https://www.nuget.org/packages/TorchVision, starting with version 0.98.1, when it was split out from TorchSharp.

[xhuan8]

I try to call the native method from torchvision, torch.ops.torchvision.nms
https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/nms.cpp
https://github.com/pytorch/vision/blob/main/torchvision/ops/_register_onnx_ops.py

how to do it in TorchSharp?

[NiklasGustafsson]

I believe someone already implemented nms in C#.

TorchSharp/src/TorchVision/Ops.cs

Line 48 in 2f89e58

public static Tensor nms(Tensor boxes, Tensor scores, double iou_threshold = 0.5)

[NiklasGustafsson]

To answer your question, though -- I haven't understood where the native code binaries and header files for TV are to be found. If you figure that out, we should bundle it in with the other native code we already have.

[xhuan8]

They are under https://github.com/pytorch/vision/tree/main/torchvision/csrc/ops, contains cpu and cuda implementations.
We don't need the IO operations, so FFMPEG and JPEG are not needed.
Currently I'll try build from souce and use it on Windows.

[NiklasGustafsson]

Have you found where pre-built binaries are available for download?

[xhuan8]

no, only build from source.

[NiklasGustafsson]

Note that the current libtorch binaries are 1.11, with CUDA 11.3 -- that will matter for building compatible TV binaries, I believe.