[feature request] torch.as_tensor to support any object that NumPy's asarray or array can consume (consume __array_interface__)

[vadimkantorov]

This would enlarge the accepted set of inputs of torch.as_tensor and would support PIL images / h5py arrays. I think this feature request goes well in the theme of standardizing support for methods like __array_interface__, __cuda_array_interface__ and such. It would be good for torch.as_tensor to support accepting __array_interface__ dicts directly. It may sometimes be conventient to store / manipulate these dictionaries directly, and then pass them to torch.as_tensor. Currently it produces Could not infer dtype of dict - which is also an unclear error message by the way.

Currently, torch.as_tensor(pil_image) fails with RuntimeError: Could not infer dtype of Image, while it can be converted with np.asarray.

As side-effect, this would also eliminate the need for torchvision's F.to_tensor(pil_image)

Related: #54138

cc @mruberry @rgommers @heitorschueroff

[mruberry]

Thanks for this suggestion; I wonder if the Python Array API will also make a proposal here

[rgommers]

gh-54187 adds __array_interface__ support.

The array API standard asarray definition (the as_tensor equivalent) says to support DLPack, and also the buffer protocol as a nice convenience. There's still some discussion about that at data-apis/array-api#155. There's a tension between having well-defined functions that have a clear purpose, and a "just swallow anything that may possibly make sense".

The use of numpy.asarray through other libraries has been pretty harmful; I'd much rather see other libraries do what PyTorch does and only accepts Tensor or tensor-like objects (those could include the buffer protocol and __array_interface__/__cuda_array_interface__ though). And not sequences, generators, etc.

[vadimkantorov]

I think both are important and useful. In some parts of code it's useful to be constraining and very explicit. In other parts of code it may be useful to swallow everything without having to think, is it PIL image, or NumPy or DLPack capsule from CuPy.

If generic part is not there, users have to roll again and again their own checks of type name etc, which is more brittle and worse than tested library helper method for the same goal.

[vadimkantorov]

@cpuhrsch At pytorch/vision#6278 (comment), it seems impossible to have a torch.is_tensor check and polymorphic code accepting both tensor and TensorList. It would be cool to be able to solve it somehow. Could it be done somehow without causing GPU->CPU synchronization by letting support torch.as_tensor consuming both Tensor and TensorLIst (doing then a torch.stack internally)?

Or maybe at least torch.stack(x) should not do anything (at most a copy of input) if a tensor is provided as input and not a list?