`exact_dtype=True` is ignored when comparing `torch.Tensor`'s and `np.ndarray`'s

[pmeier]

🐛 Bug

To Reproduce

import torch
import numpy as np

from torch.testing._internal.common_utils import TestCase


class TestFoo(TestCase):
    def test_bar(self):
        x = torch.ones((1,), dtype=torch.float16)
        y = np.ones((1,), dtype=np.float32)

        self.assertEqual(x, y, exact_dtype=True)

Expected behavior

Test should fail, but passes.

Environment

PyTorch version: 1.9.0a0+gitda7bfa1
Is debug build: True
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (crosstool-NG 1.24.0.133_b0863d8_dirty) 9.3.0
Clang version: Could not collect
CMake version: version 3.20.2
Libc version: glibc-2.9

Python version: 3.6 (64-bit runtime)
Python platform: Linux-5.12.5-arch1-1-x86_64-with-arch
Is CUDA available: False
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080
Nvidia driver version: 465.31
cuDNN version: Probably one of the following:
/usr/local/cudnn-8.1.1-cuda-11/lib64/libcudnn.so.8.1.1
/usr/local/cudnn-8.1.1-cuda-11/lib64/libcudnn_adv_infer.so.8.1.1
/usr/local/cudnn-8.1.1-cuda-11/lib64/libcudnn_adv_train.so.8.1.1
/usr/local/cudnn-8.1.1-cuda-11/lib64/libcudnn_cnn_infer.so.8.1.1
/usr/local/cudnn-8.1.1-cuda-11/lib64/libcudnn_cnn_train.so.8.1.1
/usr/local/cudnn-8.1.1-cuda-11/lib64/libcudnn_ops_infer.so.8.1.1
/usr/local/cudnn-8.1.1-cuda-11/lib64/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] pytest-pytorch==0.0.0
[pip3] torch==1.9.0a0+gitda7bfa1
[conda] magma-cuda111             2.5.2                         1    pytorch
[conda] mkl                       2021.2.0           h726a3e6_389    conda-forge
[conda] mkl-include               2021.2.0           h726a3e6_389    conda-forge
[conda] numpy                     1.19.5           py36h2aa4a07_1    conda-forge
[conda] pytest-pytorch            0.2.0              pyh44b312d_0    conda-forge
[conda] torch                     1.9.0a0+gitda7bfa1           dev_0    <develop>

cc @mruberry @rgommers @heitorschueroff

[pmeier]

#48217 introduced special cases for numpy comparisons

pytorch/torch/testing/_internal/common_utils.py

Lines 1401 to 1412 in 0e9a295

	# Tensor x Numpy array
	elif isinstance(x, torch.Tensor) and isinstance(y, np.ndarray):
	self.assertEqual(x, torch.from_numpy(y), atol=atol, rtol=rtol, msg=msg,
	exact_dtype=exact_dtype, exact_device=exact_device)
	# Numpy array x Tensor
	elif isinstance(x, np.ndarray) and isinstance(y, torch.Tensor):
	self.assertEqual(torch.from_numpy(x), y, atol=atol, rtol=rtol, msg=msg,
	exact_dtype=exact_dtype, exact_device=exact_device)
	# Numpy array x Numpy array
	elif isinstance(x, np.ndarray) and isinstance(y, np.ndarray):
	self.assertEqual(torch.from_numpy(x), torch.from_numpy(y), atol=atol, rtol=rtol, msg=msg,
	exact_dtype=exact_dtype, exact_device=exact_device)

, but they will never be hit for non-scalar arrays and tensors. In such a case the iterable case is triggered

pytorch/torch/testing/_internal/common_utils.py

Lines 1383 to 1389 in 0e9a295

	elif is_iterable(x) and is_iterable(y):
	debug_msg = ("Attempted to compare the lengths of [iterable] types: "
	f"Expected: {len(x)}; Actual: {len(y)}.")
	super().assertEqual(len(x), len(y), msg=self._get_assert_msg(msg, debug_msg=debug_msg))
	for x_, y_ in zip(x, y):
	self.assertEqual(x_, y_, atol=atol, rtol=rtol, msg=msg,
	exact_dtype=exact_dtype, exact_device=exact_device)

which in turn checks every element individually instead of the whole array or tensor. Reduced to a single element, now

pytorch/torch/testing/_internal/common_utils.py

Lines 1264 to 1270 in 0e9a295

	# Tensor x Number and Number x Tensor comparisons
	if isinstance(x, torch.Tensor) and isinstance(y, Number):
	self.assertEqual(x.item(), y, atol=atol, rtol=rtol, msg=msg,
	exact_dtype=exact_dtype, exact_device=exact_device)
	elif isinstance(y, torch.Tensor) and isinstance(x, Number):
	self.assertEqual(x, y.item(), atol=atol, rtol=rtol, msg=msg,
	exact_dtype=exact_dtype, exact_device=exact_device)

is triggered, since

assert isinstance(next(iter(np.array((1.0,)))), numbers.Number)

Finally, this triggers

pytorch/torch/testing/_internal/common_utils.py

Lines 1393 to 1400 in 0e9a295

	# Scalar x Scalar
	elif isinstance(x, Number) and isinstance(y, Number):
	result, debug_msg_scalars = self._compareScalars(x, y, rtol=rtol, atol=atol,
	equal_nan=equal_nan)
	if not result:
	assert debug_msg_scalars is not None
	debug_msg = "Scalars failed to compare as equal! " + debug_msg_scalars
	super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))

which performs no dtype check anymore.

[mruberry]

Great catch, @pmeier. Looking forward to getting this fixed!