Several torch_inductor CI test failures in arm neoverse-v1 relating to argmin/argmax

[robert-hardwick]

🐛 Describe the bug

There are a number of unit test failures happening on Arm® Neoverse™-V1 platform. These include, but are not limited to

test_argmax_argmin_with_nan_cpu, test_argmax_argmin_with_duplicates_cpu, test_argmax_argmin3_cpu, test_argmax_argmin_with_nan_cpu, test_argmax_argmin2_cpu

AssertionError: Tensor-likes are not equal!
Mismatched elements: 72 / 144 (50.0%)
Greatest absolute difference: 142 at index (28,)
Greatest relative difference: 1.0 at index (4,)
The failure occurred for item [0]
To execute this test, run the following from the base repo dir:
    python test/inductor/test_torchinductor.py CpuTests.test_argmax_argmin2_cpu
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Stacktrace
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 11425, in new_test
    return value(self)
  File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 8737, in test_argmax_argmin2
    self.common(fn, (torch.randn([144, 144]),))
  File "/var/lib/jenkins/workspace/test/inductor/test_torchinductor.py", line 471, in check_model
    self.assertEqual(
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3889, in assertEqual
    raise error_metas.pop()[0].to_error(
AssertionError: Tensor-likes are not equal!
Mismatched elements: 72 / 144 (50.0%)
Greatest absolute difference: 142 at index (28,)
Greatest relative difference: 1.0 at index (4,)
The failure occurred for item [0]
To execute this test, run the following from the base repo dir:
    python test/inductor/test_torchinductor.py CpuTests.test_argmax_argmin2_cpu

analysis

I have already debugged this and determined that this is due to a bug in GCC
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=117001

This bug in GCC causes incorrect output for argmin_vec and argmax_vec due to loop vectorization bug in Vectorized::blendv.

I will attach fix in a PR

Versions

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (aarch64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.5
Libc version: glibc-2.35

Python version: 3.10.15 | packaged by conda-forge | (main, Sep 30 2024, 17:46:42) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-1015-aws-aarch64-with-glibc2.35
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: ARM
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 48
Socket(s): -
Cluster(s): 1
Stepping: r1p1
BogoMIPS: 2100.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
L1d cache: 3 MiB (48 instances)
L1i cache: 3 MiB (48 instances)
L2 cache: 48 MiB (48 instances)
L3 cache: 32 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] mypy==1.11.2
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.2
[pip3] onnx==1.16.1
[pip3] onnxscript==0.1.0.dev20240817
[pip3] optree==0.13.0
[pip3] torch==2.6.0a0+git4cf6de2
[conda] No relevant packages

cc @malfet @snadampal @milpuz01

[malfet]

@robert-hardwick do you know if this is a regression (i.e. it used to work in 2.3 but is broken now or it was always broken?)
[Edit] Looks like it is a regression (i.e. it used to work in 2.4) but is broken in 2.5, which added support for SVE instructions
In that case, should we disable SVE support in 2.5 (and at the very least we should mention it in the release notes)

[robert-hardwick]

No it seems to be like it has always been broken as i've identified the problem code to be in

pytorch/aten/src/ATen/cpu/vec/vec_base.h

Lines 212 to 226 in b71d0ac

	static Vectorized<T> blendv(const Vectorized<T>& a, const Vectorized<T>& b,
	const Vectorized<T>& mask) {
	Vectorized vector;
	int_same_size_t<T> buffer[size()];
	mask.store(buffer);
	for (const auto i : c10::irange(size())) {
	if (buffer[i] & 0x01)
	{
	vector[i] = b[i];
	} else {
	vector[i] = a[i];
	}
	}
	return vector;
	}

GCC auto-vectorizer which is enabled with -O3 causes this blendv function to output incorrect numbers. See the reproducer in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=117001 which is the same functionality as blendv.

I have a fix for this ready, just getting it uploaded into a PR.

[robert-hardwick]

I've attached PR that should fix this. But there is follow on topic that is ciflow/linux-aarch64 is not covering SVE ( or BF16 for that matter ).

I wanted to raise this as a separate issue, however it's relevant here because you cannot see that this fixes anything

pytorch/.github/workflows/linux-aarch64.yml

Line 35 in 4d45536

{ config: "default", shard: 1, num_shards: 4, runner: "linux.arm64.2xlarge" },

Am i right to think that the linux.arm64.2xlarge runners which are used for linux-aarch64 are infact Graviton2 based t4g instances? Which doesn't have SVE or BF16 support.

https://github.com/pytorch/test-infra/blob/3a3aa50b69a1dac8be255135eb6c5ef510b92f14/.github/scale-config.yml#L171

I could create a separate issue for that discussion if you like?

[robert-hardwick]

I will check this on 2.4 to confirm what I have just said

[robert-hardwick]

$:~/pytorch-2.4.1/test$ pip install torch==2.4.1
Installing collected packages: torch
Successfully installed torch-2.4.1
$:~/pytorch-2.4.1/test$ cd ../
$:~/pytorch-2.4.1$ python test/inductor/test_torchinductor.py CpuTests.test_argmax_argmin2_cpu
inline_call []
stats [('calls_captured', 4), ('unique_graphs', 1)]
inductor [('fxgraph_cache_miss', 1)]
aot_autograd [('total', 1), ('ok', 1)]
.
----------------------------------------------------------------------
Ran 1 test in 2.915s

OK

I stand corrected, this does look like a regression.

[robert-hardwick]

Confirming that this is a regression in 2.5.0. I think if we change runner of linux-aarch64 workflow to a graviton 3 instance you will see these errors.

$/pytorch2.5$ pip install torch==2.5.0 torchvision --index-url https://download.pytorch.org/whl/test/cpu
$/pytorch2.5$ python test/inductor/test_torchinductor.py CpuTests.test_argmax_argmin2_cpu
inductor [('fxgraph_cache_miss', 1)]
inline_call []
stats [('calls_captured', 4), ('unique_graphs', 1)]
aot_autograd [('total', 1), ('ok', 1)]
F
======================================================================
FAIL: test_argmax_argmin2_cpu (__main__.CpuTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "venv/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2979, in wrapper
    method(*args, **kwargs)
  File "pytorch2.5/test/inductor/test_torchinductor.py", line 11410, in new_test
    return value(self)
  File "pytorch2.5/test/inductor/test_torchinductor.py", line 8724, in test_argmax_argmin2
    self.common(fn, (torch.randn([144, 144]),))
  File "pytorch2.5/test/inductor/test_torchinductor.py", line 471, in check_model
    self.assertEqual(
  File "pytorch/venv/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3885, in assertEqual
    raise error_metas.pop()[0].to_error(
AssertionError: Tensor-likes are not equal!

Mismatched elements: 72 / 144 (50.0%)
Greatest absolute difference: 142 at index (28,)
Greatest relative difference: 1.0 at index (4,)

The failure occurred for item [0]

To execute this test, run the following from the base repo dir:
    python test/inductor/test_torchinductor.py CpuTests.test_argmax_argmin2_cpu

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
Ran 1 test in 4.788s

FAILED (failures=1)

[robert-hardwick]

It works in 2.4 because #131016 added vectorization support in 2.5 for argmin/argmax which introduced the regression.

But i'm confident that #137606 will address the issue. ( I have confirmed this running all aarch64 unit tests locally on neoverse-v1 )