Optimization failing on AWS Graviton2 (neoverse-n1) machine

[IanButterworth]

OpenBLAS matrix multiplication optimization on an AWS EC2 ARM graviton2 (neoverse-n1) system with the following julia setup seems to be failing:

• OpenBLAS 0.3.9 (present on Julia master and v1.5-rc1)
• LLVM 10 (PR upgrade LLVM to 10 JuliaLang/julia#35318) (may be irrelevant)
• Updated ARM cpu detection (PR Update ARM feature and CPU detection (supersedes #36464) JuliaLang/julia#36485) (may be irrelevant)

julia> versioninfo()
Julia Version 1.6.0-DEV.341
Commit 8367e441ac* (2020-07-01 18:30 UTC)
Platform Info:
  OS: Linux (aarch64-linux-gnu)
  CPU: unknown
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-10.0.0 (ORCJIT, neoverse-n1)
Environment:
  JULIA_NUM_THREADS = 16

julia> LinearAlgebra.BLAS.openblas_get_config()
"OpenBLAS 0.3.9 NO_AFFINITY ARMV8 MAX_THREADS=32"

julia> using BenchmarkTools
julia> @btime x * x setup=(x=rand(Float32, 100, 100));
  21.123 ms (2 allocations: 39.14 KiB)

compared to a mac:

julia> @btime x * x setup=(x=rand(Float32, 100, 100));
18.161 μs (2 allocations: 39.14 KiB)

[martin-frbg]

This looks to me as if the cpu was not identified, leading to a generic ARMV8 build (approximately Cortex A57). Also I am not sure if the Neoverse PR was ever tested with anything other than gcc. (And I notice that Julia itself labels the CPU as "unknown")
As far as OpenBLAS is concerned, you could try if setting the environment variable OPENBLAS_CORETYPE to NEOVERSEN1 helps. (This would override the possibly incorrect autodetection in the library, assuming that the Julia binary of OpenBLAS is/was built with DYNAMIC_ARCH=1 to support multiple cpu types. What is clear from the openblas_get_config output is that OpenBLAS was compiled for a maximum number of 32 threads, so would not be able to make full use of a 64-core hardware).
But in any case the current implementation of NeoverseN1 support is basically a mix of the pre-existing ARMV8 and ThunderX2 targets so may not have optimum performance yet.

[IanButterworth]

Great. Setting OPENBLAS_CORETYPE=NEOVERSEN1 seems to enable optimization, but falls a bit short of the mac intel result, as you predicted

julia> @btime x * x setup=(x=rand(Float32, 100, 100));
  52.135 μs (2 allocations: 39.14 KiB)

[martin-frbg]

Can you get me the output of /proc/cpuinfo please ? Even if the Graviton2 is based on Neoverse, I suspect it will have a different implementer and/or part code, hence the autodetection failure

[IanButterworth]

Happily:

cat /proc/cpuinfo

$ cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 1
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 2
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 3
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 4
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 5
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 6
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 7
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 8
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 9
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 10
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 11
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 12
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 13
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 14
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

processor	: 15
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

[martin-frbg]

Strange, this shows the manufacturer and model codes of a genuine Neoverse-N1 just as they are already present in dynamic_arm64.c (and cpuid_arm64.c). Right now I do not see why autodetection would fail (unless there are actually different hardwares providing this EC2 instance)

[IanButterworth]

This instance wasn’t one of the “metal” instances. I.e. the core count is restricted, I assume through some level of virtualization. Could that be the issue? Typically OpenBLAS via julia works on non-metal instances like this on x86 though.

[yuyichao]

It's probably not the reason but note that CPUID should not be used to identify the CPU on ARM. This is because with big-LITTLE and alike the CPUID result for MIDR_EL1 is unstable and it depends on the core the code is running on. You can easily get the wrong (/unwanted) result if you happen to run on a little core.

The correct way is to read the /sys/devices/system/cpu/cpu<n>/regs/identification/midr_el1 file.

[yuyichao]

And I said this is not the reason here because the result given to me at JuliaLang/julia#36464 (comment) suggests that the instruction does return the correct value in this case, and the cpuinfo suggests that all the cores are identical.

[martin-frbg]

Actually the runtime detection is using an mrs instruction to get MIDR_EL1 (and should complain with a clear message if that fails), only the cpuid_arm64 invoked for the compile-time detection still reads from /proc/cpuinfo. Do you see anything logged like either Kernel lacks cpuid feature support. Auto detection of core type failed !!! or Falling back to generic ARMV8 core ?

[IanButterworth]

On my mac, with julia 1.4.2 & OpenBLAS 0.3.5, starting julia with OPENBLAS_VERBOSE=3 gives:

$ OPENBLAS_VERBOSE=3 julia
Core: Haswell

but nothing is printed if I do the same on the graviton2 system. No warning, and no Core: line

[IanButterworth]

Perhaps this is indicative.. I just got a fast case, without having set OPENBLAS_CORETYPE=NEOVERSEN1

$ ./julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.341 (2020-07-01)
 _/ |\__'_|_|_|\__'_|  |  vc/upgrade_llvm_10/8367e441ac* (fork: 1 commits, 1 day)
|__/                   |
julia> using BenchmarkTools
julia> @btime x * x setup=(x=rand(Float32, 100, 100));
  29.982 μs (2 allocations: 39.14 KiB)

I tried a few more julia sessions and it went back to slow speeds.

[yuyichao]

Do you ever get unstable/different results if you run the c program I asked you to run if you run it multiple times?

[IanButterworth]

I just ran it ~40 times and it didn't change

$ ./mrsemu
ID_AA64ISAR0_EL1    : 0x0000100010211120
ID_AA64ISAR1_EL1    : 0x0000000000100001
ID_AA64MMFR0_EL1    : 0x00000000ff000000
ID_AA64MMFR1_EL1    : 0x0000000000000000
ID_AA64PFR0_EL1     : 0x0000000000110011
ID_AA64PFR1_EL1     : 0x0000000000000020
ID_AA64DFR0_EL1     : 0x0000000000000006
ID_AA64DFR1_EL1     : 0x0000000000000000
MIDR_EL1            : 0x00000000413fd0c1
MPIDR_EL1           : 0x0000000080000000
REVIDR_EL1          : 0x0000000000000000

[martin-frbg]

Very strange. The code for printing the "Core:" information in verbose mode is identical between the arm64 and x86 versions of DYNAMIC_ARCH support, and I do not see how it can escape printing something.
Occasional fast cases without the CORETYPE variable could be in support of my theory that there are two types of hardware
providing your EC2 instance - one with a genuine Neoverse N1 and a "compatible" that has an unfamiliar identifier.

[martin-frbg]

Does julia filter the info that OpenBLAS tries to print ? (Unlikely, but just asking for completeness). I just did a DYNAMIC_ARCH build on my phone (in termux), and there the library complains that my kernel does not support the cpuid feature, before announcing a falback to generic armv8 and printing "Core: armv8" as it should.