Test fail for libraries/cublas/level1

[mind6]

Describe the bug

A test failure on testf(BLAS.asum, rand(T, m)).

• On my system I only get this single error with jobs=1.
• With jobs=4 I get this error plus perhaps another error.
• With jobs=16 I immediately get multiple errors over multiple tests.

To reproduce

From a dev directory with tag v5.8.2 checked out:

K:\.julia\dev\CUDA>julia --project test\runtests.jl --jobs=1 libraries/cublas/level1

Manifest.toml

I'm using the CUDA.jl manifest.

Expected behavior

Test passing.

Julia versioninfo()

Julia Version 1.10.10
Commit 95f30e51f4 (2025-06-27 09:51 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 32 × AMD Ryzen 9 9950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, generic)
Threads: 32 default, 0 interactive, 16 GC (on 32 virtual cores)
Environment:
  JULIA_CONDAPKG_BACKEND = Null
  JULIA_DEPOT_PATH = K:\.julia
  JULIA_NUM_THREADS = auto
  JULIA_PYTHONCALL_EXE = K:\Miniconda3\envs\py310\python.exe

CUDA Version info and output

┌ Info: System information:
│ CUDA runtime 12.9, artifact installation
│ CUDA driver 12.9
│ NVIDIA driver 576.80.0
│ 
│ CUDA libraries:
│ - CUBLAS: 12.9.1
│ - CURAND: 10.3.10
│ - CUFFT: 11.4.1
│ - CUSOLVER: 11.7.5
│ - CUSPARSE: 12.5.10
│ - CUPTI: 2025.2.1 (API 28.0.0)
│ - NVML: 12.0.0+576.80
│ 
│ Julia packages:
│ - CUDA: 5.8.2
│ - CUDA_Driver_jll: 0.13.1+0
│ - CUDA_Runtime_jll: 0.17.1+0
│ 
│ Toolchain:
│ - Julia: 1.10.10
│ - LLVM: 15.0.7
│ 
│ 1 device:
└   0: NVIDIA GeForce RTX 3080 (sm_86, 8.186 GiB / 10.000 GiB available)
[ Info: Testing using device 0 (NVIDIA GeForce RTX 3080). To change this, specify the `--gpu` argument to the tests, or set the `CUDA_VISIBLE_DEVICES` environment variable.
[ Info: Running 1 tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
                             |          | ---------------- GPU ---------------- | ---------------- CPU ---------------- |
Test                (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
libraries/cublas/level1  (2) |         failed at 2025-07-05T10:01:58.798
Testing finished in 20 seconds, 547 milliseconds
Worker 2 failed running test libraries/cublas/level1:
Some tests did not pass: 131 passed, 1 failed, 0 errored, 0 broken.
libraries/cublas/level1: Test Failed at K:\.julia\dev\CUDA\test\libraries\cublas\level1.jl:28
  Expression: testf(BLAS.asum, rand(T, m))

Stacktrace:
 [1] record(ts::Test.DefaultTestSet, t::Union{Test.Error, Test.Fail}; print_result::Bool)
   @ Test K:\.julia\juliaup\julia-1.10.10+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:1085
 [2] record(ts::Test.DefaultTestSet, t::Union{Test.Error, Test.Fail})
   @ Test K:\.julia\juliaup\julia-1.10.10+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:1078
 [3] top-level scope
   @ K:\.julia\dev\CUDA\test\runtests.jl:477
 [4] include(mod::Module, _path::String)
   @ Base .\Base.jl:495
 [5] exec_options(opts::Base.JLOptions)
   @ Base .\client.jl:323
 [6] _start()
   @ Base .\client.jl:557

Test Summary:               | Pass  Fail  Total  Time
  Overall                   |  131     1    132
    libraries/cublas/level1 |  131     1    132
    FAILURE

Error in testset libraries/cublas/level1:
Test Failed at K:\.julia\dev\CUDA\test\libraries\cublas\level1.jl:28
  Expression: testf(BLAS.asum, rand(T, m))

ERROR: LoadError: Test run finished with errors
in expression starting at K:\.julia\dev\CUDA\test\runtests.jl:508

I believe I'm experiencing a similar issue as #2223

[maleadt]

I wasn't able to reproduce the bug in the linked issue. Can you try and reduce this to a specific invocation that fails?

[mind6]

I wasn't able to reproduce the bug in the linked issue. Can you try and reduce this to a specific invocation that fails?

do you mean somehow run testf(BLAS.asum, rand(T, m)) by itself? I'm not quite sure how to invoke just that. But it always fails for me on the same line when I do julia --project test\runtests.jl --jobs=1 libraries/cublas/level1.

I'm on Windows with AMD 9950X. I notice he was on Linux with AMD 7950X.

I would add that with multiple jobs, the errors I get are dependency errors. Somehow the order that jobs get executed determines what dependencies are available.

[maleadt]

I would add that with multiple jobs, the errors I get are dependency errors. Somehow the order that jobs get executed determines what dependencies are available.

I'm not sure how to interpret that. Can you elaborate?

do you mean somehow run testf(BLAS.asum, rand(T, m)) by itself? I'm not quite sure how to invoke just that. But it always fails for me on the same line when I do julia --project test\runtests.jl --jobs=1 libraries/cublas/level1.

You can do julia --project=test -L test/setup.jl to activate the test environment, but generally the testf functions are simply runners that evaluate something on both the CPU and GPU and compare the results, so you should be able to rephrase the test in terms of normal operations that don't require the test environment.

[mind6]

I would add that with multiple jobs, the errors I get are dependency errors. Somehow the order that jobs get executed determines what dependencies are available.

I'm not sure how to interpret that. Can you elaborate?

do you mean somehow run testf(BLAS.asum, rand(T, m)) by itself? I'm not quite sure how to invoke just that. But it always fails for me on the same line when I do julia --project test\runtests.jl --jobs=1 libraries/cublas/level1.

You can do julia --project=test -L test/setup.jl to activate the test environment, but generally the testf functions are simply runners that evaluate something on both the CPU and GPU and compare the results, so you should be able to rephrase the test in terms of normal operations that don't require the test environment.

This is my MWE:

@show pwd()
include("setup.jl")

using CUDA.CUBLAS
using LinearAlgebra
using Adapt
using Random

m = 10

for T in [Float32, Float64, ComplexF64, ComplexF32] 
	@show T
	Random.seed!(1234)	
	A = rand(T, m)
	
	try 
		@test testf(BLAS.asum, A) 
	catch e
		@show A
		B =adapt(CuArray, A)
		@show B

		@show BLAS.asum(A)
		@show BLAS.asum(B)
		rethrow(e)
	end
end

My output:

pwd() = "K:\\.julia\\dev\\CUDA"
T = Float32
T = Float64
T = ComplexF64
T = ComplexF32
Test Failed at k:\.julia\dev\CUDA\test\myMWE.jl:17
  Expression: testf(BLAS.asum, A)

A = ComplexF32[0.32597667f0 + 0.5490511f0im, 0.21858662f0 + 0.8942454f0im, 0.35311162f0 + 0.39425534f0im, 0.9531246f0 + 0.7955469f0im, 0.49424982f0 + 0.748415f0im, 0.57823193f0 + 0.72793496f0im, 0.007447958f0 + 0.19937658f0im, 0.43924308f0 + 0.6825326f0im, 0.9567409f0 + 0.6478553f0im, 0.99666524f0 + 0.749194f0im]
B = ComplexF32[0.32597667f0 + 0.5490511f0im, 0.21858662f0 + 0.8942454f0im, 0.35311162f0 + 0.39425534f0im, 0.9531246f0 + 0.7955469f0im, 0.49424982f0 + 0.748415f0im, 0.57823193f0 + 0.72793496f0im, 0.007447958f0 + 0.19937658f0im, 0.43924308f0 + 0.6825326f0im, 0.9567409f0 + 0.6478553f0im, 0.99666524f0 + 0.749194f0im]
BLAS.asum(A) = 10.34919f0
BLAS.asum(B) = 11.711785f0
ERROR: Test.FallbackTestSetException("There was an error during testing")
Stacktrace:
 [1] record(ts::Test.FallbackTestSet, t::Union{Test.Error, Test.Fail})
   @ Test K:\.julia\juliaup\julia-1.10.10+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:1000
 [2] do_test(result::Test.ExecutionResult, orig_expr::Any)
   @ Test K:\.julia\juliaup\julia-1.10.10+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:705
 [3] macro expansion
   @ K:\.julia\juliaup\julia-1.10.10+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Test\src\Test.jl:672 [inlined]
 [4] top-level scope
   @ k:\.julia\dev\CUDA\test\myMWE.jl:17

• the results of calling BLAS.asum is deterministic on both cpu and gpu
• the input arrays look like they have the correct values
• the results are always different
• it only fails for ComplexF32, passing Float32, Float64, and ComplexF64

[mind6]

The dependency issue would be when it fails with something like:

K:\.julia\dev\CUDA>julia --project test\runtests.jl --jobs=1 base/array
┌ Info: System information:
│ CUDA runtime 12.9, artifact installation
│ CUDA driver 12.9
│ NVIDIA driver 576.80.0
│
│ CUDA libraries:
│ - CUBLAS: 12.9.1
│ - CURAND: 10.3.10
│ - CUFFT: 11.4.1
│ - CUSOLVER: 11.7.5
│ - CUSPARSE: 12.5.10
│ - CUPTI: 2025.2.1 (API 28.0.0)
│ - NVML: 12.0.0+576.80
│
│ Julia packages:
│ - CUDA: 5.8.2
│ - CUDA_Driver_jll: 0.13.1+0
│ - CUDA_Runtime_jll: 0.17.1+0
│
│ Toolchain:
│ - Julia: 1.10.10
│ - LLVM: 15.0.7
│
│ 1 device:
└   0: NVIDIA GeForce RTX 3080 (sm_86, 8.212 GiB / 10.000 GiB available)
[ Info: Testing using device 0 (NVIDIA GeForce RTX 3080). To change this, specify the `--gpu` argument to the tests, or set the `CUDA_VISIBLE_DEVICES` environment variable.
[ Info: Running 1 tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
                |          | ---------------- GPU ---------------- | ---------------- CPU ---------------- |
Test   (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
base/array  (2) |         failed at 2025-07-08T11:06:15.302
Testing finished in 2 seconds, 402 milliseconds
Worker 2 failed running test base/array:
Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
base/array: Error During Test at K:\.julia\dev\CUDA\test\setup.jl:62
  Got exception outside of a @test
  LoadError: ArgumentError: Package ChainRulesCore not found in current path.
  - Run `import Pkg; Pkg.add("ChainRulesCore")` to install the ChainRulesCore package.
  Stacktrace:
    [1] macro expansion
      @ .\loading.jl:1842 [inlined]
    [2] macro expansion
      @ .\lock.jl:267 [inlined]

It would be unable to find an import package, but in a nondeterministic way when there are multiple jobs.

[maleadt]

OK fascinating. Can you confirm the following simplified MWE (without the test infrastructure) also fails?

using CUDA, LinearAlgebra, Random

for T in [Float32, Float64, ComplexF64, ComplexF32]
	@show T
	Random.seed!(1234)
	A = rand(T, 10)

	try
        @assert BLAS.asum(A) ≈ BLAS.asum(CuArray(A))
	catch
		@show A BLAS.asum(A)
        @show CuArray(A) BLAS.asum(CuArray(A))
		rethrow()
	end
end

[maleadt]

The dependency issue would be when it fails with something like:

That's because you're running in the CUDA.jl project, not its test project (which include test-only dependencies like ChainRulesCore).

[mind6]

OK fascinating. Can you confirm the following simplified MWE (without the test infrastructure) also fails?

using CUDA, LinearAlgebra, Random

for T in [Float32, Float64, ComplexF64, ComplexF32]
@show T
Random.seed!(1234)
A = rand(T, 10)

try
@Assert BLAS.asum(A) ≈ BLAS.asum(CuArray(A))
catch
@show A BLAS.asum(A)
@show CuArray(A) BLAS.asum(CuArray(A))
rethrow()
end
end

yes I get the following output:

T = Float32
T = Float64
T = ComplexF64
T = ComplexF32
A = ComplexF32[0.32597667f0 + 0.5490511f0im, 0.21858662f0 + 0.8942454f0im, 0.35311162f0 + 0.39425534f0im, 0.9531246f0 + 0.7955469f0im, 0.49424982f0 + 0.748415f0im, 0.57823193f0 + 0.72793496f0im, 0.007447958f0 + 0.19937658f0im, 0.43924308f0 + 0.6825326f0im, 0.9567409f0 + 0.6478553f0im, 0.99666524f0 + 0.749194f0im]
BLAS.asum(A) = 10.34919f0
CuArray(A) = ComplexF32[0.32597667f0 + 0.5490511f0im, 0.21858662f0 + 0.8942454f0im, 0.35311162f0 + 0.39425534f0im, 0.9531246f0 + 0.7955469f0im, 0.49424982f0 + 0.748415f0im, 0.57823193f0 + 0.72793496f0im, 0.007447958f0 + 0.19937658f0im, 0.43924308f0 + 0.6825326f0im, 0.9567409f0 + 0.6478553f0im, 0.99666524f0 + 0.749194f0im]
BLAS.asum(CuArray(A)) = 11.711785f0
ERROR: AssertionError: BLAS.asum(A) ≈ BLAS.asum(CuArray(A))
Stacktrace:
 [1] top-level scope
   @ k:\.julia\dev\CUDA\test\mwe2.jl:9

[maleadt]

A = ComplexF32[0.32597667f0 + 0.5490511f0im, 0.21858662f0 + 0.8942454f0im, 0.35311162f0 + 0.39425534f0im, 0.9531246f0 + 0.7955469f0im, 0.49424982f0 + 0.748415f0im, 0.57823193f0 + 0.72793496f0im, 0.007447958f0 + 0.19937658f0im, 0.43924308f0 + 0.6825326f0im, 0.9567409f0 + 0.6478553f0im, 0.99666524f0 + 0.749194f0im]
BLAS.asum(A) = 10.34919f0
CuArray(A) = ComplexF32[0.32597667f0 + 0.5490511f0im, 0.21858662f0 + 0.8942454f0im, 0.35311162f0 + 0.39425534f0im, 0.9531246f0 + 0.7955469f0im, 0.49424982f0 + 0.748415f0im, 0.57823193f0 + 0.72793496f0im, 0.007447958f0 + 0.19937658f0im, 0.43924308f0 + 0.6825326f0im, 0.9567409f0 + 0.6478553f0im, 0.99666524f0 + 0.749194f0im]
BLAS.asum(CuArray(A)) = 11.711785f0

Wait, this is your CPU BLAS being wrong:

T = ComplexF32
BLAS.asum(A) = 11.711785f0
BLAS.asum(CuArray(A)) = 11.711785f0

[mind6]

We should try to get more CPU results on this random seed for AMD 9950X and 7950X

[maleadt]

Can you file this upstream (e.g. on the Julia repo) instead? This looks like a bug in BLAS for your CPU.

FWIW, the pure Julia version agrees with CUDA.jl:

julia> BLAS.asum(A)
11.711785f0

julia> sum(z -> abs(real(z)) + abs(imag(z)), A)
11.711785f0