RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` #8143

zhoufengfan · 2021-06-26T08:23:13Z

zhoufengfan
Jun 26, 2021

Hi, everyone.

I have a repo whose link is https://github.com/zhoufengfan/pytorch-lightning-cifar10.

When I run python train.py, it returns this error:

Traceback (most recent call last):
  File "train.py", line 21, in <module>
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
    self.accelerator.start_training(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
    return self.run_train()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in run_train
    self.run_sanity_check(self.lightning_module)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1107, in run_sanity_check
    self.run_evaluation()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 174, in evaluation_step
    output = self.trainer.accelerator.validation_step(args)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 226, in validation_step
    return self.training_type_plugin.validation_step(*args)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 322, in validation_step
    return self.model(*args, **kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 57, in forward
    output = self.module.validation_step(*inputs, **kwargs)
  File "/root/code/test-of-pytorch-lightning/backbone.py", line 42, in validation_step
    self.evaluate(batch, 'val')
  File "/root/code/test-of-pytorch-lightning/backbone.py", line 32, in evaluate
    logits = self(x)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/code/test-of-pytorch-lightning/backbone.py", line 20, in forward
    out = self.model(x)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/torchvision/models/resnet.py", line 220, in forward
    return self._forward_impl(x)
  File "/root/miniconda3/lib/python3.8/site-packages/torchvision/models/resnet.py", line 215, in _forward_impl
    x = self.fc(x)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 1690, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

But after I change the AVAIL_GPUS in the hyper_var.py file to min(1, torch.cuda.device_count()), the bug disappear()(see the comment in the hyper_var.py file).

When I change the AVAIL_GPUS in the hyper_var.py file to "1,2", it returns this error:

Traceback (most recent call last):
  File "train.py", line 21, in <module>
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
Traceback (most recent call last):
  File "/root/code/test-of-pytorch-lightning/train.py", line 21, in <module>
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    work = _default_pg.barrier()
    self.training_type_plugin.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8

The hardware parameters of my server are:

NVIDIA GeForce 3090
CUDA Version: 11.2
Driver Version: 460.39

Can anyone help me to fix the bug?

Thanks.

Answered by zhoufengfan

Jul 16, 2021

OK, I know the reason. The reason is that I haven't installed NCCL on my machine.

After I installed NCCL on my machine, the errors disappeared.

Thanks to all the people who have replied to me. @awaelchli @SkafteNicki

View full answer

SkafteNicki · 2021-07-02T13:28:09Z

SkafteNicki
Jul 2, 2021
Collaborator

This error can happen due to a number of reasons, could you try launching your script as:
CUDA_LAUNCH_BLOCKING=1 python script.py args
to hopefully get a more meaningfull error.

7 replies

zhoufengfan Jul 4, 2021
Author

After I run CUDA_VISIBLE_DEVICES=2 python train.py, it returns this error:

Traceback (most recent call last):
  File "train.py", line 13, in <module>
    trainer = Trainer(
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 40, in insert_env_defaults
    return fn(self, **kwargs)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 319, in __init__
    self.accelerator_connector = AcceleratorConnector(
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 129, in __init__
    self.parallel_device_ids = device_parser.parse_gpu_ids(self.gpus)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/utilities/device_parser.py", line 81, in parse_gpu_ids
    gpus = _sanitize_gpu_ids(gpus)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/utilities/device_parser.py", line 149, in _sanitize_gpu_ids
    raise MisconfigurationException(
pytorch_lightning.utilities.exceptions.MisconfigurationException: You requested GPUs: [2]
 But your machine only has: [0]

zhoufengfan Jul 5, 2021
Author

But after I run CUDA_VISIBLE_DEVICES=0,1,2 python train.py, it runs normally. The process allocates memory in both GPU0 and GPU2.

OK, I know the reason for my error.

pytorch_lightning will allocate memory in the 0th available GPU. However, someone else filled the memory of GPU0. Then it returns an error.

How can I make the program allocate memory only on GPU2?

SkafteNicki Jul 5, 2021
Collaborator

@zhoufengfan could you try setting the auto_select_gpus=True. This should auto pick gpus that are not already occupied.
Read more here: https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#auto-select-gpus

zhoufengfan Jul 5, 2021
Author

But after I set the auto_select_gpus=True and run CUDA_LAUNCH_BLOCKING=1 python train.py, it returns this error:

Global seed set to 7
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Global seed set to 7
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Global seed set to 7
Global seed set to 7
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
Traceback (most recent call last):
  File "/root/code/test-of-pytorch-lightning/train.py", line 22, in <module>
Traceback (most recent call last):
  File "train.py", line 22, in <module>
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8

Is this a bug in pytorch_lightning? I suggest that the developers make the return errors more understandable for the users.

SkafteNicki Jul 5, 2021
Collaborator

This does not seem to be an error in pytorch-lightning since the last line refers to torch:
"/opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:"

zhoufengfan · 2021-07-05T02:22:46Z

zhoufengfan
Jul 5, 2021
Author

However, after I change the AVAIL_GPUS to "1,2" and I run CUDA_LAUNCH_BLOCKING=1 python train.py, it returns this error:

Global seed set to 7
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Global seed set to 7
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Global seed set to 7
Global seed set to 7
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2]
Traceback (most recent call last):
  File "/root/code/test-of-pytorch-lightning/train.py", line 21, in <module>
Traceback (most recent call last):
  File "train.py", line 21, in <module>
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8

2 replies

awaelchli Jul 5, 2021

PL version 1.3.8 has a fix for the barrier with device ids
Fixed it in #8165. Try to upgrade, maybe it helps.

zhoufengfan Jul 5, 2021
Author

After I set the upgrade to 1.3.8 and run CUDA_LAUNCH_BLOCKING=1 python train.py, it returns this error:

Global seed set to 7
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Global seed set to 7
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Global seed set to 7
Global seed set to 7
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All DDP processes registered. Starting ddp with 2 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2]
Traceback (most recent call last):
  File "train.py", line 22, in <module>
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 755, in _run
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 780, in pre_dispatch
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 293, in pre_dispatch
    self._share_information_to_prevent_deadlock()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 362, in _share_information_to_prevent_deadlock
    self._share_pids()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 371, in _share_pids
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 304, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
Traceback (most recent call last):
  File "/root/code/test-of-pytorch-lightning/train.py", line 22, in <module>
    trainer.fit(model, cifar10_dm)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 755, in _run
    self.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 780, in pre_dispatch
    self.accelerator.pre_dispatch(self)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 293, in pre_dispatch
    self._share_information_to_prevent_deadlock()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 362, in _share_information_to_prevent_deadlock
    self._share_pids()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 371, in _share_pids
    self.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 304, in barrier
    torch_distrib.barrier()
  File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8

zhoufengfan · 2021-07-16T13:52:00Z

zhoufengfan
Jul 16, 2021
Author

OK, I know the reason. The reason is that I haven't installed NCCL on my machine.

After I installed NCCL on my machine, the errors disappeared.

Thanks to all the people who have replied to me. @awaelchli @SkafteNicki

0 replies

RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle) #8143

Uh oh!

zhoufengfan Jun 26, 2021

Replies: 3 comments · 9 replies

Uh oh!

SkafteNicki Jul 2, 2021 Collaborator

Uh oh!

zhoufengfan Jul 4, 2021 Author

Uh oh!

zhoufengfan Jul 5, 2021 Author

Uh oh!

SkafteNicki Jul 5, 2021 Collaborator

Uh oh!

zhoufengfan Jul 5, 2021 Author

Uh oh!

SkafteNicki Jul 5, 2021 Collaborator

Uh oh!

zhoufengfan Jul 5, 2021 Author

Uh oh!

awaelchli Jul 5, 2021

Uh oh!

zhoufengfan Jul 5, 2021 Author

Uh oh!

Uh oh!

zhoufengfan Jul 16, 2021 Author

RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` #8143

zhoufengfan
Jun 26, 2021

Replies: 3 comments 9 replies

SkafteNicki
Jul 2, 2021
Collaborator

zhoufengfan Jul 4, 2021
Author

zhoufengfan Jul 5, 2021
Author

SkafteNicki Jul 5, 2021
Collaborator

zhoufengfan Jul 5, 2021
Author

SkafteNicki Jul 5, 2021
Collaborator

zhoufengfan
Jul 5, 2021
Author

zhoufengfan Jul 5, 2021
Author

zhoufengfan
Jul 16, 2021
Author