Data parallel error with O2 and not O1

[aclyde11]

When using O2, data parallel does not work:
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)

however with O1, everything works just fine.

model = GeneralVae(encoder, decoder, rep_size=500).cuda()
optimizer = optim.Adam(model.parameters(), lr=LR)
model, optimizer = amp.initialize(model, optimizer, opt_level='O2')
if data_para and torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)
    model = model.cuda()

loss_picture = customLoss()

val_losses = []
train_losses = []

def train(epoch):
    train_loader_food = generate_data_loader(train_root, get_batch_size(epoch), int(rampDataSize * data_size))
    print("Epoch {}: batch_size {}".format(epoch, get_batch_size(epoch)))
    model.train()
    train_loss = 0
    loss = None
    for batch_idx, (data, _, aff) in enumerate(train_loader_food):
        data = data[0].cuda(0)

[tullie]

I'm running into the same error for O0, O2, O3:
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)

O1 is working as expected.

[jshanna100]

Same.

[zplizzi]

Same here, at least for O2. O1 does work.

[mcarilli]

In general we strongly recommend DistributedDataParallel (either torch.nn.parallel.DistributedDataParallel or apex.parallel.DistributedDataParallel) over DataParallel, because global interpreter lock sharing within a single process is not great for performance. Currently, I don't test with or claim to support DataParallel. If you are open to trying DistributedDataParallel, I have a simple example showing proper DDP initialization and launch. The Imagenet example also shows DDP use along with distributed data sampling.

That being said, I don't think DataParallel is fundamentally incompatible with Amp control flow. I see one potential problem with your code above: you are calling .cuda on the model after it's been returned from amp.initialize. You should be doing things in the following order:

model.cuda() # Cuda-ing your model should occur before the call to amp.initialize
model, optimizer = amp.initialize(model, optimizer)
model = nn.DataParallel(model)

Try this and let me know if it works.

The fact that cuda-ing your model should occur before amp.initialize is a general truth, independent of DataParallel or DistributedDataParallel. However, I can't really set up hard checks for that, because people may legitimately want part of their model to remain on the CPU.

[aclyde11]

@mcarilli I tried the fix above but it still produces the same error:
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)

I will transition to distributed data parallel

[mcarilli]

Historically we only test with DistributedDataParallel because performance tends to be better, but the dataset sharing issue raised by @seongwook-ham in #269 is a compelling use case. @ptrblck and I will look into it. Current to-do list is better fused optimizers, checkpointing, sparse gradients, and then DataParallel, so it may be a couple weeks before I can give it undivided attention.

[seongwook-ham]

i find that old api(FP16_Optimizer) works well with nn.dataparalllel
if you need dataparallel, you could use it with FP16_Optimizer and model.half()

[mcarilli]

Please don't use the old FP16_Optimizer API, that may break at any time. It might already be broken.

In general, O1 is preferable over O2 anyway, so if O1 works with DataParallel currently, using DataParallel + O1 with the current API is a much better workaround than DataParallel + reverting to the old FP16_Optimizer.

We do want to support O2 + DataParallel but haven't gotten a chance to look at it yet. @seongwook-ham I am interested to hear if people have compelling reasons for requiring O2 (or FP16_Optimizer) over O1, because O1 is safer in general. Is O1 significantly slower?

[seongwook-ham]

yes new api o1 is significantly slower than old api with FP16_Optimizer and half in nn.dataparallel case.
in this case 1.41it/s vs 2.4it/s. in same setting fp32 is 1.2it/s
also in apex distributed dataparallel case, o1 with adam is significantly slower than o2 with fusedadam
2.9 its/s vs 4.58 its/s
this test is based on modified version of https://github.com/huggingface/pytorch-pretrained-BERT
in environment with 6950x 3x2080ti 1xtitan rtx
when tested on dgx-1(8xv100(32G NVlink)), result is similar

[greathope]

Same

[mwyborski]

Same

[askerlee]

@mcarilli Unfortunately I'm using FusedAdam, which requires O2. Seems a deadlock so I have to revert to FP16_Optimizer...

[iariav]

@mcarilli
still seeing this issue.
any idea when the support for O2 + DataParallel will kick in?

thanks