-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Closed
Labels
3rd partyRelated to a 3rd-partyRelated to a 3rd-partybugSomething isn't workingSomething isn't workinglightningclipl.cli.LightningCLIpl.cli.LightningCLI
Milestone
Description
🐛 Bug
Instantiate DDPStrategy with find_unused_parameters=False raise exceptions when using lazy_instance().
The non-lazy mode and no-parameter mode work as expected.
To Reproduce
import os
import torch
from torch.utils.data import DataLoader, Dataset
import pytorch_lightning as pl
from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning.utilities import cli
from pytorch_lightning.strategies import DDPStrategy
from jsonargparse import lazy_instance
class RandomDataset(Dataset):
def __init__(self, size, length):
self.len = length
self.data = torch.randn(length, size)
def __getitem__(self, index):
return self.data[index]
def __len__(self):
return self.len
class RandomDataModule(pl.LightningDataModule):
def train_dataloader(self):
return DataLoader(RandomDataset(32, 64), batch_size=2)
class BoringModel(LightningModule):
def __init__(self):
super().__init__()
self.layer = torch.nn.Linear(32, 2)
def forward(self, x):
return self.layer(x)
def training_step(self, batch, batch_idx):
loss = self(batch).sum()
self.log("train_loss", loss)
return {"loss": loss}
def configure_optimizers(self):
return torch.optim.SGD(self.layer.parameters(), lr=0.1)
if __name__ == "__main__":
c = cli.LightningCLI(
BoringModel,
RandomDataModule,
trainer_defaults={
# Case 1:
# ValueError: 'Configuration check failed :: No action for
# destination key "find_unused_parameters" to check its value.'
'strategy': lazy_instance(DDPStrategy, find_unused_parameters=False),
# Case 2: (lazy mode without find_unused_parameters) work as expected
'strategy': lazy_instance(DDPStrategy),
# Case 3: (non-lazy mode) work as expected
'strategy': DDPStrategy(find_unused_parameters=False),
}
)Expected behavior
All three cases are supposed to work.
Environment
- CUDA:
- GPU:
- NVIDIA GeForce RTX 3090
- available: True
- version: 11.3
- GPU:
- Packages:
- numpy: 1.22.3
- pyTorch_debug: False
- pyTorch_version: 1.11.0+cu113
- pytorch-lightning: 1.6.0
- tqdm: 4.64.0
- System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.9.11
- version: Extend CI #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022
Additional context
Metadata
Metadata
Assignees
Labels
3rd partyRelated to a 3rd-partyRelated to a 3rd-partybugSomething isn't workingSomething isn't workinglightningclipl.cli.LightningCLIpl.cli.LightningCLI