DDP-spawn on LSF cluster (multi-node) should not be supported

[awaelchli]

🚀 Bug

DDP-spawn on multi-node configurations where processes get spawned externally should not be supported.

To Reproduce

Running with

Trainer(strategy="ddp_spawn", num_nodes>1, devices=...)

on LSF is currently faulty but should not be supported.

Expected behavior

Should not be supported. Two options:

A) Raise an error and suggest to use strategy="ddp".
B) Fall-back to DDP. A similar check already exists for SLURM:

https://github.com/Lightning-AI/lightning/blob/d2840a20bd27a0a86fc59ba54d94543e63523133/src/pytorch_lightning/trainer/connectors/accelerator_connector.py#L616-L619

I suggest to go with B) as we already do it with the other environments.

Additional context

Originally reported on Slack.

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.

[carmocca]

I think this fall-back logic is a bit faulty. I would prefer to do (A) and even update the previous logic to also error-out.

[Atharva-Phatak]

@awaelchli and @carmocca Quick question you want to fallback to option (A) correct? I.e raise a warning and set strategy to DDP, right?
Do you have any specific warning in mind ?
If you could divulge some details I can write a fix and create a PR :)

I can take this up if this is open :)

[awaelchli]

@Atharva-Phatak I show the fallback logic in the description above, LSFEnvironment needs to be added to these checks there. That's the quick fix to align the logic with other environments where we fall back to "ddp".

I wouldn't say that the logic is faulty, just brittle, and easy to miss updating when adding a new environment.

I would definitely go for the quick fix (fallback) first (for both Trainer and Lite). Then, the error handling can be investigated.

[Atharva-Phatak]

@awaelchli Essentially right now we just set strategy to DDP right? So we need to implement LSFEnvironment class something like TorchElasticEnvironment right ?

[awaelchli]

There is nothing to implement really. All we have to do is add the check here as I pointed out above:
https://github.com/Lightning-AI/lightning/blob/d2840a20bd27a0a86fc59ba54d94543e63523133/src/pytorch_lightning/trainer/connectors/accelerator_connector.py#L616-L619

We just add or LSFEnvironment.detect() and extend the existing test.

[Atharva-Phatak]

@awaelchli Thanks for clearing that I though LSFEnironment was not implemented :p