Skip to content

Move _worker_check() logic to process_dataloader() in DDPSpawnStrategy class.  #12216

@ninginthecloud

Description

@ninginthecloud

Proposed refactor

This issue follows @ananthsub 's #11756 to move strategy-specific dataloader logic to the stategies.

Motivation

_worker_check() in data_connector.py is used to check the number of workers for the dataloader when user sets up DDP_SPAWN strategy.
https://github.com/PyTorchLightning/pytorch-lightning/blob/cc43d07db1ab77385feff04c01f040c5cad805a9/pytorch_lightning/trainer/connectors/data_connector.py#L274

Since this dataloader verfication logic is dpp_spawn strategy specific, it'll be better we moved the logic to DDPSpawnStrategy class. By doing so, _worker_check() can be removed from data_connector.py.

Pitch

This logic check could be moved to process_dataloader() in DDPSpawnStrategy class.

Additional context

cc: @edward-io @four4fish @ananthsub


If you enjoy Lightning, check out our other projects! ⚡

  • Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

  • Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

  • Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

  • Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

  • Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @justusschock @awaelchli @ninginthecloud @rohitgr7 @otaj @tchaton @akihironitta

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions