Missing details in the document for the random seed

[DKandrew]

📚 Documentation

Even though in the documentation, lightning very thoughtfully reminds readers of setting the random seed when using distributed data parallel. However, it does not mention where should we set them. Based on my experience on PyTorch, the seed can be set after calling the torch.distributed.init_process_group. For lightning, this call is handled by the trainer, so where should we set the random seed? One thought from me is that we set the seed in the __init__() of our LightningModule, any other suggestions? What would be the best practice for lightning?

Also, I believe that the seed_everything() function is called in the wrong place on the lightning's official example.

[awaelchli]

@DKandrew is there anything wrong with calling seed_everything at the entry point of the training script? I ask because you mention doing it after distributed init group.

[DKandrew]

Hi @ashwinb,

TL;DR. The training script is correct.

I did a little experiment to see if the child processes could inherit the random state. Here is my script

import argparse
import os
import numpy.random
import random
import torch
import torch.distributed as distributed

from pytorch_lightning import seed_everything

seed_everything(207)


def python_random(counts):
    return [random.random() for _ in range(counts)]


def numpy_random(counts):
    return [numpy.random.random() for _ in range(counts)]


def torch_random(counts):
    return [torch.rand(1).item() for _ in range(counts)]


def parse_args():
    """
    Parse the command line arguments
    """
    parser = argparse.ArgumentParser(description='Playground')
    parser.add_argument(
        '--local_rank',
        default=0,
        type=int,
        required=True,
        help='local rank of this process (i.e. process id)'
    )

    args = parser.parse_args()
    return args


def main():
    args = parse_args()

    # Determine current process's local rank
    local_rank = args.local_rank

    # Initialize distributed training
    distributed.init_process_group('nccl', init_method='env://')

    msg = f"local range: {local_rank}\n"
    msg += "Python random: {}\n".format(python_random(3))
    msg += "Numpy random: {}\n".format(numpy_random(3))
    msg += "Torch random: {}\n".format(torch_random(3))

    print(msg)


if __name__ == '__main__':
    main()

You can invoke this script by calling

python -m torch.distributed.launch --nproc_per_node=[number of gpu] [script_name].py

Here is what I got by using two GPUs

local range: 1
Python random: [0.5059766669498406, 0.6690652117641676, 0.8534819330516183]
Numpy random: [0.5029648605599233, 0.3660225706426885, 0.44731428418920105]
Torch random: [0.7590030431747437, 0.13656914234161377, 0.7017777562141418]

local range: 0
Python random: [0.5059766669498406, 0.6690652117641676, 0.8534819330516183]
Numpy random: [0.5029648605599233, 0.3660225706426885, 0.44731428418920105]
Torch random: [0.7590030431747437, 0.13656914234161377, 0.7017777562141418]

They are clearly the same.
If you comment out the seed_everything(), you will get different results.

local range: 1
Python random: [0.3214156803428436, 0.49857925013622195, 0.9934577900320609]
Numpy random: [0.3255639188497673, 0.6924278687540534, 0.8184138384742942]
Torch random: [0.5103110074996948, 0.812697172164917, 0.7066746950149536]

local range: 0
Python random: [0.016087008610051856, 0.1289228962942407, 0.08177483232901683]
Numpy random: [0.5057701032003784, 0.47114302901911165, 0.26440427354457685]
Torch random: [0.25142884254455566, 0.7235633134841919, 0.8742098808288574]

so you are right. The script is correct, and my understanding of PyTorch distributed training gets updated.

Reference:
Code inspired from link

[DKandrew]

I think it would be helpful to explicitly tell users that they should set the random seed before the Trainer is instantiated. Making a PR for this.