How to make stages?

[thepowerfuldeez]

Hi! Thank you for a great framework! I've tried to write down stages for training. E.g. in my config:

    <<: *default

  stage2:
    <<: *default

    datasets:
      <<: *datasets
             # 240k
      root: ['path1',
             'path2,
             'path3]
      per_folder_ratio: [1.0, 1.0, 1.0]
      transform:
        <<: *transform
        augs_lvl: "light"

    optimizer:
      class_name: RAdam
      lr: 0.0001

and in train.py:

    for stage_name, stage in selected_stages.items():
        print(f"running stage {stage_name}")
        model = build_model(stage) # BUILD LIGHTNING MODULE from func

        trainer = build_trainer(stage_config=stage,
                                module=model,
                                ...) # MAKE pytorch_lightning Trainer using this model
        trainer.fit(model)

I'm using early stopping which make one iteration in aforementioned loop (another way which i'm also tried is to wait max_epochs epochs).
The problem is that second call to trainer.fit initializes DDP one more time and program is crashing because ip address is not freed from previous DDP init.

RuntimeError: Address already in use

I've tried master version of pytorch lightning but the problem did not dissapear

[Borda]

it is kind of fine-tuning, right? @lgvaz was talking about something similar...
cc: @PyTorchLightning/core-contributors

[thepowerfuldeez]

yes, it's used in many different tasks, e.g. training on synthetic data and fine-tuning on real data then. This approach mostly uses training state as some separate entity which you can change.
See catalyst implementation: https://github.com/catalyst-team/catalyst/blob/068f119c9708fde9d044b6f994bc28dd65194f67/catalyst/core/runner.py#L373
They just replace self.model with model prepared from specific stage. Same for criterion, loaders etc.

When I try to do similar (I am making new trainer with LightningModule inside) inside for stage in stages loop program crashes because of DDP initialization. I'm just wondering if you can take out ddp init stage out of fit then I believe my approach will work.
It might be like that:

1. trainer = Trainer(...)
2. trainer.init_ddp()
3. trainer.fit(model)
4. new_trainer = Trainer(..., resume=CHECKPOINT_OF_LAST_MODEL)
5. new_trainer.fit(new_model)```

or another approach which I also tried but not succeeded
```1. trainer = Trainer(...)
2. trainer.fit(model)
3. del trainer  # THIS LINE DOESN"T REMOVE DDP INITIALIZATION
4. new_trainer = Trainer(...)
5. new_trainer.fit(new_model). # HERE I GOT ADDRESS IN USE EXCEPTION FROM torch.distributed

[williamFalcon]

i think @awaelchli solved this problem?

[awaelchli]

Yes, I am working on a fix so that you can call fit and test as many times as you want. #2790
I ran into some other unexpected complications with nccl, but I try to get this bugfix finished asap.

[awaelchli]

@thepowerfuldeez sorry for the late reply.
I assume you were using distributed_backend="ddp", but that doesn't work when you have multiple calls to .fit() in your script.
You need to use distributed_backend="ddp_spawn" in this case. Then you can do your for loop.