Number of steps per epoch

[celsofranssa]

Some learning rate schedulers as OneCycleLR requires the number of steps per epoch.

Then, how to get the number of steps in configure_optimizers(self) scope?

Note: Training data is given during Trainer instantiation:

# training
    dm.setup('fit')
    trainer.fit(model, datamodule=dm)

[tchaton]

This should work just fine.

def configure_optimizers(self):
       num_batches = len(self.train_dataloader()) / self.trainer.accumulate_grad_batches
      ....
      return [optim..], [scheduler...]

Note: If you pass for train/val_dataloader or datamodule directly into the .fit function, Lightning will override the train_dataloader() function with the provided one, so it should work fine.

Here is a good approximation for the total number of steps.

    @property
    def num_training_steps(self) -> int:
        """Total training steps inferred from datamodule and devices."""
        dataset = self.train_dataloader()
        if self.trainer.max_steps:
            return self.trainer.max_steps

        dataset_size = (
            self.trainer.limit_train_batches
            if self.trainer.limit_train_batches != 0
            else len(dataset)
        )

        num_devices = max(1, self.trainer.num_gpus, self.trainer.num_processes)
        if self.trainer.tpu_cores:
            num_devices = max(num_devices, self.trainer.tpu_cores)

        effective_batch_size = dataset.batch_size * self.trainer.accumulate_grad_batches * num_devices
        return (dataset_size // effective_batch_size) * self.trainer.max_epochs

Closing this issue as the answer should work out. Feel free to re-open it if it doesn't solve your problem.

[celsofranssa]

When using num_batches = len(self.train_dataloader()) / self.trainer.accumulate_grad_baches, the following error happens:

Traceback (most recent call last):
  File "xCoFormer.py", line 169, in perform_tasks
    fit(hparams)
  File "xCoFormer.py", line 84, in fit
    trainer.fit(model, datamodule=dm)
  File "/home/celso/projects/venvs/xCoFormer/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 456, in fit
    self.accelerator_backend.setup(model)
  File "/home/celso/projects/venvs/xCoFormer/lib/python3.7/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 52, in setup
    self.setup_optimizers(model)
  File "/home/celso/projects/venvs/xCoFormer/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 145, in setup_optimizers
    optimizers, lr_schedulers, optimizer_frequencies = self.trainer.init_optimizers(model)
  File "/home/celso/projects/venvs/xCoFormer/lib/python3.7/site-packages/pytorch_lightning/trainer/optimizers.py", line 30, in init_optimizers
    optim_conf = model.configure_optimizers()
  File "/home/celso/projects/xCoFormer/source/model/JointEncoder.py", line 44, in configure_optimizers
    print("num_batches: ", len(self.train_dataloader()) / self.trainer.accumulate_grad_baches)
AttributeError: 'Trainer' object has no attribute 'accumulate_grad_baches'

[celsofranssa]

Closing this issue as the answer should work out. Feel free to re-open it if it doesn't solve your problem.

Did you mean create a new issue?

[rohitgr7]

@celsofranssa it's self.trainer.accumulate_grad_batches. just a typo there.

[celsofranssa]

I got it, thanks!

[celsofranssa]

This should work just fine.

def configure_optimizers(self):
       num_batches = len(self.train_dataloader()) / self.trainer.accumulate_grad_batches
      ....
      return [optim..], [scheduler...]

Note: If you pass for train/val_dataloader or datamodule directly into the .fit function, Lightning will override the train_dataloader() function with the provided one, so it should work fine.

Here is a good approximation for the total number of steps.

    @property
    def num_training_steps(self) -> int:
        """Total training steps inferred from datamodule and devices."""
        dataset = self.train_dataloader()
        if self.trainer.max_steps:
            return self.trainer.max_steps

        dataset_size = (
            self.trainer.limit_train_batches
            if self.trainer.limit_train_batches != 0
            else len(dataset)
        )

        num_devices = max(1, self.trainer.num_gpus, self.trainer.num_processes)
        if self.trainer.tpu_cores:
            num_devices = max(num_devices, self.trainer.tpu_cores)

        effective_batch_size = dataset.batch_size * self.trainer.accumulate_grad_batches * num_devices
        return (dataset_size // effective_batch_size) * self.trainer.max_epochs

Closing this issue as the answer should work out. Feel free to re-open it if it doesn't solve your problem.

dataset_size is being set to 1.0 instead of the real number of batches in the dataloader because:

# default used by the Trainer
trainer = Trainer(limit_train_batches=1.0)

[KevinMusgrave]

@tchaton I don't think the num_training_steps function works. As @celsofranssa pointed out, dataset_size gets set to 1, so the function returns 0 because (dataset_size // effective_batch_size) equals 0.

[tsteffek]

I just stumbled upon the same problem, and tried len(self.train_dataloader()) // self.trainer.accumulate_grad_batches. Works for me.

Your misunderstanding is that dataset_size sets the size to 1. Note that the default is 1.0, a float. Floats will be used to scale the actual size: The default is therefore actual_size * 1.0.

Then again, I'm not opposed to a convenience function/parameter steps_per_epoch.

[KevinMusgrave]

I was referring to the num_training_steps function:

Here is a good approximation for the total number of steps.

    @property
    def num_training_steps(self) -> int:
        """Total training steps inferred from datamodule and devices."""
        dataset = self.train_dataloader()
        if self.trainer.max_steps:
            return self.trainer.max_steps

        dataset_size = (
            self.trainer.limit_train_batches
            if self.trainer.limit_train_batches != 0
            else len(dataset)
        )

        num_devices = max(1, self.trainer.num_gpus, self.trainer.num_processes)
        if self.trainer.tpu_cores:
            num_devices = max(num_devices, self.trainer.tpu_cores)

        effective_batch_size = dataset.batch_size * self.trainer.accumulate_grad_batches * num_devices
        return (dataset_size // effective_batch_size) * self.trainer.max_epochs

[tsteffek]

Oh, that dataset_size. Same name everywhere, it's too confusing. Sorry then.

[rohitgr7]

@property
def num_training_steps(self) -> int:
    """Total training steps inferred from datamodule and devices."""
    if self.trainer.max_steps:
        return self.trainer.max_steps

    limit_batches = self.trainer.limit_train_batches
    batches = len(self.train_dataloader())
    batches = min(batches, limit_batches) if isinstance(limit_batches, int) else int(limit_batches * batches)     

    num_devices = max(1, self.trainer.num_gpus, self.trainer.num_processes)
    if self.trainer.tpu_cores:
        num_devices = max(num_devices, self.trainer.tpu_cores)

    effective_accum = self.trainer.accumulate_grad_batches * num_devices
    return (batches // effective_accum) * self.trainer.max_epochs

@tsteffek @KevinMusgrave this shall work? pls, do let me know if I missed any edge cases, will update.

[KevinMusgrave]

@rohitgr7 Thanks, that seems to work 👍