training_epoch_end called before all steps of epoch were completed. always at about 0.25 size of steps.

[ganitps]

🐛 Bug

GPU available: False, used: False
TPU available: None, using: 0 TPU cores
Validation sanity check:   0%|          | 0/2 [00:00<?, ?it/s]
  | Name                | Type                 | Params
-------------------------------------------------------------

-------------------------------------------------------------

Epoch 0:   0%|          | 0/13 [00:00<?, ?it/s] 
Epoch 0:  23%|██▎       | 3/13 [01:38<05:27, 32.75s/it, loss=4.73, v_num=7]
// training_epoch_end:  outputs = [{'loss': tensor(6.4593)}, {'loss': tensor(5.7653)}, {'loss': tensor(1.9642)}]

Validating: 0it [00:00, ?it/s]
Validating:   0%|          | 0/10 [00:00<?, ?it/s]
Epoch 0:  38%|███▊      | 5/13 [01:48<02:54, 21.78s/it, loss=4.73, v_num=7]
Epoch 0:  46%|████▌     | 6/13 [01:59<02:19, 19.91s/it, loss=4.73, v_num=7]
Epoch 0:  54%|█████▍    | 7/13 [02:10<01:51, 18.58s/it, loss=4.73, v_num=7]
Epoch 0:  62%|██████▏   | 8/13 [02:20<01:27, 17.60s/it, loss=4.73, v_num=7]
Epoch 0:  69%|██████▉   | 9/13 [02:31<01:07, 16.83s/it, loss=4.73, v_num=7]
Epoch 0:  77%|███████▋  | 10/13 [02:42<00:48, 16.21s/it, loss=4.73, v_num=7]
Epoch 0:  85%|████████▍ | 11/13 [02:52<00:31, 15.71s/it, loss=4.73, v_num=7]
Epoch 0:  92%|█████████▏| 12/13 [03:04<00:15, 15.34s/it, loss=4.73, v_num=7]
Epoch 0: 100%|██████████| 13/13 [03:15<00:00, 15.00s/it, loss=4.73, v_num=7]
Epoch 0: 100%|██████████| 13/13 [03:16<00:00, 15.15s/it, loss=4.73, v_num=7]
Epoch 1:  23%|██▎       | 3/13 [01:42<05:42, 34.24s/it, loss=3.39, v_num=7]
// training_epoch_end:  outputs = [{'loss': tensor(2.6766)}, {'loss': tensor(2.3010)}, {'loss': tensor(1.1722)}]
Epoch 1:  31%|███       | 4/13 [01:48<04:04, 27.22s/it, loss=3.39, v_num=7]
Validating: 0it [00:00, ?it/s]
Epoch 1:  38%|███▊      | 5/13 [02:02<03:15, 24.42s/it, loss=3.39, v_num=7]
Completed 6.8 MiB/327.9 MiB (48.7 KiB/s) with 2 file(s) remaining







Epoch 1: 100%|██████████| 13/13 [03:48<00:00, 17.54s/it, loss=3.39, v_num=7]
Epoch 2:  23%|██▎       | 3/13 [01:44<05:47, 34.72s/it, loss=2.72, v_num=7]
NUM EL TRAINING: 3   [{'loss': tensor(1.2504)}, {'loss': tensor(1.4905)}, {'loss': tensor(1.4158)}]
Epoch 2:  31%|███       | 4/13 [01:49<04:07, 27.48s/it, loss=2.72, v_num=7]
Validating: 0it [00:00, ?it/s]
Epoch 2: 100%|██████████| 13/13 [03:50<00:00, 17.75s/it, loss=2.72, v_num=7]
Epoch 3:  23%|██▎       | 3/13 [01:43<05:46, 34.62s/it, loss=2.27, v_num=7]
//training_epoch_end:   outputs = [{'loss': tensor(0.6632)}, {'loss': tensor(0.9215)}, {'loss': tensor(1.1396)}]
Epoch 3:  31%|███       | 4/13 [01:49<04:06, 27.41s/it, loss=2.27, v_num=7]
Validating: 0it [00:00, ?it/s]

• PyTorch Version (e.g., 1.0):
• OS (e.g., Linux): mac Catalina (this happens on all environments , linux etc)
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.7
• CUDA/cuDNN version:
• GPU models and configuration: happens also with 0 gpus.
• Any other relevant information:

[awaelchli]

As of now, training_epoch_end runs before validation starts (validation runs outside the training epoch). The progress bar just shows the combined training + validation steps. So this is fine and you get all the steps for the training epoch.

After #7357, training_epoch_end will run after the last validation loop each epoch.
Hope this clears it up :)

[ganitps]

Thanks @awaelchli for your reply
This is not what I understood form the documentation:

// the pseudocode for these calls

train_outs = []
for train_batch in train_data:
        out = training_step(train_batch)
        train_outs.append(out)
training_epoch_end(train_outs)

So If I want to take actions when epoch ends I must do it in the last training step? (training_step_end())
Can I take the latest version After #7357?

[awaelchli]

No this is still all good. The pseudo code is correct, all outputs from training_step get passed into training_epoch_end. You get as many outputs as training steps. In your case, there seems to be three training steps, so you get 3 outputs, correct? And that's when the training epoch ends and the validation starts.

[ganitps]

No... I have 13 steps.
training_epoch_end called after the third one... with 3 outputs...

[awaelchli]

No... I have 13 steps.

How do you know?
Have you printed len(dataloader)? Have you set limit_train_batches=13?
From the output you shared it just looks like you have 3 training steps and 10 validation steps.

Maybe I can help better if you try to explain what you want to achieve.

[ganitps]

Dear @awaelchli
Thank you so much for sticking with me, your last answer helped me understand that the epoch printouts doesn't reflect epoch steps but also validation. and every thing really works as designed :)