-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Closed
Labels
needs triageWaiting to be triaged by maintainersWaiting to be triaged by maintainers
Description
🐛 Bug
TQDM Progress Bar shows negative time and speed after resuming from checkpoint
Screencast.from.07-01-2022.10.12.29.PM.webm
To Reproduce
Train any model for sometime, then resume. The progress bar will register negative numbers
Expected behavior
Resuming should NOT cause the progress bar to display negative ETA/speed
Environment
- CUDA:
- GPU:
- NVIDIA GeForce RTX 3090
- available: True
- version: 11.6 - Packages:
- numpy: 1.23.0
- pyTorch_debug: False
- pyTorch_version: 1.12.0+cu116
- pytorch-lightning: 1.6.4
- tqdm: 4.64.0 - System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.9.13
- version: fixed clip grad warning #42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022
awaelchli
Metadata
Metadata
Assignees
Labels
needs triageWaiting to be triaged by maintainersWaiting to be triaged by maintainers