Skip to content

TQDM Progress Bar shows negative time and speed after resuming from checkpoint #13493

@ductai199x

Description

@ductai199x

🐛 Bug

TQDM Progress Bar shows negative time and speed after resuming from checkpoint

Screencast.from.07-01-2022.10.12.29.PM.webm

To Reproduce

Train any model for sometime, then resume. The progress bar will register negative numbers

Expected behavior

Resuming should NOT cause the progress bar to display negative ETA/speed

Environment

  • CUDA:
    - GPU:
    - NVIDIA GeForce RTX 3090
    - available: True
    - version: 11.6
  • Packages:
    - numpy: 1.23.0
    - pyTorch_debug: False
    - pyTorch_version: 1.12.0+cu116
    - pytorch-lightning: 1.6.4
    - tqdm: 4.64.0
  • System:
    - OS: Linux
    - architecture:
    - 64bit
    - ELF
    - processor: x86_64
    - python: 3.9.13
    - version: fixed clip grad warning #42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs triageWaiting to be triaged by maintainers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions