use warmup steps for lr scheduler, ban steps == -1

[wanchaol]

as titled, we don't want to allow steps == -1 case as it would blow up the lr scheduler

[lessw2020]

nit - don't we want 2 here if the default steps is 10 (i.e. 20% rather than make it 50%).

[tianyu-l]

Shall we add a comment here suggesting the pct should be around 20%? A user might have no idea how many steps to warm up when changing total number of training steps.

[wanchaol]

sounds good I'll update. Is 20% the rule of thumb here already? I thought this is more like a tuning thing?

[lessw2020]

it's definitely a tuning thing - I did 20% to try and be safe bc we are doing such short runs by default (10 steps!) while still showing some reasonable learning gains.
A more rigourous approach would be 2/(1−β2) training iterations for the warmup with the current linear scheduling, which is 2000 iterations assuming default AdamW. (see https://arxiv.org/abs/1910.04209).

The issue here is we want to show a reasonable curve in the short term and doing that in 10 steps or even 100 is very different than the more rigorous algo above (which can be simplified to 2k warmup iterations) or even plugged in to the lr if we wanted to get fancy and be more exact.

[tianyu-l]

Please fix the CI errors:

1. apply pre-commit
2. modify default training_steps in the test file
3. the GPU test failed because HF is down and dataset cannot be downloaded ... I guess we don't expect this to happen a lot? But for test it may make sense to use a fake dataset.

[tianyu-l]

Shall we add a comment here suggesting the pct should be around 20%? A user might have no idea how many steps to warm up when changing total number of training steps.