[dreambooth] low precision guard

[williamberman]

re: #1246

training/fine tuning shouldn't be done with fp16 weights¹, fp16 inputs are ok with amp + gradient scaling. fp16 weights throw an error when used with amp + gradient scaling. We should check the dtype of the loaded model and throw an informative error before training begins.

We add a guard checking the datatype of the unet. We also add a guard checking the datatype of the text encoder if we are training the text encoder.

Footnotes

1. precision issues when adding small gradient updates to fp16 weights. Reason why training with amp recommends to keep weights as fp32 for gradient updates and makes a copy in half precision for forward and backward passes. ↩

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[williamberman]

@patil-suraj I added docs to the relevant cli flags. Are there any other additional locations we should update docs around loaded model precision?

[patrickvonplaten]

Thanks a lot!

[patil-suraj]

LGTM!

[patil-suraj]

@patil-suraj I added docs to the relevant cli flags. Are there any other additional locations we should update docs around loaded model precision?

No, I think this covers it :)

[franz101]

Am I missing something from the readme?
Getting the following errors.
UnfilteredStackTrace: TypeError: add_noise() takes 4 positional arguments but 5
or
TypeError: get_scheduler() got an unexpected keyword argument 'num_cycles'

@patil-suraj @patrickvonplaten

Or is it something related to the readme?

Will try out the solution of #1817