fix DDPMScheduler.set_timesteps() when num_inference_steps not a divisor of num_train_timesteps #1835

Joqsan · 2022-12-26T15:52:43Z

Fix the following issue:

After calling DDPMScheduler.set_timesteps(num_inference_steps), the number of timesteps doesn't match num_inference_steps , if num_train_timesteps % num_inference_steps != 0.

Example:

from diffusers import DDPMScheduler, DDIMScheduler

repo_id = "runwayml/stable-diffusion-v1-5"
ddpm = DDPMScheduler.from_pretrained(repo_id, subfolder="scheduler")
ddim = DDIMScheduler.from_pretrained(repo_id, subfolder="scheduler")  # Just to compare

num_inference_steps = 53

ddpm.set_timesteps(num_inference_steps)
len(ddpm.timesteps) == num_inference_steps  # False (56 != 53)

ddim.set_timesteps(num_inference_steps)
len(ddim.timesteps) == num_inference_steps  # True  (53 == 53)

HuggingFaceDocBuilderDev · 2022-12-26T15:57:03Z

The documentation is not available anymore as the PR was closed or merged.

* Device to use (e.g. cpu, cuda:0, cuda:1, etc.) * "cuda" if torch.cuda.is_available() else "cpu"

patil-suraj

Thanks a lot for fixing this, the PR looks good!

patil-suraj · 2022-12-27T13:59:13Z

src/diffusers/schedulers/scheduling_ddpm.py

+
+        step_ratio = self.config.num_train_timesteps // self.num_inference_steps
+        timesteps = (np.arange(0, num_inference_steps) * step_ratio).round()[::-1].copy().astype(np.int64)


Looks good!

Good for me as well!

* Width was typod as weight * Run Black

* Make xformers optional even if it is available * Raise exception if xformers is used but not available * Rename use_xformers to enable_xformers_memory_efficient_attention * Add a note about xformers in README * Reformat code style

* allow selecting precision to make DB class images addresses #1831 * add prior_generation_precision argument * correct prior_generation_precision's description Co-authored-by: Suraj Patil <[email protected]> Co-authored-by: Suraj Patil <[email protected]>

@pcuenca

* unCLIP image variation * remove prior comment re: @pcuenca * stable diffusion -> unCLIP re: @pcuenca * add copy froms re: @patil-suraj

* initial * type hints * update scheduler type hint * add to README * add example generation to README * v -> mix_factor * load scheduler from pretrained

fix versatile

…#1725) * [Dtype] Align automatic dtype * up * up * fix * re-add accelerate

Co-authored-by: Henrik Forstén <[email protected]> * update TI script * make flake happy * fix typo

* Flax: Add components function * Flax: Fix img2img and align with other pipeline * Flax: Fix PRNGKey type * Refactor strength to start_timestep * Fix preprocess images * Fix processed_images dimen * latents.shape -> latents_shape * Fix typo * Remove "static" comment * Remove unnecessary optional types in _generate * Apply doc-builder code style. Co-authored-by: Pedro Cuenca <[email protected]>

* move files a bit * more refactors * fix more * more fixes * fix more onnx * make style * upload * fix * up * fix more * up again * up * small fix * Update src/diffusers/__init__.py Co-authored-by: Pedro Cuenca <[email protected]> * correct Co-authored-by: Pedro Cuenca <[email protected]>

… passed to enable interpolation tasks. (#1858) * [Unclip] Make sure latents can be reused * allow one to directly pass embeddings * up * make unclip for text work * finish allowing to pass embeddings * correct more * make style

* Fix ema decay and clarify nomenclature. * Rename var.

* [Docs] Improve docs * up

update loss computation

patrickvonplaten · 2022-12-30T14:23:54Z

src/diffusers/schedulers/scheduling_ddim.py

            num_inference_steps (`int`):
                the number of diffusion steps used when generating samples with a pre-trained model.
        """
+        num_inference_steps = min(self.config.num_train_timesteps, num_inference_steps)


Here, I think it's actually better to throw an error if num_inference_steps > self.config.num_train_timesteps . It's usually better to educate the user instead of silently correcting a bug IMO.

More specifically, it's just wrong to pass num_inference_steps > self.config.num_train_timesteps since it's impossible for the model to handle it => so the user should understand this and correct the input.

Suggested change

num_inference_steps = min(self.config.num_train_timesteps, num_inference_steps)

if num_inference_steps > self.config.num_train_timesteps:

raise ValueError(f"`num_inference_steps`: {num_inference_steps} cannot be larger than `self.config.train_timesteps`: {self.config.num_train_steps} as the unet model trained with this scheduler can only handle maximal {self.config.num_train_steps} timesteps.")

num_inference_steps = min(self.config.num_train_timesteps, num_inference_steps)

Yep. I agree that it's better to throw an error. I'll add the suggested changes.

patrickvonplaten · 2022-12-30T14:24:49Z

Hey @Joqsan,

Thanks a lot for the PR! Could we maybe change the "silent correction" to an error message :-) Apart from this this PR looks nice!

* allow using non-ema weights for training * Apply suggestions from code review Co-authored-by: Pedro Cuenca <[email protected]> * address more review comment * reorganise a few lines * always pad text to max_length to match original training * ifx collate_fn * remove unused code * don't prepare ema_unet, don't register lr scheduler * style * assert => ValueError * add allow_tf32 * set log level * fix comment Co-authored-by: Pedro Cuenca <[email protected]>

Joqsan · 2022-12-31T13:35:09Z

Hi @patrickvonplaten

Thanks for the suggestion! I pushed a commit adding the error message to the ddpm and ddim files.

For consistency's sake I think we should add the error message to the other schedulers too, shouldn't we? If agreed, I can take a look at it.

Just not sure whether to include that in this PR or do it in a separate one...

patrickvonplaten · 2023-01-01T17:47:05Z

It seems like the git history is messed up sadly :-/ This happens sometimes when incorrectly rebasing the branch I think. I usually just merge "main" into the branches to avoid this. It's not that trivial to solve actually - I always recommend to just open a new PR and close this one (copy-paste the relevant files from your PR) - sorry about this!

fix: DDPMScheduler.set_timesteps()

f74ae6e

camenduru and others added 2 commits December 27, 2022 14:42

Device to use (e.g. cpu, cuda:0, cuda:1, etc.) (#1844)

1f1b6c6

* Device to use (e.g. cpu, cuda:0, cuda:1, etc.) * "cuda" if torch.cuda.is_available() else "cpu"

Avoid duplicating PyTorch + safetensors downloads. (#1836)

68e2425

patil-suraj approved these changes Dec 27, 2022

View reviewed changes

patil-suraj requested a review from patrickvonplaten December 27, 2022 14:02

Helw150 and others added 16 commits December 27, 2022 15:09

Width was typod as weight (#1800)

8a4c3e5

* Width was typod as weight * Run Black

fix: resize transform now preserves aspect ratio (#1804)

b693aff

unCLIP image variation (#1781)

53c8147

* unCLIP image variation * remove prior comment re: @pcuenca * stable diffusion -> unCLIP re: @pcuenca * add copy froms re: @patil-suraj

[Community Pipeline] MagicMix (#1839)

2ba42aa

* initial * type hints * update scheduler type hint * add to README * add example generation to README * v -> mix_factor * load scheduler from pretrained

[Versatile Diffusion] Fix cross_attention_kwargs (#1849)

debc74f

fix versatile

[Dtype] Align dtype casting behavior with Transformers and Accelerate (…

f2e521c

…#1725) * [Dtype] Align automatic dtype * up * up * fix * re-add accelerate

[StableDiffusionInpaint] Correct test (#1859)

03bf877

[textual inversion] add gradient checkpointing and small fixes. (#1848)

9ea7052

Co-authored-by: Henrik Forstén <[email protected]> * update TI script * make flake happy * fix typo

Fix ema decay (#1868)

a6e2c1f

* Fix ema decay and clarify nomenclature. * Rename var.

[Docs] Improve docs (#1870)

ac37384

* [Docs] Improve docs * up

[examples] update loss computation (#1861)

e4fe941

update loss computation

patrickvonplaten reviewed Dec 30, 2022

View reviewed changes

patil-suraj and others added 2 commits December 30, 2022 21:49

Throw error when num_inference_steps > num_train_timesteps

4f7000f

Joqsan added 3 commits December 31, 2022 17:09

fix: DDPMScheduler.set_timesteps()

6f6db25

Throw error when num_inference_steps > num_train_timesteps

8328fe2

Merge branch 'main' of github.com:Joqsan/diffusers into main

0cd5c36

Joqsan closed this by deleting the head repository Jan 4, 2023

Joqsan mentioned this pull request Jan 4, 2023

fix: DDPMScheduler.set_timesteps() #1912

Merged

Warvito mentioned this pull request Jan 5, 2023

Wrong number of timesteps when num_inference_steps is not a divisor of num_train_timesteps Project-MONAI/GenerativeModels#162

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix DDPMScheduler.set_timesteps() when num_inference_steps not a divisor of num_train_timesteps #1835

fix DDPMScheduler.set_timesteps() when num_inference_steps not a divisor of num_train_timesteps #1835

Uh oh!

Joqsan commented Dec 26, 2022

Uh oh!

HuggingFaceDocBuilderDev commented Dec 26, 2022 •

edited

Loading

Uh oh!

patil-suraj left a comment

Uh oh!

patil-suraj Dec 27, 2022

Uh oh!

patrickvonplaten Dec 30, 2022

Uh oh!

patrickvonplaten Dec 30, 2022

Uh oh!

Joqsan Dec 31, 2022

Uh oh!

patrickvonplaten commented Dec 30, 2022

Uh oh!

Joqsan commented Dec 31, 2022

Uh oh!

patrickvonplaten commented Jan 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants


		step_ratio = self.config.num_train_timesteps // self.num_inference_steps
		timesteps = (np.arange(0, num_inference_steps) * step_ratio).round()[::-1].copy().astype(np.int64)

fix DDPMScheduler.set_timesteps() when num_inference_steps not a divisor of num_train_timesteps #1835

fix DDPMScheduler.set_timesteps() when num_inference_steps not a divisor of num_train_timesteps #1835

Uh oh!

Conversation

Joqsan commented Dec 26, 2022

Example:

Uh oh!

HuggingFaceDocBuilderDev commented Dec 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patil-suraj left a comment

Choose a reason for hiding this comment

Uh oh!

patil-suraj Dec 27, 2022

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Dec 30, 2022

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Dec 30, 2022

Choose a reason for hiding this comment

Uh oh!

Joqsan Dec 31, 2022

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Dec 30, 2022

Uh oh!

Joqsan commented Dec 31, 2022

Uh oh!

patrickvonplaten commented Jan 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

HuggingFaceDocBuilderDev commented Dec 26, 2022 •

edited

Loading