multiple prediction options in ddpm, ddim #818

natolambert · 2022-10-13T00:26:17Z

starting to work on discussion in #778.
Please contribute and leave feedback. This is mostly a placeholder for my work right now as I figure out how to do it.

Some relevant repositories:

HuggingFaceDocBuilderDev · 2022-10-13T00:30:03Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

patrickvonplaten · 2022-10-14T17:40:44Z

Yes we indeed need this now I think :-) (also for dance diffusion)

src/diffusers/schedulers/scheduling_ddpm.py

patrickvonplaten

Generally, this looks good to me :-) We'll definitely need tests here though

natolambert · 2022-10-14T18:20:30Z

I want to make a colab comparing the prediction for training on one scheduler to start (make sure it works).
Then we need to make it on more schedulers. Though if I get DDPM done this weekend, it would be good first issues for people to add and test it on other schedulers.

aandyw · 2022-10-14T20:48:42Z

I'm new to contributing and so I'm a little confused about what I should be doing. Should I clone the changes and make a colab to compare with original predictions?

patrickvonplaten · 2022-10-17T17:37:03Z

Hey @pie31415,

Since you mentioned you were interested in this PR, I think it'd be super useful to do a PR review here :-)

natolambert · 2022-10-18T18:24:12Z

@pie31415 Another really useful thing would be to just verify the implementation from the original papers and links above. This is a pretty tricky port so I will do this too, but it would be hugely useful.

For example, I actually think the DDIM implementation is much closer than DDPM.

* v diffusion support for ddpm * quality and style * variable name consistency * missing base case * pass prediction type along in the pipeline * put prediction type in scheduler config * style

natolambert · 2022-11-09T19:42:47Z

FYI here are some examples for DDPM butterfly generations from @bglick13 , we want to see similar results on DDIM than maybe merge this initial version.

patrickvonplaten · 2022-11-15T21:52:21Z

src/diffusers/schedulers/scheduling_ddim.py

        model_output: torch.FloatTensor,
        timestep: int,
        sample: torch.FloatTensor,
+        prediction_type: str = "epsilon",


Think this should go in the __init__ function and we've somewhat settled on predict_epsilon: bool I think in terms of naming :-)

Ah ok now we actually have three types, so we might have to reconsider this choice 😅

But I think it should definitely go in the config of the scheduler and not be an arg of __call__

I actually messaged Nathan about my progress on DDIM v prediction in a separate branch as you commented - crazy timing!

I'll make sure I make these changes in my branch before opening a PR

patrickvonplaten · 2022-11-15T21:54:14Z

That's very interesting here actually - @patil-suraj @anton-l could you also take a look? :-)

bglick13 · 2022-11-15T22:07:04Z

DDIM will hopefully be ready for review soon too.

Results training on it are still a little pixelated, but you can clearly see the shape of a butterfly. I'm guessing I have something not quite right with the variance calculation. Will hopefully have updates here soon!

* v diffusion support for ddpm * quality and style * variable name consistency * missing base case * pass prediction type along in the pipeline * put prediction type in scheduler config * style * try to train on ddim * changes to ddim * ddim v prediction works to train butterflies example * fix bad merge, style and quality * try to fix broken doc strings * second pass * one more * white space * Update src/diffusers/schedulers/scheduling_ddim.py * remove extra lines * Update src/diffusers/schedulers/scheduling_ddim.py Co-authored-by: Ben Glickenhaus <[email protected]> Co-authored-by: Nathan Lambert <[email protected]>

natolambert · 2022-11-17T18:28:47Z

Update for the diffusers team (@patrickvonplaten , @anton-l , @patil-suraj ). We updated DDIM now (promising results), and I'll add tests / fix merge issues this afternoon.

natolambert · 2022-11-17T22:52:55Z

@patrickvonplaten this should be go to go. Now, this leaves only DPMSolverMultistepSchedulerTest with predict_epsilon rather than prediction_type, but I think it is okay until v-prediction expands in the library.

Lots more good work from @bglick13

patrickvonplaten · 2022-11-20T18:42:31Z

src/diffusers/schedulers/scheduling_ddim.py

        set_alpha_to_one: bool = True,
+        variance_type: str = "fixed",
        steps_offset: int = 0,
+        prediction_type: Literal["epsilon", "sample", "velocity"] = "epsilon",


Note we currently have a config parameter called predict_epsilon that is already used in multiple schedulers:

diffusers/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py

Line 131 in ab1f01e

predict_epsilon: bool = True,

So we cannot really add this prediciton_type here without deprecating the other one and also deprecating arguments like this one:

diffusers/examples/unconditional_image_generation/train_unconditional.py

Line 197 in ab1f01e

"--predict_epsilon",

pcuenca

LGTM. Same comments about deprecating predict_epsilon everywhere.

pcuenca · 2022-11-23T13:56:18Z

src/diffusers/schedulers/scheduling_ddpm.py

+def expand_to_shape(input, timesteps, shape, device):
+    """
+    Helper indexes a 1D tensor `input` using a 1D index tensor `timesteps`, then reshapes the result to broadcast
+    nicely with `shape`. Useful for parallelizing operations over `shape[0]` number of diffusion steps at once.
+    """
+    out = torch.gather(input.to(device), 0, timesteps.to(device))
+    reshape = [shape[0]] + [1] * (len(shape) - 1)
+    out = out.reshape(*reshape)
+    return out
+
+


How do we feel about moving this to scheduling_utils.py? Maybe get_alpha_sigma as well.

I'm good with this. I had this as a TODO in my mind. Could also be made more elegant, but wasn't 100% sure how yet.

My only concern with these two is

expand_to_shape would be the only function like this. It's okay to start the trend.

get_alpha_sigma won't work with many of the schedulers, so I'm okay with leaving it in the ones that use v-prediction for now.

src/diffusers/schedulers/scheduling_ddpm.py

Co-authored-by: Pedro Cuenca <[email protected]>

natolambert · 2022-11-23T20:23:04Z

Added more deprecating across the board. I tried to address @patrickvonplaten's comment above, but would like a double check on that!

natolambert · 2022-11-23T20:32:43Z

src/diffusers/schedulers/scheduling_ddpm.py

+            )
+
+        # not check on predict_epsilon for depreciation flag above
+        elif self.prediction_type == "sample" or not self.config.predict_epsilon:


These if statement's I had to mess with a little bit to get the tests to pass. All will be much cleaner when its deprecated.

natolambert · 2022-11-24T01:40:47Z

The code isn't as clear, but you can see some details on model parametrization in the SD 2.0 code here.

The option parametrization = "v" doesn't show up in the schedulers, though (search query here).

noise prediction function is here
not much in the ddim / ddpm files, which is confusing.
I asked on twitter for clarity.

@patil-suraj @patrickvonplaten @bglick13
(see y'all after thanksgiving)

natolambert · 2022-12-05T23:06:40Z

Closing this as the changes were integrated into #1505 and other earlier PRs.
The added type of v-prediction for DDIM will be addressed in #1010.

init v-pred pr

798263f

natolambert added 3 commits October 12, 2022 17:32

placeholder code

b7d0c1e

up

7eb4bfa

a few more additions

3eb2593

patrickvonplaten reviewed Oct 14, 2022

View reviewed changes

src/diffusers/schedulers/scheduling_ddpm.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Oct 14, 2022

View reviewed changes

natolambert mentioned this pull request Oct 14, 2022

Add v-prediction #778

Closed

add ddim

4c68504

natolambert and others added 2 commits October 18, 2022 11:42

style

ac6be90

DDPM changes to support v diffusion (#1121)

f00d896

* v diffusion support for ddpm * quality and style * variable name consistency * missing base case * pass prediction type along in the pipeline * put prediction type in scheduler config * style

natolambert changed the title ~~[WIP] multiple prediction options in schedulers~~ multiple prediction options in ddpm, ddim Nov 9, 2022

natolambert added 2 commits November 9, 2022 11:50

Merge branch 'main' into v_prediction

8fe2ff4

quality

56164f5

patrickvonplaten reviewed Nov 15, 2022

View reviewed changes

patrickvonplaten requested review from anton-l and patil-suraj November 15, 2022 21:54

patrickvonplaten assigned anton-l and patil-suraj Nov 15, 2022

fix tests

e391983

natolambert added 3 commits November 17, 2022 14:47

Merge branch 'main' into v_prediction

5a509db

add ddim pred type test

3adf87b

style

c1a0584

natolambert added 2 commits November 17, 2022 14:56

change name from v to velocity

e701a97

fix loose comments

172b242

patrickvonplaten reviewed Nov 20, 2022

View reviewed changes

pcuenca reviewed Nov 23, 2022

View reviewed changes

src/diffusers/schedulers/scheduling_ddpm.py Outdated Show resolved Hide resolved

Nathan Lambert and others added 3 commits November 23, 2022 07:42

Update src/diffusers/schedulers/scheduling_ddpm.py

66951ec

Co-authored-by: Pedro Cuenca <[email protected]>

move expand_to_shape

b70f6cd

remove Literal, add deprecates

da5e677

natolambert commented Nov 23, 2022

View reviewed changes

Merge branch 'main' into v_prediction

79ec3a8

natolambert mentioned this pull request Dec 1, 2022

support v prediction in other schedulers #1505

Merged

natolambert closed this Dec 5, 2022

leopoldmaillard mentioned this pull request Dec 13, 2022

Question about noise schedule parametrization & samplers #1656

Closed

multiple prediction options in ddpm, ddim #818

multiple prediction options in ddpm, ddim #818

Uh oh!

Conversation

natolambert commented Oct 13, 2022

Uh oh!

HuggingFaceDocBuilderDev commented Oct 13, 2022

Uh oh!

patrickvonplaten commented Oct 14, 2022

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

natolambert commented Oct 14, 2022

Uh oh!

aandyw commented Oct 14, 2022

Uh oh!

patrickvonplaten commented Oct 17, 2022

Uh oh!

natolambert commented Oct 18, 2022

Uh oh!

natolambert commented Nov 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Nov 15, 2022

Uh oh!

bglick13 commented Nov 15, 2022

Uh oh!

natolambert commented Nov 17, 2022

Uh oh!

natolambert commented Nov 17, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

natolambert commented Nov 23, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natolambert commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

natolambert commented Dec 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

natolambert commented Nov 24, 2022 •

edited

Loading