Unconditional Image Generation generating noise

[agneet42]

Hi, As mentioned in https://huggingface.co/docs/diffusers/training/unconditional_training, I am trying to fine-tune on the CartoonSet[https://google.github.io/cartoonset/] dataset. However, after training for 100 epochs, on inspecting via Tensorboard, I still see only noise being generated.

Dataset stats -
10k images, 500*500 RGB images

Script/params -

accelerate launch train_unconditional.py \
--train_data_dir=/home/achatt39/cartoonset10k/training_images \
--resolution=64 \
--output_dir="ddpm-ema-cartoons-64_2" \
--train_batch_size=16 \
--num_epochs=100 \
--gradient_accumulation_steps=1 \
--learning_rate=1e-4 \
--lr_warmup_steps=500 \
--mixed_precision=no

Logs attached -
logs.zip

Looking for any assistance on what might be the issue!

[patrickvonplaten]

Hmm, I'm not really sure here @agneet42

It could just be that this dataset is difficult to fit. Gently pinging @anton-l here in case you have an idea :-)

[PKURio]

I have met the same issue.
Using my dataset, I get some noise after 1000epochs using ema. Disable ema, and I can generate some meaningful images after 100epoch.

[PKURio]

I have met the same issue. Using my dataset, I get some noise after 1000epochs using ema. Disable ema, and I can generate some meaningful images after 100epoch.

But from now, I don't know why ema is not working here (maybe somg bugs?). Using openai's guided diffusion Repo, I can get really good results using ema.

[anton-l]

@agneet42 @PKURio does the provided flowers example work for you?
I still get reasonable results when running it with EMA enabled: https://huggingface.co/anton-l/ddpm-ema-flowers-64-testt/tensorboard?scroll=1#images

[SuwoongHeo]

I have a similar problem on train_uncoditional.py. I can see that the model seems "train" when the training code is launched without the accelerate. Here is the test result of the first epoch both on multi(using accelerate, left) and single GPU (right) cases. But I don't know what causes this difference...

[PKURio]

@agneet42 @PKURio does the provided flowers example work for you? I still get reasonable results when running it with EMA enabled: https://huggingface.co/anton-l/ddpm-ema-flowers-64-testt/tensorboard?scroll=1#images

Thanks for reply. But the answer is no... I got noise just like the left image shown by @SuwoongHeo

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[anton-l]

The issue should be solved by #1930 now :)