what is the meaning of parameter: "num_class_images"

[himmetozcan]

What is the num_class_images parameter used for? I see that in some examples it is 50, sometimes it is 200. In the source code it is said that: "Minimal class images for prior preservation loss. If not have enough images, additional images will be sampled with class_prompt."
I still do not fully grasp it. For example if I have 20 images to train, what should I select this "num_class_images"?

[patrickvonplaten]

Hey @himmetozcan,

Could you please link to a code line where we use the name "num_class_images"?
It's hard to guess what you are referring to here - a training script?

[averad]

@patrickvonplaten I believe @himmetozcan is referring to Dreambooth training

Example Code:

export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export INSTANCE_DIR="path_to_training_images"
export CLASS_DIR="path_to_class_images"
export OUTPUT_DIR="path_to_saved_model"

accelerate launch train_dreambooth.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --class_data_dir=$CLASS_DIR \
  --output_dir=$OUTPUT_DIR \
  --with_prior_preservation --prior_loss_weight=1.0 \
  --instance_prompt="a photo of sks dog" \
  --class_prompt="a photo of dog" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --num_class_images=200 \
  --max_train_steps=800

If we look at train_dreambooth.py on line 84 we see the following about num_class_images:

    parser.add_argument("--prior_loss_weight", type=float, default=1.0, help="The weight of prior preservation loss.")
    parser.add_argument(
        "--num_class_images",
        type=int,
        default=100,
        help=(
            "Minimal class images for prior preservation loss. If not have enough images, additional images will be"
            " sampled with class_prompt."
        ),
    )

[himmetozcan]

@patrickvonplaten I believe @himmetozcan is referring to Dreambooth training

Example Code:

export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export INSTANCE_DIR="path_to_training_images"
export CLASS_DIR="path_to_class_images"
export OUTPUT_DIR="path_to_saved_model"

accelerate launch train_dreambooth.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --class_data_dir=$CLASS_DIR \
  --output_dir=$OUTPUT_DIR \
  --with_prior_preservation --prior_loss_weight=1.0 \
  --instance_prompt="a photo of sks dog" \
  --class_prompt="a photo of dog" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --num_class_images=200 \
  --max_train_steps=800

If we look at train_dreambooth.py on line 84 we see the following about num_class_images:

    parser.add_argument("--prior_loss_weight", type=float, default=1.0, help="The weight of prior preservation loss.")
    parser.add_argument(
        "--num_class_images",
        type=int,
        default=100,
        help=(
            "Minimal class images for prior preservation loss. If not have enough images, additional images will be"
            " sampled with class_prompt."
        ),
    )

I am sorry for the confusion. That is right, I am referring to the num_class_images parameter in the training script.

[patrickvonplaten]

@patil-suraj probably knows :-)

[williamberman]

hey @himmetozcan thanks for the issue!

The num_class_images parameter controls the number of images that are used for training with the prior preservation loss. The original paper provides a better explanation than I could give of training with the prior preservation loss :) https://dreambooth.github.io/

in parallel, we apply a class-specific prior preservation loss, which leverages the semantic prior that the model has on the class and encourages it to generate diverse instances belong to the subject's class by injecting the class name in the text prompt (e.g., "A photo of a dog”).

I added some additional docs here about how it specifically impacts the training script that we'll merge in #1508

Does that clear things up, or could we provide additional clarity anywhere?

[himmetozcan]

Thanks, @williamberman for the details. Is it possible to define some examples such as if I have 20 images to train, what should I select this "num_class_images"?

[williamberman]

@himmetozcan So you could include your 20 images in the data directory but you would still want to set num_class_images as 200-300 and let the model generate the additional needed images. I think the updated docs cover this case where there are some initial images but we still want to generate the additional needed images :)

I'm going to close the issue for now but @himmetozcan please feel free to follow up with any other questions either here if there's still some confusion around num_class_images or on another issue if on another topic!