Textual Inversion got error of cannot find config.json file

[XavierXiao]

Describe the bug

When I follow every step described here, I got the following error:

OSError: CompVis/stable-diffusion-v1-4 does not appear to have a file named config.json.

I can use inference Pipeline with no issue. It should be related to token issue, as I can use the colab with notebook_login

Reproduction

python textual_inversion.py --pretrained_model_name_or_path=$MODEL_NAME --use_auth_token  --train_data_dir=$DATA_DIR  --learnable_property="object" --placeholder_token="<cat-toy>" --initializer_token="toy" --resolution=512 --train_batch_size=1 --gradient_accumulation_steps=4 --max_train_steps=3000 --learning_rate=5.0e-04 --scale_lr --lr_scheduler="constant" --lr_warmup_steps=0  --output_dir="textual_inversion_cat" 

Logs

Traceback (most recent call last):
  File "textual_inversion.py", line 587, in <module>
    main()
  File "textual_inversion.py", line 390, in main
    text_encoder = CLIPTextModel.from_pretrained(
  File "/home/yanq/.conda/envs/ldm/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1764, in from_pretrained
    config, model_kwargs = cls.config_class.from_pretrained(
  File "/home/yanq/.conda/envs/ldm/lib/python3.8/site-packages/transformers/models/clip/configuration_clip.py", line 126, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/yanq/.conda/envs/ldm/lib/python3.8/site-packages/transformers/configuration_utils.py", line 553, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/yanq/.conda/envs/ldm/lib/python3.8/site-packages/transformers/configuration_utils.py", line 626, in _get_config_dict
    raise EnvironmentError(
OSError: CompVis/stable-diffusion-v1-4 does not appear to have a file named config.json.

System Info

• diffusers version: 0.3.0
• Platform: Linux-5.4.0-1054-aws-x86_64-with-glibc2.10
• Python version: 3.8.5
• PyTorch version (GPU?): 1.11.0 (True)
• Huggingface_hub version: 0.9.1
• Transformers version: 4.19.2
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[Oragi]

I had to git clone the CompVis/stable-diffusion-v1-4 repo and make this change to get mine to work (using a local path as --pretrained_model_name_or_path) :

    # Load models and create wrapper for stable diffusion
    text_encoder = CLIPTextModel.from_pretrained(
        args.pretrained_model_name_or_path, subfolder="text_encoder", use_auth_token=args.use_auth_token
    )
    vae = AutoencoderKL.from_pretrained(
        args.pretrained_model_name_or_path, subfolder="vae", use_auth_token=args.use_auth_token
    )
    unet = UNet2DConditionModel.from_pretrained(
        args.pretrained_model_name_or_path, subfolder="unet", use_auth_token=args.use_auth_token
    )

to

    # Load models and create wrapper for stable diffusion
    text_encoder = CLIPTextModel.from_pretrained(
        args.pretrained_model_name_or_path + "/text_encoder", use_auth_token=args.use_auth_token
    )
    vae = AutoencoderKL.from_pretrained(
        args.pretrained_model_name_or_path + "/vae", use_auth_token=args.use_auth_token
    )
    unet = UNet2DConditionModel.from_pretrained(
        args.pretrained_model_name_or_path + "/unet", use_auth_token=args.use_auth_token
    )

Seems like the subfolder="" parameter isn't working?
It loads fine on the Colab when using a hosted runtime, but if you try to run the exact same steps with a local windows jupyter runtime it gives the same issue

[patrickvonplaten]

Gently pinging @patil-suraj

[CrackedBeefcake]

might be related, I get something similar.

cwd = path.join(os.getcwd())
modelpath = 'models/ldm/stable-diffusion-v1/model.ckpt'
loadpath = path.normpath(path.join(cwd, '..', modelpath))
vae = AutoencoderKL.from_pretrained(loadpath)

leads to:

nfigMixin._dict_from_json_file(cls, json_file)
    298 with open(json_file, "r", encoding="utf-8") as reader:
--> 299     text = reader.read()
    300 return json.loads(text)

File ~/miniconda3/envs/ldm/lib/python3.8/codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
    321 data = self.buffer + input
--> 322 (result, consumed) = self._buffer_decode(data, self.errors, final)
    323 # keep undecoded input until the next call

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
/home/finn/code/lstein-diffusion/notebooks/sd_custom.ipynb Cell 4 in <cell line: 1>()
----> 1 vae = AutoencoderKL.from_pretrained(loadpath)
...
    262 except (json.JSONDecodeError, UnicodeDecodeError):
--> 263     raise EnvironmentError(f"It looks like the config file at '{config_file}' is not a valid JSON file.")
    265 return config_dict

OSError: It looks like the config file at '/home/finn/code/lstein-diffusion/models/ldm/stable-diffusion-v1/model.ckpt' is not a valid JSON file.

[2132660698]

try this:
pip install transformers==4.22.1

[patil-suraj]

Hey @XavierXiao , @Oragi ! IAs pointed out by @2132660698 if you upgrade transformers version. to >=4.22.0 this should work. Because the subfolder argument isn't available in previous versions. I will update the minimum version in example requirements.

@FinnFrotscher It looks like you are using the compvis checkpoint, you should clone the diffusers checkpoint to use it with this library. https://huggingface.co/CompVis/stable-diffusion-v1-4

[s3rgeym]

Блять, че я должен сделать чтобы эта хуета работала?

pip install transformers==4.22.0

Traceback (most recent call last):
  File "/home/sergey/workspace/test-diffusers/./image_gen.py", line 5, in <module>
    from diffusers import DiffusionPipeline
  File "/home/sergey/workspace/test-diffusers/.venv/lib/python3.10/site-packages/diffusers/__init__.py", line 55, in <module>
    from .pipelines import (
  File "/home/sergey/workspace/test-diffusers/.venv/lib/python3.10/site-packages/diffusers/pipelines/__init__.py", line 18, in <module>
    from .dance_diffusion import DanceDiffusionPipeline
  File "/home/sergey/workspace/test-diffusers/.venv/lib/python3.10/site-packages/diffusers/pipelines/dance_diffusion/__init__.py", line 1, in <module>
    from .pipeline_dance_diffusion import DanceDiffusionPipeline
  File "/home/sergey/workspace/test-diffusers/.venv/lib/python3.10/site-packages/diffusers/pipelines/dance_diffusion/pipeline_dance_diffusion.py", line 21, in <module>
    from ..pipeline_utils import AudioPipelineOutput, DiffusionPipeline
  File "/home/sergey/workspace/test-diffusers/.venv/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 64, in <module>
    from transformers.utils import SAFE_WEIGHTS_NAME as TRANSFORMERS_SAFE_WEIGHTS_NAME
ImportError: cannot import name 'SAFE_WEIGHTS_NAME' from 'transformers.utils' (/home/sergey/workspace/test-diffusers/.venv/lib/python3.10/site-packages/transformers/utils/__init__.py)

pip install -U transformers

OSError: It looks like the config file at '/home/sergey/workspace/test-diffusers/CounterfeitV25_25.safetensors' is not a valid JSON file.

Мудацкое говно

[s3rgeym]

#!/usr/bin/env python
import argparse
from pathlib import Path

from diffusers import DiffusionPipeline

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("prompt", nargs="+")
    parser.add_argument(
        "-o", "--output", default="output.png", type=argparse.FileType("wb+")
    )
    args = parser.parse_args()
    print(args)
    safetensors_path = Path(__file__).parent / "CounterfeitV25_25.safetensors"
    pipe = DiffusionPipeline.from_pretrained(safetensors_path)
    pipe.to("cuda")
    img, = pipe(" ".join(args.prompt)).images
    img.save(args.output)