Add SEGA for DiTs (SD3, AurraFlow, Hunyuan, FLUX) #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Sayak Paul <[email protected]>
…s. (huggingface#8692) create a utility for calculating the expected number of shards.
…huggingface#8630) * add clip text-encoder training * no dora * text encoder traing fixes * text encoder traing fixes * text encoder training fixes * text encoder training fixes * text encoder training fixes * text encoder training fixes * add text_encoder layers to save_lora * style * fix imports * style * fix text encoder * review changes * review changes * review changes * minor change * add lora tag * style * add readme notes * add tests for clip encoders * style * typo * fixes * style * Update tests/lora/test_lora_layers_sd3.py Co-authored-by: Sayak Paul <[email protected]> * Update examples/dreambooth/README_sd3.md Co-authored-by: Sayak Paul <[email protected]> * minor readme change --------- Co-authored-by: YiYi Xu <[email protected]> Co-authored-by: Sayak Paul <[email protected]>
* first draft --------- Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: Junhwa Song <[email protected]> Co-authored-by: Ahn Donghoon (안동훈 / suno) <[email protected]> Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: Steven Liu <[email protected]>
* doc for max_sequence_length * better position and changed note to tip * apply suggestions --------- Co-authored-by: Sayak Paul <[email protected]>
add conversion script
* add docs on model sharding * add entry to _toctree. * Apply suggestions from code review Co-authored-by: Steven Liu <[email protected]> * simplify wording * add a note on transformer library handling * move device placement section * Update docs/source/en/training/distributed_inference.md Co-authored-by: Steven Liu <[email protected]> --------- Co-authored-by: Steven Liu <[email protected]>
* update * update --------- Co-authored-by: Sayak Paul <[email protected]>
* add more about from_pipe API * Update docs/source/en/using-diffusers/pag.md * Update docs/source/en/using-diffusers/pag.md --------- Co-authored-by: yiyixuxu <yixu310@gmail,com>
…ace#8694) * add controlnet support --------- Co-authored-by: xingchaoliu <[email protected]> Co-authored-by: yiyixuxu <yixu310@gmail,com>
…lines. (huggingface#8676) * add reporting mechanism when mirroring community pipelines. * remove unneeded argument * get the actual PATH_IN_REPO * don't need tag
…#8699) * fix: unet save_attn_procs at custom diffusion * style: recover unchanaged parts(max line length 119) / mod: add condition * style: recover unchanaged parts(max line length 119) --------- Co-authored-by: Sayak Paul <[email protected]>
…ass. (huggingface#8698) * remove deprecation from transformer2d regarding the output class. * up * deprecate more
fix vanilla fine-tuned lora loading.
…ingface#8688) fix conversion utility so that lora dora loads correctly
* modify PR and issue templates * add single file poc.
…face#8718) add some info when there is an error.
* initial fix * apply suggestion * delete step_index line
* Add check for WindowsPath in to_json_string On Windows, os.path.join returns a WindowsPath. to_json_string does not convert this from a WindowsPath to a string. Added check for WindowsPath to to_json_saveable. * Remove extraneous convert to string in test_check_path_types (tests/others/test_config.py) * Fix style issues in tests/others/test_config.py * Add unit test to test_config.py to verify that PosixPath and WindowsPath (depending on system) both work when converted to JSON * Remove distinction between PosixPath and WindowsPath in ConfigMixIn.to_json_string(). Conditional now tests for Path, and uses Path.as_posix() to convert to string. --------- Co-authored-by: Vincent Dovydaitis <[email protected]> Co-authored-by: Sayak Paul <[email protected]>
add pag sd3 --------- Co-authored-by: HyoungwonCho <[email protected]> Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: crepejung00 <[email protected]> Co-authored-by: YiYi Xu <[email protected]> Co-authored-by: Aryan <[email protected]> Co-authored-by: Aryan <[email protected]>
* Fix loading sharded checkpoint when we have variant * add test * remote print --------- Co-authored-by: Sayak Paul <[email protected]>
…ce#9083) * update * update * update --------- Co-authored-by: Sayak Paul <[email protected]>
* txt2img pag added * autopipe added, fixed case * style * apply suggestions * added fast tests, added todo tests * revert dummy objects for kolors * fix pag dummies * fix test imports * update pag tests * add kolor pag to docs --------- Co-authored-by: Sayak Paul <[email protected]>
* initial work draft for freenoise; needs massive cleanup * fix freeinit bug * add animatediff controlnet implementation * revert attention changes * add freenoise * remove old helper functions * add decode batch size param to all pipelines * make style * fix copied from comments * make fix-copies * make style * copy animatediff controlnet implementation from huggingface#8972 * add experimental support for num_frames not perfectly fitting context length, ocntext stride * make unet motion model lora work again based on huggingface#8995 * copy load video utils from huggingface#8972 * copied from AnimateDiff::prepare_latents * address the case where last batch of frames does not match length of indices in prepare latents * decode_batch_size->vae_batch_size; batch vae encode support in animatediff vid2vid * revert sparsectrl and sdxl freenoise changes * revert pia * add freenoise tests * make fix-copies * improve docstrings * add freenoise tests to animatediff controlnet * update tests * Update src/diffusers/models/unets/unet_motion_model.py * add freenoise to animatediff pag * address review comments * make style * update tests * make fix-copies * fix error message * remove copied from comment * fix imports in tests * update --------- Co-authored-by: Dhruv Nair <[email protected]>
* clipping for fp16 * fix typo * added fp16 inference to docs * fix docs typo * include link for fp16 investigation --------- Co-authored-by: Sayak Paul <[email protected]>
* allow sparsectrl to be loaded with single file * update --------- Co-authored-by: Dhruv Nair <[email protected]>
…#9110) * update * update
* add CogVideoX --------- Co-authored-by: Aryan <[email protected]> Co-authored-by: sayakpaul <[email protected]> Co-authored-by: Aryan <[email protected]> Co-authored-by: yiyixuxu <[email protected]> Co-authored-by: Steven Liu <[email protected]>
* toctree * fix
* fix for lr scheduler in distributed training * Fixed the recalculation of the total training step section * Fixed lint error --------- Co-authored-by: Sayak Paul <[email protected]>
Co-authored-by: Aryan <[email protected]>
* Add Differential Pipeline. * Fix Styling Issue using ruff -fix * Add details to Contributing.md * Revert "Fix Styling Issue using ruff -fix" This reverts commit d347de1. * Revert "Revert "Fix Styling Issue using ruff -fix"" This reverts commit ce7c3ff. * Revert README changes * Restore README.md * Update README.md * Resolved Comments: * Fix Readme based on review * Fix formatting after make style --------- Co-authored-by: Aryan <[email protected]>
Co-authored-by: YiYi Xu <[email protected]>
* initial commit - dreambooth for flux * update transformer to be FluxTransformer2DModel * update training loop and validation inference * fix sd3->flux docs * add guidance handling, not sure if it makes sense(?) * inital dreambooth lora commit * fix text_ids in compute_text_embeddings * fix imports of static methods * fix pipeline loading in readme, remove auto1111 docs for now * fix pipeline loading in readme, remove auto1111 docs for now, remove some irrelevant text_encoder_3 refs * Update examples/dreambooth/train_dreambooth_flux.py Co-authored-by: Bagheera <[email protected]> * fix te2 loading and remove te2 refs from text encoder training * fix tokenizer_2 initialization * remove text_encoder training refs from lora script (for now) * try with vae in bfloat16, fix model hook save * fix tokenization * fix static imports * fix CLIP import * remove text_encoder training refs (for now) from lora script * fix minor bug in encode_prompt, add guidance def in lora script, ... * fix unpack_latents args * fix license in readme * add "none" to weighting_scheme options for uniform sampling * style * adapt model saving - remove text encoder refs * adapt model loading - remove text encoder refs * initial commit for readme * Update examples/dreambooth/train_dreambooth_lora_flux.py Co-authored-by: Sayak Paul <[email protected]> * Update examples/dreambooth/train_dreambooth_lora_flux.py Co-authored-by: Sayak Paul <[email protected]> * fix vae casting * remove precondition_outputs * readme * readme * style * readme * readme * update weighting scheme default & docs * style * add text_encoder training to lora script, change vae_scale_factor value in both * style * text encoder training fixes * style * update readme * minor fixes * fix te params * fix te params --------- Co-authored-by: Bagheera <[email protected]> Co-authored-by: Sayak Paul <[email protected]>
…ingface#9010) * Fix textual inversion SDXL and add support for 2nd text encoder Signed-off-by: Daniel Socek <[email protected]> * Fix style/quality of text inv for sdxl Signed-off-by: Daniel Socek <[email protected]> --------- Signed-off-by: Daniel Socek <[email protected]> Co-authored-by: Sayak Paul <[email protected]>
* resolve peft links * fuse_lora
# Conflicts: # src/diffusers/__init__.py
# Conflicts: # src/diffusers/__init__.py
Collaborator
Author
Semantic sd3
Add SEGA to AuraFlow
Marlon154
changed the title
Collaborator
Author
|
suggest to first bump main before merging |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
59 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
add sega implementation for FLUX and HunyuanDiT