Starting prediction The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. Token is valid (permission: write). Your token has been saved to /root/.cache/huggingface/token Login successful Detected zip file Extracted zip file 2024-08-11 17:05:35,322 [WARNING] (ArgsParser) The VAE model madebyollin/sdxl-vae-fp16-fix is not compatible. Please use a compatible VAE to eliminate this warning. The baked-in VAE will be used, instead. 2024-08-11 17:05:35,322 [INFO] (ArgsParser) VAE Model: black-forest-labs/FLUX.1-dev 2024-08-11 17:05:35,322 [INFO] (ArgsParser) Default VAE Cache location: 2024-08-11 17:05:35,322 [INFO] (ArgsParser) Text Cache location: cache 2024-08-11 17:05:35,322 [WARNING] (ArgsParser) Updating T5 XXL tokeniser max length to 512 for Flux. 2024-08-11 17:05:35,467 [INFO] (__main__) Enabling tf32 precision boost for NVIDIA devices due to --allow_tf32. 2024-08-11 17:05:35,467 [INFO] (__main__) Load tokenizers You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers 2024-08-11 17:05:39,068 [INFO] (__main__) Loading OpenAI CLIP-L text encoder from black-forest-labs/FLUX.1-dev/text_encoder.. 2024-08-11 17:05:40,917 [INFO] (__main__) Loading T5 XXL v1.1 text encoder from black-forest-labs/FLUX.1-dev/text_encoder_2.. Downloading shards: 0%| | 0/2 [00:00, 'instance_data_dir': 'datasets/Replicate', 'metadata_backend': } (Rank: 0) | Bucket | Image Count (per-GPU) ------------------------------ (Rank: 0) | 1.0 | 14 2024-08-11 17:06:29,211 [INFO] (DataBackendFactory) (id=Replicate) Collecting captions. Loading captions: 0%| | 0/16 [00:00, 'instance_data_dir': 'datasets/Replicate', 'metadata_backend': , 'train_dataset': , 'sampler': , 'train_dataloader': , 'text_embed_cache': , 'vaecache': } 2024-08-11 17:06:32,027 [INFO] (validation) Precomputing the negative prompt embed for validations. 2024-08-11 17:06:32,087 [INFO] (__main__) Unloading text encoders, as they are not being trained. 2024-08-11 17:06:38,541 [INFO] (__main__) After nuking text encoders from orbit, we freed 9.11 GB of VRAM. The real memories were the friends we trained a model on along the way. Fetching 3 files: 0%| | 0/3 [00:00 Number of seen images: 0 (Rank: 0) -> Number of unseen images: 14 (Rank: 0) -> Current Bucket: None (Rank: 0) -> 1 Buckets: ['1.0'] (Rank: 0) -> 0 Exhausted Buckets: [] wandb: WARNING `resume` will be ignored since W&B syncing is set to `offline`. Starting a new run with run id ed222061d2f06bbc2a69a1b21517d221. wandb: Tracking run with wandb version 0.17.5 wandb: W&B syncing is set to `offline` in this directory. wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing. 2024-08-11 17:07:39,067 [INFO] (__main__) Moving the diffusion transformer to GPU in torch.bfloat16 precision. 2024-08-11 17:07:39,086 [INFO] (__main__) ***** Running training ***** - Num batches = 14 - Num Epochs = 65 - Current Epoch = 1 - Total train batch size (w. parallel, distributed & accumulation) = 1 - Instantaneous batch size per device = 1 - Gradient Accumulation steps = 1 - Total optimization steps = 900 - Total optimization steps remaining = 900 Epoch 1/65 Steps: 0%| | 0/900 [00:00 global_step = 900, max_train_steps = 900, epoch = 65, num_train_epochs = 65 2024-08-11 17:37:49,695 [INFO] (__main__) Exiting training loop. Beginning model unwind at epoch 65, step 900 [2024-08-11 17:37:49,710] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) df: /root/.triton/autotune: No such file or directory [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.4 [WARNING] using untested triton version (3.0.0), only 1.0.0 is known to be compatible Model weights saved in output/models/pytorch_lora_weights.safetensors wandb: - 0.000 MB of 0.000 MB uploaded wandb: wandb: wandb: Run history: wandb: epoch ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███ wandb: grad_norm ▆▁▂▇▂▆▁▅▁█▁▁▄▁▇▂▇▂▁▆▂▁▁▁▁▁▂▂▁▃▁▁▁▁▅▁▁▁▁▁ wandb: learning_rate ▁▄███████▇▇▇▇▇▇▇▇▇▇▆▆▆▆▆▆▆▆▆▆▅▅▅▅▅▅▅▄▄▄▄ wandb: optimization_loss ▆▁▄█▄▅▂▆▃▇▂▄▆▃▆▄▆▄▂▅▃▁▂▂▁▁▄▄▃▄▂▁▃▂▅▃▁▂▃▃ wandb: train_loss ▆▁▄█▄▅▂▆▃▇▂▄▆▃▆▄▆▄▂▅▃▁▂▂▁▁▄▄▃▄▂▁▃▂▅▃▁▂▃▃ wandb: train_luminance ▆█▅▅▆▇▇▆▇▇▇▁▄▁▁▅▁▆▆▆▇▅▇▆▇▆▇▆▇▅▆█▆▆▆▇▄▇▆█ wandb: wandb: Run summary: wandb: epoch 65 wandb: grad_norm 0.0249 wandb: learning_rate 0.0 wandb: optimization_loss 0.23446 wandb: train_loss 0.23446 wandb: train_luminance 189.78902 wandb: wandb: You can sync this run to the cloud by running: wandb: wandb sync /src/wandb/offline-run-20240811_170738-ed222061d2f06bbc2a69a1b21517d221 wandb: Find logs at: ./wandb/offline-run-20240811_170738-ed222061d2f06bbc2a69a1b21517d221/logs wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information. Epoch 65/65, Steps: 100%|████████████████████████████████████████| 900/900 [30:12<00:00, 2.01s/it, lr=4e-7, step_loss=0.234] adding: output/models/ (stored 0%) adding: output/models/all_text_cache_files_text-embeds.json (stored 0%) adding: output/models/all_image_files_Replicate.json (deflated 84%) adding: output/models/all_vae_cache_files_Replicate.json (stored 0%) adding: output/models/all_caption_files.json(stored 0%) adding: output/models/validation_images/ (stored 0%) adding: output/models/pytorch_lora_weights.safetensors(deflated 19%)