fast-kohya-trainer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/fast-kohya-trainer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BHxO7OWcixGI"
   },
   "source": [
    "![visitors](https://visitor-badge.glitch.me/badge?page_id=linaqruf.fast-kohya-trainer) [![ko-fi](https://img.shields.io/badge/Support%20me%20on%20Ko--fi-F16061?logo=ko-fi&logoColor=white&style=flat)](https://ko-fi.com/linaqruf) <a href=\"https://saweria.co/linaqruf\"><img alt=\"Saweria\" src=\"https://img.shields.io/badge/Saweria-7B3F00?style=flat&logo=ko-fi&logoColor=white\"/></a>\n",
    "\n",
    "# **Fast Kohya Trainer** (Deprecated) <small></small><br><small><small>All Kohya Training Script in 1-click cell</small></small>\n",
    "\n",
    "Only planned to fix bugs from now on.\n",
    "<details>\n",
    "  <summary><big>Feature</big></summary>\n",
    "<ul>\n",
    "  <li>1-click cell</li>\n",
    "  <li>Support both <code>LoRA</code> and <code>Native Training</code></li>\n",
    "  <li>Support <code>Dreambooth</code> method training</li>\n",
    "  <li>Support load dataset from<code>Google Drive</code></li>\n",
    "  <li>Xformers precompiled wheels available up to<code>A100</code></li>\n",
    "  <li>Support input custom tag</li>\n",
    "  <li>Faster download for pretrained model and vae using <code>aria2c 16 threading</code></li>\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary><big>Limitation</big></summary>\n",
    "<ul>\n",
    "  <li>Not Flexible</li>\n",
    "  <li>Currently registering diffusers model for pretrained model is not supported</li>\n",
    "  <li>Textual Inversion Training Script aren't included</li>\n",
    "  <li>Doesn't support multi-styled training</li>\n",
    "  <li>Don't expect much from this version because most of important features are filtered out</li>\n",
    "\n",
    "</ul>\n",
    "\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary><big>Whats new?</big></summary>\n",
    "<ul>\n",
    "  <li>(26/02):</li>\n",
    "  <ul>\n",
    "    <li>Fix missing <code>=</code> causing some variable not passed to accelerate</li>\n",
    "    <li>Fix inappropriate linebreak <code>\\</code> causing some variable not passed to accelerate</li>\n",
    "    <li>Fix missing comma causing prettytable can't print the table</li>\n",
    "    <li>If any file endswith .txt or .caption don't start WD 14 tagger or BLIP</li>\n",
    "  </ul>\n",
    "  <li>(25/02):</li>\n",
    "  <ul>\n",
    "    <li>Added Noise Offset</li>\n",
    "    <li>Added Optimizer Section for selecting optimizer, learning rate, scheduler, etc</li>\n",
    "    <li>Added lowram</li>\n",
    "  </ul>\n",
    "  <li>(13/02):</li>\n",
    "  <ul>\n",
    "    <li>Now you can train with <code>Dreambooth</code> method, but it's too complicated because a folder named <code>&lt;repeats&gt;_&lt;token&gt;</code> will created inside your <code>train_data_dir</code> and moved all of your files to that folder. So it's not recommended.</li>\n",
    "  </ul>\n",
    "  <li>(11/02):</li>\n",
    "  <ul>\n",
    "    <li>Fix <code>broken path</code> problem, like <code>/content/drive/</code> to <code>/content/drive/MyDrive/</code></li>\n",
    "    <li>Deleted advanced args like <code>lr_scheduler_num_cycles</code>, <code>lr_scheduler_power</code>, and <code>network_train_on</code>. You can specify it in <code>additional arguments</code> field instead.</li>\n",
    "    <li>Now support downloading compressed dataset (<code>.zip</code>) and automatically extract it to <code>train_data_dir</code></li>\n",
    "    <li>It's <code>2-in-1</code> feature so you can download-and-extract zipfile from <code>url</code> or only extract zipfile from mounted gdrive <code>path</code></li>\n",
    "  </ul>\n",
    "</ul>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gZ1WxKjp_1-L"
   },
   "source": [
    "| Notebook Name | Description | Link | Old Commit |\n",
    "| --- | --- | --- | --- |\n",
    "| [Kohya LoRA Dreambooth](https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb) | LoRA Training (Dreambooth method) | [![](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=5c5c5c&color=0f80c1&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb) | [![](https://img.shields.io/static/v1?message=Oldest%20Version&logo=googlecolab&labelColor=5c5c5c&color=e74c3c&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/bc0892647cb17492a106ad1d05716e091eda13f6/kohya-LoRA-dreambooth.ipynb) | \n",
    "| [Kohya LoRA Fine-Tuning](https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-finetuner.ipynb) | LoRA Training (Fine-tune method) | [![](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=5c5c5c&color=0f80c1&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-finetuner.ipynb) | [![](https://img.shields.io/static/v1?message=Oldest%20Version&logo=googlecolab&labelColor=5c5c5c&color=e74c3c&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/fb96280233d3434819ba5850b2c968150c4720f7/kohya-LoRA-finetuner.ipynb) | \n",
    "| [Kohya Trainer](https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-trainer.ipynb) | Native Training | [![](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=5c5c5c&color=0f80c1&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/kohya-trainer.ipynb) | [![](https://img.shields.io/static/v1?message=Oldest%20Version&logo=googlecolab&labelColor=5c5c5c&color=e74c3c&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/21ad4942a917d3fd1ad6c03d87d16677b427254b/kohya-trainer.ipynb) | \n",
    "| [Kohya Dreambooth](https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-dreambooth.ipynb) | Dreambooth Training | [![](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=5c5c5c&color=0f80c1&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/kohya-dreambooth.ipynb) | [![](https://img.shields.io/static/v1?message=Oldest%20Version&logo=googlecolab&labelColor=5c5c5c&color=e74c3c&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/9c7f891981bee92cc7690f2094f892c46feb99e2/kohya-dreambooth.ipynb) | \n",
    "| [Fast Kohya Trainer](https://github.com/Linaqruf/kohya-trainer/blob/main/fast-kohya-trainer.ipynb) `NEW`| Easy 1-click LoRA & Native Training| [![](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=5c5c5c&color=0f80c1&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/fast-kohya-trainer.ipynb) |\n",
    "| [Cagliostro Colab UI](https://github.com/Linaqruf/sd-notebook-collection/blob/main/cagliostro-colab-ui.ipynb) `NEW`| A Customizable Stable Diffusion Web UI| [![](https://img.shields.io/static/v1?message=Open%20in%20Colab&logo=googlecolab&labelColor=5c5c5c&color=0f80c1&label=%20&style=for-the-badge)](https://colab.research.google.com/github/Linaqruf/sd-notebook-collection/blob/main/cagliostro-colab-ui.ipynb) | "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "NVxpMKh_FcbY"
   },
   "outputs": [],
   "source": [
    "# @title ## Mount Drive\n",
    "from google.colab import drive\n",
    "\n",
    "drive.mount(\"/content/drive\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TN0aDXTsiwW9",
    "outputId": "bb6ca31f-9b18-4411-fde4-0bf5dc45d344"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for library (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\n",
      "Download Results:\n",
      "gid   |stat|avg speed  |path/URI\n",
      "======+====+===========+=======================================================\n",
      "4c305d|\u001b[1;32mOK\u001b[0m  |       0B/s|/content/pretrained_model/anything-v3-fp32-pruned.safetensors\n",
      "\n",
      "Status Legend:\n",
      "(OK):download completed.\n",
      "\n",
      "Download Results:\n",
      "gid   |stat|avg speed  |path/URI\n",
      "======+====+===========+=======================================================\n",
      "a883ab|\u001b[1;32mOK\u001b[0m  |       0B/s|/content/vae/anime.vae.pt\n",
      "\n",
      "Status Legend:\n",
      "(OK):download completed.\n",
      "\u001b[0m\n",
      "Download Results:\n",
      "gid   |stat|avg speed  |path/URI\n",
      "======+====+===========+=======================================================\n",
      "759130|\u001b[1;32mOK\u001b[0m  |   259MiB/s|/content/train_data.zip\n",
      "\n",
      "Status Legend:\n",
      "(OK):download completed.\n",
      "Archive:  /content/train_data.zip\n",
      " extracting: /content/training_dir/train_data/hito_komoru088.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru044.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru044.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru024.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru073.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru067.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru082.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru081.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru019.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru009.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru047.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru057.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru077.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru026.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru037.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru094.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru014.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru060.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru069.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru003.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru051.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru058.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru033.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru078.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru081.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru054.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru036.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru058.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru065.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru080.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru049.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru068.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru085.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru051.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru052.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru070.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru084.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru072.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru055.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru083.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru010.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru002.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru096.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru026.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru007.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru094.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru084.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru084.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru016.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru068.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru051.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru089.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru083.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru039.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru050.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru035.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru030.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru088.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru038.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru086.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru047.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru011.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru041.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru038.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru046.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru004.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru024.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru098.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru066.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru026.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru063.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru071.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru017.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru056.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru069.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru082.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru050.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru073.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru096.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru020.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru094.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru070.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru012.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru019.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru011.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru027.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru012.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru055.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru001.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru035.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru018.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru069.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru004.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru039.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru015.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru023.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru063.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru008.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru062.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru079.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru012.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru085.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru064.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru070.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru037.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru081.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru056.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru010.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru064.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru086.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru003.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru017.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru019.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru090.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru013.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru007.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru019.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru086.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru058.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru052.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru055.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru040.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru074.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru010.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru032.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru047.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru090.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru045.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru080.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru007.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru027.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru068.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru063.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru098.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru045.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru097.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru060.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru075.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru002.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru034.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru055.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru034.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru046.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru046.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru014.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru028.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru061.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru078.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru049.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru079.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru067.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru072.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru024.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru074.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru040.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru005.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru031.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru023.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru014.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru067.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru091.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru032.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru025.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru097.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru047.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru036.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru057.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru042.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru059.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru059.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru046.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru053.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru076.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru022.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru041.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru057.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru004.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru022.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru043.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru076.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru049.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru053.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru078.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru009.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru042.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru093.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru036.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru031.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru090.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru049.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru008.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru059.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru051.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru096.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru072.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru016.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru077.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru093.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru033.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru077.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru057.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru096.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru003.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru033.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru090.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru073.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru058.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru065.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru080.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru066.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru062.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru089.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru042.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru082.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru043.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru087.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru062.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru018.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru029.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru013.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru088.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru021.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru001.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru021.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru029.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru015.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru006.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru045.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru038.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru045.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru008.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru074.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru092.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru069.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru052.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru080.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru053.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru034.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru032.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru003.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru011.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru035.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru089.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru083.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru029.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru035.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru005.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru022.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru095.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru016.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru032.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru084.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru056.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru050.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru023.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru086.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru048.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru083.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru006.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru044.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru024.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru082.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru095.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru030.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru048.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru008.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru085.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru087.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru044.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru031.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru022.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru081.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru036.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru091.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru041.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru061.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru050.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru071.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru011.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru072.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru030.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru093.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru005.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru076.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru028.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru028.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru064.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru061.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru077.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru023.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru078.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru017.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru054.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru020.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru025.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru009.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru085.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru048.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru002.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru014.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru037.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru067.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru009.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru048.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru007.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru041.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru071.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru040.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru091.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru005.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru091.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru039.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru054.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru068.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru059.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru073.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru020.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru098.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru013.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru002.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru075.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru061.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru028.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru015.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru098.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru092.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru056.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru071.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru029.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru088.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru042.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru030.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru054.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru031.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru018.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru064.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru020.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru013.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru063.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru010.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru062.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru066.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru026.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru034.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru025.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru093.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru018.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru053.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru043.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru095.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru027.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru027.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru079.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru006.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru097.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru075.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru092.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru087.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru001.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru025.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru043.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru033.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru066.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru075.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru038.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru052.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru040.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru016.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru092.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru004.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru037.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru087.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru070.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru094.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru095.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru076.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru065.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru079.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru001.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru021.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru012.png  \n",
      " extracting: /content/training_dir/train_data/hito_komoru039.npz  \n",
      " extracting: /content/training_dir/train_data/hito_komoru017.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru074.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru097.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru060.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru060.jpg  \n",
      " extracting: /content/training_dir/train_data/hito_komoru065.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru021.txt  \n",
      " extracting: /content/training_dir/train_data/hito_komoru089.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru015.caption  \n",
      " extracting: /content/training_dir/train_data/hito_komoru006.jpg  \n",
      "2023-03-09 22:44:34.417212: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-03-09 22:44:35.778852: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:44:35.778982: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:44:35.779006: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "found 98 images.\n",
      "loading existing metadata: /content/training_dir/meta_cap.json\n",
      "captions for existing images will be overwritten / 既存の画像のキャプションは上書きされます\n",
      "merge caption texts to metadata json.\n",
      "100% 98/98 [00:00<00:00, 32870.20it/s]\n",
      "writing metadata: /content/training_dir/meta_cap.json\n",
      "done!\n",
      "2023-03-09 22:44:42.297220: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-03-09 22:44:43.059864: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:44:43.059975: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:44:43.059995: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "found 98 images.\n",
      "loading existing metadata: /content/training_dir/meta_cap_dd.json\n",
      "tags data for existing images will be overwritten / 既存の画像のタグは上書きされます\n",
      "merge tags to metadata json.\n",
      "100% 98/98 [00:00<00:00, 22552.50it/s]\n",
      "writing metadata: /content/training_dir/meta_cap_dd.json\n",
      "done!\n",
      "2023-03-09 22:44:50.485451: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-03-09 22:44:51.662084: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:44:51.662245: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:44:51.662276: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "found 98 images.\n",
      "loading existing metadata: /content/training_dir/meta_cap.json\n",
      "tags data for existing images will be overwritten / 既存の画像のタグは上書きされます\n",
      "merge tags to metadata json.\n",
      "100% 98/98 [00:00<00:00, 35955.37it/s]\n",
      "writing metadata: /content/training_dir/meta_cap_dd.json\n",
      "done!\n",
      "loading existing metadata: /content/training_dir/meta_cap_dd.json\n",
      "cleaning captions and tags.\n",
      "100% 98/98 [00:00<00:00, 3942.39it/s]\n",
      "writing metadata: /content/training_dir/meta_clean.json\n",
      "done!\n",
      "2023-03-09 22:44:58.919919: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-03-09 22:44:59.697232: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.9/dist-packages/cv2/../../lib64:/usr/lib64-nvidia\n",
      "2023-03-09 22:44:59.697366: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.9/dist-packages/cv2/../../lib64:/usr/lib64-nvidia\n",
      "2023-03-09 22:44:59.697389: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "found 98 images.\n",
      "loading existing metadata: /content/training_dir/meta_clean.json\n",
      "load VAE: /content/pretrained_model/anything-v3-fp32-pruned.safetensors\n",
      "100% 98/98 [01:13<00:00,  1.33it/s]\n",
      "bucket 0 (384, 640): 8\n",
      "bucket 1 (448, 576): 53\n",
      "bucket 2 (512, 512): 14\n",
      "bucket 3 (576, 448): 13\n",
      "bucket 4 (640, 384): 10\n",
      "mean ar error: 0.05776305027796921\n",
      "writing metadata: /content/training_dir/meta_lat.json\n",
      "done!\n",
      "+--------------------------+---------------------------------------------------------------+\n",
      "| Hyperparameter           | Value                                                         |\n",
      "+--------------------------+---------------------------------------------------------------+\n",
      "| mode                     | LoRA                                                          |\n",
      "| use_dreambooth_method    | False                                                         |\n",
      "| lowram                   | True                                                          |\n",
      "| v2                       | False                                                         |\n",
      "| v_parameterization       | False                                                         |\n",
      "| project_name             | last                                                          |\n",
      "| modelPath                | /content/pretrained_model/anything-v3-fp32-pruned.safetensors |\n",
      "| vaePath                  | /content/vae/anime.vae.pt                                     |\n",
      "| train_data_dir           | /content/training_dir/train_data                              |\n",
      "| meta_lat                 | /content/training_dir/meta_lat.json                           |\n",
      "| output_dir               | /content/training_dir/output                                  |\n",
      "| network_dim              | 128                                                           |\n",
      "| network_alpha            | 128                                                           |\n",
      "| network_weights          | False                                                         |\n",
      "| unet_lr                  | 0.0001                                                        |\n",
      "| text_encoder_lr          | 5e-05                                                         |\n",
      "| optimizer_type           | AdamW8bit                                                     |\n",
      "| optimizer_args           | False                                                         |\n",
      "| learning_rate            | 2e-06                                                         |\n",
      "| lr_scheduler             | constant                                                      |\n",
      "| lr_warmup_steps          | 250                                                           |\n",
      "| lr_scheduler_args        | 1                                                             |\n",
      "| dataset_repeats          | 10                                                            |\n",
      "| resolution               | 512                                                           |\n",
      "| noise_offset             | 0                                                             |\n",
      "| mixed_precision          | fp16                                                          |\n",
      "| save_precision           | fp16                                                          |\n",
      "| save_n_epochs_type       | save_n_epoch_ratio                                            |\n",
      "| save_n_epochs_type_value | 3                                                             |\n",
      "| save_model_as            | safetensors                                                   |\n",
      "| train_batch_size         | 4                                                             |\n",
      "| max_train_type           | max_train_epochs                                              |\n",
      "| max_train_type_value     | 20                                                            |\n",
      "| clip_skip                | 2                                                             |\n",
      "| logging_dir              | /content/training_dir/logs                                    |\n",
      "| additional_argument      | --shuffle_caption --xformers                                  |\n",
      "+--------------------------+---------------------------------------------------------------+\n",
      "2023-03-09 22:46:38.634469: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-03-09 22:46:39.939229: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:46:39.939429: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:46:39.939461: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "2023-03-09 22:46:44.066753: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-03-09 22:46:44.836809: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:46:44.836917: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia\n",
      "2023-03-09 22:46:44.836937: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "prepare tokenizer\n",
      "update token length: 225\n",
      "Train with captions.\n",
      "loading existing metadata: /content/training_dir/meta_lat.json\n",
      "metadata has bucket info, enable bucketing / メタデータにbucket情報があるためbucketを有効にします\n",
      "using bucket info in metadata / メタデータ内のbucket情報を使います\n",
      "[Dataset 0]\n",
      "  batch_size: 4\n",
      "  resolution: (512, 512)\n",
      "  enable_bucket: True\n",
      "  min_bucket_reso: None\n",
      "  max_bucket_reso: None\n",
      "  bucket_reso_steps: None\n",
      "  bucket_no_upscale: None\n",
      "\n",
      "  [Subset 0 of Dataset 0]\n",
      "    image_dir: \"/content/training_dir/train_data\"\n",
      "    image_count: 98\n",
      "    num_repeats: 10\n",
      "    shuffle_caption: True\n",
      "    keep_tokens: 0\n",
      "    caption_dropout_rate: 0.0\n",
      "    caption_dropout_every_n_epoches: 0\n",
      "    caption_tag_dropout_rate: 0.0\n",
      "    color_aug: False\n",
      "    flip_aug: False\n",
      "    face_crop_aug_range: None\n",
      "    random_crop: False\n",
      "    metadata_file: /content/training_dir/meta_lat.json\n",
      "\n",
      "\n",
      "[Dataset 0]\n",
      "loading image sizes.\n",
      "100% 98/98 [00:00<00:00, 1194888.93it/s]\n",
      "make buckets\n",
      "number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）\n",
      "bucket 0: resolution (384, 640), count: 80\n",
      "bucket 1: resolution (448, 576), count: 530\n",
      "bucket 2: resolution (512, 512), count: 140\n",
      "bucket 3: resolution (576, 448), count: 130\n",
      "bucket 4: resolution (640, 384), count: 100\n",
      "mean ar error (without repeats): 0.0\n",
      "prepare accelerator\n",
      "Using accelerator 0.15.0 or above.\n",
      "load StableDiffusion checkpoint\n",
      "loading u-net: <All keys matched successfully>\n",
      "loading vae: <All keys matched successfully>\n",
      "Downloading (…)lve/main/config.json: 100% 4.52k/4.52k [00:00<00:00, 637kB/s]\n",
      "Downloading (…)\"pytorch_model.bin\";: 100% 1.71G/1.71G [00:56<00:00, 30.1MB/s]\n",
      "Some weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel: ['vision_model.encoder.layers.14.mlp.fc1.weight', 'vision_model.encoder.layers.11.layer_norm1.bias', 'vision_model.encoder.layers.1.layer_norm2.bias', 'vision_model.encoder.layers.22.self_attn.v_proj.bias', 'vision_model.post_layernorm.bias', 'vision_model.encoder.layers.19.self_attn.q_proj.bias', 'vision_model.encoder.layers.0.layer_norm1.weight', 'vision_model.encoder.layers.5.self_attn.v_proj.weight', 'vision_model.encoder.layers.11.mlp.fc2.bias', 'vision_model.encoder.layers.8.mlp.fc2.weight', 'vision_model.encoder.layers.20.self_attn.v_proj.weight', 'vision_model.encoder.layers.4.self_attn.q_proj.weight', 'vision_model.encoder.layers.18.mlp.fc1.bias', 'vision_model.encoder.layers.13.self_attn.v_proj.bias', 'vision_model.encoder.layers.18.layer_norm1.weight', 'vision_model.encoder.layers.6.layer_norm1.bias', 'vision_model.encoder.layers.9.self_attn.q_proj.weight', 'vision_model.encoder.layers.7.self_attn.q_proj.weight', 'vision_model.encoder.layers.20.mlp.fc2.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.weight', 'vision_model.encoder.layers.8.layer_norm2.bias', 'vision_model.encoder.layers.7.mlp.fc2.weight', 'vision_model.encoder.layers.19.self_attn.out_proj.weight', 'vision_model.encoder.layers.18.self_attn.out_proj.bias', 'vision_model.encoder.layers.14.self_attn.out_proj.weight', 'vision_model.encoder.layers.15.self_attn.v_proj.bias', 'vision_model.encoder.layers.16.mlp.fc1.bias', 'vision_model.encoder.layers.1.mlp.fc2.bias', 'vision_model.encoder.layers.7.self_attn.k_proj.bias', 'vision_model.encoder.layers.16.layer_norm1.weight', 'visual_projection.weight', 'vision_model.encoder.layers.7.self_attn.q_proj.bias', 'vision_model.encoder.layers.20.self_attn.k_proj.weight', 'vision_model.encoder.layers.3.self_attn.v_proj.bias', 'vision_model.encoder.layers.12.layer_norm1.bias', 'vision_model.encoder.layers.4.mlp.fc1.weight', 'vision_model.encoder.layers.22.layer_norm2.weight', 'vision_model.encoder.layers.22.layer_norm1.bias', 'vision_model.encoder.layers.13.layer_norm2.weight', 'vision_model.encoder.layers.12.mlp.fc1.bias', 'vision_model.encoder.layers.13.self_attn.out_proj.weight', 'vision_model.encoder.layers.3.layer_norm1.weight', 'vision_model.encoder.layers.2.mlp.fc2.weight', 'vision_model.encoder.layers.10.self_attn.v_proj.bias', 'vision_model.encoder.layers.0.self_attn.k_proj.bias', 'vision_model.encoder.layers.0.self_attn.out_proj.weight', 'vision_model.encoder.layers.19.mlp.fc1.weight', 'vision_model.encoder.layers.21.self_attn.v_proj.bias', 'vision_model.encoder.layers.7.self_attn.k_proj.weight', 'vision_model.encoder.layers.3.self_attn.k_proj.bias', 'vision_model.encoder.layers.6.self_attn.out_proj.bias', 'vision_model.encoder.layers.9.layer_norm1.weight', 'vision_model.encoder.layers.8.layer_norm2.weight', 'vision_model.encoder.layers.17.self_attn.v_proj.weight', 'vision_model.encoder.layers.7.layer_norm1.weight', 'vision_model.encoder.layers.17.mlp.fc2.weight', 'vision_model.encoder.layers.10.mlp.fc1.bias', 'vision_model.encoder.layers.3.self_attn.q_proj.bias', 'vision_model.encoder.layers.7.layer_norm1.bias', 'vision_model.encoder.layers.4.layer_norm2.bias', 'vision_model.encoder.layers.0.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.self_attn.k_proj.weight', 'vision_model.encoder.layers.11.self_attn.v_proj.bias', 'vision_model.encoder.layers.6.mlp.fc2.bias', 'vision_model.encoder.layers.6.self_attn.k_proj.bias', 'vision_model.encoder.layers.8.mlp.fc1.bias', 'vision_model.encoder.layers.14.self_attn.q_proj.weight', 'vision_model.encoder.layers.21.mlp.fc1.weight', 'vision_model.encoder.layers.0.layer_norm1.bias', 'vision_model.encoder.layers.22.mlp.fc1.weight', 'vision_model.encoder.layers.17.mlp.fc1.weight', 'vision_model.encoder.layers.1.self_attn.q_proj.weight', 'vision_model.encoder.layers.6.layer_norm2.weight', 'vision_model.encoder.layers.18.mlp.fc2.bias', 'vision_model.encoder.layers.4.layer_norm2.weight', 'vision_model.encoder.layers.17.self_attn.out_proj.weight', 'vision_model.encoder.layers.9.self_attn.q_proj.bias', 'vision_model.encoder.layers.21.mlp.fc2.weight', 'vision_model.encoder.layers.15.mlp.fc2.weight', 'vision_model.encoder.layers.1.layer_norm1.bias', 'vision_model.encoder.layers.14.mlp.fc2.weight', 'vision_model.encoder.layers.9.self_attn.k_proj.bias', 'vision_model.encoder.layers.8.self_attn.k_proj.weight', 'vision_model.encoder.layers.18.self_attn.out_proj.weight', 'vision_model.encoder.layers.8.layer_norm1.bias', 'vision_model.encoder.layers.23.self_attn.q_proj.weight', 'vision_model.encoder.layers.9.layer_norm2.bias', 'vision_model.encoder.layers.20.layer_norm2.bias', 'vision_model.encoder.layers.11.self_attn.q_proj.bias', 'vision_model.encoder.layers.12.self_attn.v_proj.weight', 'vision_model.encoder.layers.23.self_attn.k_proj.bias', 'vision_model.encoder.layers.1.mlp.fc1.weight', 'vision_model.encoder.layers.5.mlp.fc2.weight', 'vision_model.encoder.layers.5.self_attn.q_proj.bias', 'vision_model.encoder.layers.7.self_attn.v_proj.bias', 'vision_model.encoder.layers.11.self_attn.v_proj.weight', 'vision_model.encoder.layers.17.layer_norm1.bias', 'vision_model.encoder.layers.2.self_attn.q_proj.bias', 'vision_model.encoder.layers.14.layer_norm1.weight', 'vision_model.encoder.layers.19.self_attn.v_proj.weight', 'vision_model.encoder.layers.7.mlp.fc1.weight', 'vision_model.encoder.layers.19.layer_norm2.bias', 'vision_model.encoder.layers.13.self_attn.v_proj.weight', 'vision_model.encoder.layers.23.layer_norm2.weight', 'vision_model.encoder.layers.23.self_attn.k_proj.weight', 'vision_model.encoder.layers.1.mlp.fc1.bias', 'vision_model.encoder.layers.1.layer_norm1.weight', 'vision_model.encoder.layers.5.layer_norm2.weight', 'vision_model.encoder.layers.8.mlp.fc1.weight', 'vision_model.encoder.layers.15.layer_norm2.weight', 'vision_model.encoder.layers.3.mlp.fc1.weight', 'vision_model.encoder.layers.20.self_attn.k_proj.bias', 'vision_model.encoder.layers.10.self_attn.k_proj.bias', 'vision_model.encoder.layers.12.self_attn.k_proj.weight', 'vision_model.encoder.layers.20.self_attn.out_proj.bias', 'vision_model.encoder.layers.20.mlp.fc2.weight', 'vision_model.encoder.layers.8.self_attn.out_proj.weight', 'vision_model.encoder.layers.2.layer_norm1.weight', 'vision_model.encoder.layers.17.layer_norm2.bias', 'vision_model.encoder.layers.11.self_attn.k_proj.bias', 'vision_model.encoder.layers.2.layer_norm2.bias', 'vision_model.encoder.layers.6.self_attn.q_proj.bias', 'vision_model.embeddings.position_embedding.weight', 'vision_model.encoder.layers.20.self_attn.out_proj.weight', 'vision_model.encoder.layers.15.mlp.fc1.bias', 'vision_model.encoder.layers.10.self_attn.out_proj.weight', 'vision_model.encoder.layers.19.self_attn.out_proj.bias', 'vision_model.encoder.layers.1.self_attn.q_proj.bias', 'vision_model.encoder.layers.12.self_attn.out_proj.bias', 'vision_model.encoder.layers.1.self_attn.v_proj.bias', 'vision_model.encoder.layers.10.self_attn.v_proj.weight', 'vision_model.encoder.layers.2.self_attn.v_proj.weight', 'vision_model.encoder.layers.10.mlp.fc1.weight', 'vision_model.encoder.layers.0.layer_norm2.weight', 'vision_model.encoder.layers.21.self_attn.q_proj.weight', 'vision_model.encoder.layers.2.layer_norm2.weight', 'vision_model.encoder.layers.2.self_attn.out_proj.weight', 'vision_model.encoder.layers.3.self_attn.out_proj.bias', 'vision_model.encoder.layers.23.layer_norm1.bias', 'vision_model.encoder.layers.1.layer_norm2.weight', 'vision_model.encoder.layers.2.self_attn.q_proj.weight', 'vision_model.encoder.layers.8.self_attn.q_proj.bias', 'vision_model.encoder.layers.13.mlp.fc2.weight', 'vision_model.encoder.layers.19.mlp.fc1.bias', 'vision_model.encoder.layers.20.self_attn.q_proj.bias', 'vision_model.encoder.layers.0.mlp.fc2.weight', 'vision_model.encoder.layers.14.mlp.fc1.bias', 'vision_model.encoder.layers.18.self_attn.k_proj.bias', 'vision_model.encoder.layers.2.mlp.fc2.bias', 'vision_model.pre_layrnorm.bias', 'vision_model.encoder.layers.20.layer_norm1.bias', 'vision_model.encoder.layers.13.mlp.fc1.weight', 'vision_model.encoder.layers.21.layer_norm1.bias', 'vision_model.encoder.layers.10.self_attn.q_proj.weight', 'vision_model.encoder.layers.5.layer_norm1.weight', 'vision_model.encoder.layers.13.layer_norm1.bias', 'vision_model.encoder.layers.10.mlp.fc2.bias', 'vision_model.encoder.layers.0.self_attn.q_proj.bias', 'vision_model.encoder.layers.5.layer_norm2.bias', 'vision_model.encoder.layers.2.self_attn.out_proj.bias', 'vision_model.encoder.layers.8.mlp.fc2.bias', 'vision_model.encoder.layers.12.layer_norm2.bias', 'vision_model.encoder.layers.6.mlp.fc2.weight', 'vision_model.encoder.layers.18.layer_norm2.bias', 'vision_model.encoder.layers.16.mlp.fc2.bias', 'vision_model.encoder.layers.21.self_attn.v_proj.weight', 'vision_model.encoder.layers.11.self_attn.out_proj.bias', 'vision_model.encoder.layers.19.layer_norm1.weight', 'vision_model.encoder.layers.3.layer_norm2.weight', 'vision_model.encoder.layers.21.self_attn.q_proj.bias', 'vision_model.encoder.layers.9.mlp.fc1.bias', 'vision_model.encoder.layers.15.self_attn.q_proj.bias', 'vision_model.encoder.layers.22.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.self_attn.v_proj.bias', 'vision_model.encoder.layers.16.layer_norm2.weight', 'vision_model.encoder.layers.10.layer_norm2.bias', 'vision_model.encoder.layers.9.self_attn.out_proj.bias', 'vision_model.encoder.layers.6.mlp.fc1.bias', 'vision_model.encoder.layers.8.self_attn.k_proj.bias', 'vision_model.encoder.layers.17.mlp.fc2.bias', 'vision_model.encoder.layers.18.self_attn.q_proj.weight', 'vision_model.encoder.layers.6.self_attn.v_proj.weight', 'vision_model.embeddings.position_ids', 'vision_model.encoder.layers.4.self_attn.out_proj.weight', 'vision_model.encoder.layers.16.self_attn.k_proj.bias', 'vision_model.encoder.layers.13.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.mlp.fc2.bias', 'vision_model.encoder.layers.16.layer_norm2.bias', 'vision_model.encoder.layers.5.mlp.fc1.bias', 'vision_model.encoder.layers.9.layer_norm1.bias', 'vision_model.encoder.layers.3.layer_norm1.bias', 'vision_model.encoder.layers.14.self_attn.k_proj.weight', 'vision_model.encoder.layers.19.mlp.fc2.weight', 'vision_model.encoder.layers.22.self_attn.q_proj.weight', 'vision_model.encoder.layers.22.mlp.fc2.weight', 'vision_model.encoder.layers.12.mlp.fc2.weight', 'vision_model.encoder.layers.20.mlp.fc1.weight', 'vision_model.encoder.layers.12.self_attn.q_proj.weight', 'vision_model.encoder.layers.4.layer_norm1.bias', 'vision_model.encoder.layers.20.mlp.fc1.bias', 'vision_model.encoder.layers.20.self_attn.q_proj.weight', 'text_projection.weight', 'vision_model.encoder.layers.18.layer_norm2.weight', 'vision_model.encoder.layers.22.self_attn.q_proj.bias', 'vision_model.encoder.layers.17.self_attn.v_proj.bias', 'vision_model.encoder.layers.17.self_attn.k_proj.weight', 'vision_model.encoder.layers.21.mlp.fc2.bias', 'vision_model.encoder.layers.18.layer_norm1.bias', 'vision_model.encoder.layers.9.self_attn.out_proj.weight', 'vision_model.encoder.layers.21.layer_norm1.weight', 'vision_model.encoder.layers.14.layer_norm1.bias', 'vision_model.encoder.layers.4.mlp.fc1.bias', 'vision_model.encoder.layers.23.self_attn.v_proj.weight', 'vision_model.encoder.layers.23.self_attn.out_proj.weight', 'vision_model.encoder.layers.17.layer_norm1.weight', 'vision_model.encoder.layers.2.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.mlp.fc1.bias', 'vision_model.encoder.layers.14.self_attn.out_proj.bias', 'vision_model.encoder.layers.6.self_attn.out_proj.weight', 'logit_scale', 'vision_model.encoder.layers.6.layer_norm2.bias', 'vision_model.encoder.layers.11.layer_norm2.weight', 'vision_model.encoder.layers.21.self_attn.k_proj.bias', 'vision_model.encoder.layers.22.self_attn.k_proj.bias', 'vision_model.encoder.layers.16.self_attn.v_proj.bias', 'vision_model.encoder.layers.20.layer_norm2.weight', 'vision_model.encoder.layers.17.self_attn.q_proj.weight', 'vision_model.encoder.layers.15.self_attn.out_proj.bias', 'vision_model.encoder.layers.9.self_attn.v_proj.bias', 'vision_model.encoder.layers.0.self_attn.out_proj.bias', 'vision_model.encoder.layers.18.mlp.fc1.weight', 'vision_model.encoder.layers.9.mlp.fc2.weight', 'vision_model.encoder.layers.18.self_attn.v_proj.weight', 'vision_model.encoder.layers.19.self_attn.k_proj.bias', 'vision_model.encoder.layers.14.mlp.fc2.bias', 'vision_model.encoder.layers.0.mlp.fc1.bias', 'vision_model.encoder.layers.20.layer_norm1.weight', 'vision_model.encoder.layers.8.self_attn.q_proj.weight', 'vision_model.embeddings.patch_embedding.weight', 'vision_model.encoder.layers.8.self_attn.v_proj.bias', 'vision_model.encoder.layers.3.mlp.fc2.weight', 'vision_model.encoder.layers.12.self_attn.k_proj.bias', 'vision_model.encoder.layers.15.mlp.fc1.weight', 'vision_model.encoder.layers.23.self_attn.out_proj.bias', 'vision_model.encoder.layers.10.self_attn.out_proj.bias', 'vision_model.encoder.layers.10.self_attn.q_proj.bias', 'vision_model.encoder.layers.15.self_attn.k_proj.bias', 'vision_model.encoder.layers.13.layer_norm1.weight', 'vision_model.encoder.layers.17.self_attn.q_proj.bias', 'vision_model.encoder.layers.15.mlp.fc2.bias', 'vision_model.encoder.layers.7.layer_norm2.bias', 'vision_model.encoder.layers.16.layer_norm1.bias', 'vision_model.encoder.layers.13.mlp.fc1.bias', 'vision_model.encoder.layers.23.self_attn.v_proj.bias', 'vision_model.encoder.layers.14.self_attn.k_proj.bias', 'vision_model.encoder.layers.1.self_attn.v_proj.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.weight', 'vision_model.encoder.layers.22.self_attn.k_proj.weight', 'vision_model.encoder.layers.11.layer_norm1.weight', 'vision_model.encoder.layers.23.mlp.fc2.bias', 'vision_model.encoder.layers.7.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.self_attn.out_proj.bias', 'vision_model.encoder.layers.18.self_attn.v_proj.bias', 'vision_model.encoder.layers.4.mlp.fc2.bias', 'vision_model.encoder.layers.18.self_attn.k_proj.weight', 'vision_model.encoder.layers.2.layer_norm1.bias', 'vision_model.encoder.layers.22.self_attn.out_proj.weight', 'vision_model.encoder.layers.1.self_attn.k_proj.bias', 'vision_model.encoder.layers.15.self_attn.out_proj.weight', 'vision_model.encoder.layers.2.self_attn.v_proj.bias', 'vision_model.encoder.layers.15.self_attn.v_proj.weight', 'vision_model.encoder.layers.18.self_attn.q_proj.bias', 'vision_model.encoder.layers.19.self_attn.k_proj.weight', 'vision_model.encoder.layers.17.self_attn.k_proj.bias', 'vision_model.encoder.layers.19.layer_norm2.weight', 'vision_model.encoder.layers.10.layer_norm1.weight', 'vision_model.encoder.layers.11.self_attn.k_proj.weight', 'vision_model.encoder.layers.11.layer_norm2.bias', 'vision_model.encoder.layers.12.self_attn.out_proj.weight', 'vision_model.encoder.layers.23.layer_norm2.bias', 'vision_model.encoder.layers.3.mlp.fc1.bias', 'vision_model.encoder.layers.0.layer_norm2.bias', 'vision_model.encoder.layers.14.layer_norm2.bias', 'vision_model.encoder.layers.4.self_attn.v_proj.bias', 'vision_model.encoder.layers.7.layer_norm2.weight', 'vision_model.encoder.layers.21.mlp.fc1.bias', 'vision_model.encoder.layers.23.layer_norm1.weight', 'vision_model.encoder.layers.7.mlp.fc2.bias', 'vision_model.encoder.layers.2.self_attn.k_proj.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.bias', 'vision_model.encoder.layers.0.self_attn.k_proj.weight', 'vision_model.encoder.layers.16.mlp.fc1.weight', 'vision_model.encoder.layers.9.mlp.fc1.weight', 'vision_model.encoder.layers.6.self_attn.k_proj.weight', 'vision_model.encoder.layers.10.mlp.fc2.weight', 'vision_model.encoder.layers.7.mlp.fc1.bias', 'vision_model.encoder.layers.1.self_attn.k_proj.weight', 'vision_model.encoder.layers.16.self_attn.k_proj.weight', 'vision_model.encoder.layers.12.layer_norm1.weight', 'vision_model.encoder.layers.13.self_attn.k_proj.bias', 'vision_model.encoder.layers.5.self_attn.k_proj.bias', 'vision_model.encoder.layers.21.self_attn.k_proj.weight', 'vision_model.encoder.layers.7.self_attn.out_proj.bias', 'vision_model.encoder.layers.11.mlp.fc1.weight', 'vision_model.encoder.layers.12.layer_norm2.weight', 'vision_model.encoder.layers.16.self_attn.out_proj.weight', 'vision_model.encoder.layers.12.self_attn.v_proj.bias', 'vision_model.encoder.layers.4.self_attn.q_proj.bias', 'vision_model.encoder.layers.15.self_attn.k_proj.weight', 'vision_model.encoder.layers.19.self_attn.q_proj.weight', 'vision_model.encoder.layers.22.mlp.fc2.bias', 'vision_model.encoder.layers.13.self_attn.q_proj.bias', 'vision_model.encoder.layers.16.self_attn.q_proj.bias', 'vision_model.encoder.layers.0.mlp.fc1.weight', 'vision_model.encoder.layers.14.self_attn.q_proj.bias', 'vision_model.encoder.layers.20.self_attn.v_proj.bias', 'vision_model.encoder.layers.15.layer_norm1.bias', 'vision_model.encoder.layers.4.self_attn.k_proj.bias', 'vision_model.encoder.layers.5.self_attn.q_proj.weight', 'vision_model.encoder.layers.12.self_attn.q_proj.bias', 'vision_model.encoder.layers.17.self_attn.out_proj.bias', 'vision_model.encoder.layers.1.self_attn.out_proj.bias', 'vision_model.encoder.layers.8.layer_norm1.weight', 'vision_model.encoder.layers.22.layer_norm2.bias', 'vision_model.encoder.layers.4.self_attn.k_proj.weight', 'vision_model.encoder.layers.11.mlp.fc1.bias', 'vision_model.encoder.layers.5.mlp.fc2.bias', 'vision_model.encoder.layers.4.layer_norm1.weight', 'vision_model.encoder.layers.8.self_attn.out_proj.bias', 'vision_model.encoder.layers.12.mlp.fc2.bias', 'vision_model.encoder.layers.3.self_attn.out_proj.weight', 'vision_model.encoder.layers.15.self_attn.q_proj.weight', 'vision_model.encoder.layers.1.mlp.fc2.weight', 'vision_model.encoder.layers.23.mlp.fc2.weight', 'vision_model.encoder.layers.0.mlp.fc2.bias', 'vision_model.encoder.layers.5.self_attn.out_proj.bias', 'vision_model.embeddings.class_embedding', 'vision_model.encoder.layers.23.mlp.fc1.bias', 'vision_model.encoder.layers.15.layer_norm1.weight', 'vision_model.encoder.layers.4.self_attn.v_proj.weight', 'vision_model.encoder.layers.12.mlp.fc1.weight', 'vision_model.encoder.layers.4.self_attn.out_proj.bias', 'vision_model.encoder.layers.10.self_attn.k_proj.weight', 'vision_model.encoder.layers.1.self_attn.out_proj.weight', 'vision_model.encoder.layers.15.layer_norm2.bias', 'vision_model.encoder.layers.19.layer_norm1.bias', 'vision_model.encoder.layers.10.layer_norm2.weight', 'vision_model.encoder.layers.6.self_attn.v_proj.bias', 'vision_model.encoder.layers.22.layer_norm1.weight', 'vision_model.encoder.layers.23.self_attn.q_proj.bias', 'vision_model.encoder.layers.13.layer_norm2.bias', 'vision_model.encoder.layers.9.self_attn.k_proj.weight', 'vision_model.encoder.layers.9.mlp.fc2.bias', 'vision_model.encoder.layers.14.self_attn.v_proj.bias', 'vision_model.encoder.layers.18.mlp.fc2.weight', 'vision_model.encoder.layers.2.mlp.fc1.weight', 'vision_model.encoder.layers.21.self_attn.out_proj.weight', 'vision_model.encoder.layers.7.self_attn.out_proj.weight', 'vision_model.encoder.layers.13.self_attn.k_proj.weight', 'vision_model.encoder.layers.13.mlp.fc2.bias', 'vision_model.encoder.layers.5.mlp.fc1.weight', 'vision_model.encoder.layers.3.self_attn.v_proj.weight', 'vision_model.encoder.layers.3.layer_norm2.bias', 'vision_model.encoder.layers.16.self_attn.q_proj.weight', 'vision_model.post_layernorm.weight', 'vision_model.encoder.layers.6.layer_norm1.weight', 'vision_model.encoder.layers.16.self_attn.v_proj.weight', 'vision_model.encoder.layers.19.self_attn.v_proj.bias', 'vision_model.encoder.layers.13.self_attn.out_proj.bias', 'vision_model.encoder.layers.14.layer_norm2.weight', 'vision_model.encoder.layers.14.self_attn.v_proj.weight', 'vision_model.encoder.layers.16.self_attn.out_proj.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.weight', 'vision_model.encoder.layers.17.mlp.fc1.bias', 'vision_model.encoder.layers.2.mlp.fc1.bias', 'vision_model.encoder.layers.3.self_attn.q_proj.weight', 'vision_model.encoder.layers.4.mlp.fc2.weight', 'vision_model.encoder.layers.10.layer_norm1.bias', 'vision_model.encoder.layers.5.self_attn.out_proj.weight', 'vision_model.encoder.layers.11.mlp.fc2.weight', 'vision_model.encoder.layers.23.mlp.fc1.weight', 'vision_model.encoder.layers.19.mlp.fc2.bias', 'vision_model.encoder.layers.5.layer_norm1.bias', 'vision_model.encoder.layers.21.layer_norm2.bias', 'vision_model.pre_layrnorm.weight', 'vision_model.encoder.layers.9.self_attn.v_proj.weight', 'vision_model.encoder.layers.17.layer_norm2.weight', 'vision_model.encoder.layers.6.self_attn.q_proj.weight', 'vision_model.encoder.layers.21.layer_norm2.weight', 'vision_model.encoder.layers.21.self_attn.out_proj.bias', 'vision_model.encoder.layers.11.self_attn.out_proj.weight', 'vision_model.encoder.layers.6.mlp.fc1.weight', 'vision_model.encoder.layers.16.mlp.fc2.weight', 'vision_model.encoder.layers.9.layer_norm2.weight']\n",
      "- This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "loading text encoder: <All keys matched successfully>\n",
      "load VAE: /content/vae/anime.vae.pt\n",
      "additional VAE loaded\n",
      "Replace CrossAttention.forward to use xformers\n",
      "import network module: networks.lora\n",
      "create LoRA for Text Encoder: 72 modules.\n",
      "create LoRA for U-Net: 192 modules.\n",
      "enable LoRA for text encoder\n",
      "enable LoRA for U-Net\n",
      "prepare optimizer, data loader etc.\n",
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link\n",
      "================================================================================\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 7.5\n",
      "CUDA SETUP: Detected CUDA version 118\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...\n",
      "use 8-bit AdamW optimizer | {}\n",
      "override steps. steps for 20 epochs is / 指定エポックまでのステップ数: 4920\n",
      "running training / 学習開始\n",
      "  num train images * repeats / 学習画像の数×繰り返し回数: 980\n",
      "  num reg images / 正則化画像の数: 0\n",
      "  num batches per epoch / 1epochのバッチ数: 246\n",
      "  num epochs / epoch数: 20\n",
      "  batch size per device / バッチサイズ: 4\n",
      "  gradient accumulation steps / 勾配を合計するステップ数 = 1\n",
      "  total optimization steps / 学習ステップ数: 4920\n",
      "steps:   0% 0/4920 [00:00<?, ?it/s]epoch 1/20\n",
      "steps:   5% 230/4920 [04:13<1:26:05,  1.10s/it, loss=0.157]"
     ]
    }
   ],
   "source": [
    "#@title ## Start Training\n",
    "import os\n",
    "import html\n",
    "import time\n",
    "import textwrap\n",
    "import yaml\n",
    "import zipfile\n",
    "import shutil\n",
    "from IPython.utils import capture\n",
    "from subprocess import getoutput\n",
    "from google.colab import drive\n",
    "from prettytable import PrettyTable\n",
    "\n",
    "# List\n",
    "installVae = []\n",
    "\n",
    "# huggingface token for download\n",
    "user_token = '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'\n",
    "user_header = f\"\\\"Authorization: Bearer {user_token}\\\"\"\n",
    "\n",
    "#@markdown ## General Config\n",
    "#@markdown <small><font color=gray> **HINT**: `LoRA` or `Native Training`? you can specify it here. </small><br>\n",
    "mode = \"LoRA\" #@param [\"LoRA\", \"native-training\"]\n",
    "lowram = True #@param {type:\"boolean\"}\n",
    "output_to_drive = False #@param {'type':'boolean'}\n",
    "install_xformers = True #@param {'type':'boolean'}\n",
    "\n",
    "# Define Variable\n",
    "root_dir = \"/content\"\n",
    "deps_dir = os.path.join(root_dir, \"deps\")\n",
    "repo_dir = os.path.join(root_dir, \"kohya-trainer\")\n",
    "tools_dir = os.path.join(root_dir, \"kohya-trainer/tools\")\n",
    "finetune_dir = os.path.join(root_dir, \"kohya-trainer/finetune\")\n",
    "training_dir =  os.path.join(root_dir, \"training_dir\")\n",
    "pretrained_model = os.path.join(root_dir, \"pretrained_model\")\n",
    "vae = os.path.join(root_dir, \"vae\")\n",
    "\n",
    "if output_to_drive:\n",
    "  if not os.path.exists(\"/content/drive\"):\n",
    "    drive.mount('/content/drive')\n",
    "  training_dir = \"/content/drive/MyDrive/training_dir\"\n",
    "\n",
    "# Accelerate Config\n",
    "accelerate_config = os.path.join(repo_dir, \"accelerate_config/config.yaml\")\n",
    "\n",
    "project_name = \"\" #@param {'type' : 'string'}\n",
    "if not project_name:\n",
    "  project_name = \"last\"\n",
    "  \n",
    "train_data_dir = \"/content/training_dir/train_data\" #@param {'type' : 'string'}\n",
    "#@markdown <small><font color=gray> **HINT**: specify this part if your dataset are in `zip` and uploaded somewhere, this will download your dataset and automatically extract them to `train_data_dir` </small><br> \n",
    "dataset_zip_url = \"\" #@param {'type': 'string'}\n",
    "zipfile = \"train_data.zip\"\n",
    "\n",
    "meta_clean = os.path.join(training_dir, \"meta_clean.json\")\n",
    "meta_cap_dd = os.path.join(training_dir, \"meta_cap_dd.json\")\n",
    "meta_cap = os.path.join(training_dir, \"meta_cap.json\")\n",
    "meta_lat = os.path.join(training_dir, \"meta_lat.json\")\n",
    "output_dir = os.path.join(training_dir, \"output\")\n",
    "\n",
    "# V2 Inference\n",
    "inference_url = \"https://raw.githubusercontent.com/Stability-AI/stablediffusion/main/configs/stable-diffusion/\"\n",
    "\n",
    "# For Dataset Cleaning\n",
    "supported_types = [\".png\", \".jpg\", \".jpeg\", \".webp\", \".bmp\", \".caption\", \".npz\", \".txt\", \".json\"]\n",
    "\n",
    "# Make Directory\n",
    "os.chdir(root_dir)\n",
    "if not os.path.exists(repo_dir):\n",
    " !git clone \"https://github.com/Linaqruf/kohya-trainer\" {repo_dir}\n",
    "\n",
    "for dir in [repo_dir, tools_dir, finetune_dir, deps_dir, training_dir, pretrained_model, vae, training_dir]:\n",
    "  os.makedirs(dir, exist_ok=True)\n",
    "\n",
    "def ubuntu_deps(url, name, dst):\n",
    "  with capture.capture_output() as cap:\n",
    "    !wget -q --show-progress {url}\n",
    "    !unzip -j -o {name} -d \"{dst}\"\n",
    "    !dpkg -i {dst}/*\n",
    "    os.remove(name)\n",
    "    shutil.rmtree(dst)\n",
    "    del cap \n",
    "\n",
    "def install_dependencies():\n",
    "  !pip -q install --upgrade -r requirements.txt\n",
    "\n",
    "  if install_xformers:\n",
    "    !pip install xformers==0.0.18 triton\n",
    "\n",
    "  from accelerate.utils import write_basic_config\n",
    "  if not os.path.exists(accelerate_config):\n",
    "    write_basic_config(save_location=accelerate_config)\n",
    "\n",
    "\n",
    "ubuntu_deps(\"https://huggingface.co/Linaqruf/fast-repo/resolve/main/deb-libs.zip\", \"deb-libs.zip\", deps_dir)\n",
    "os.chdir(repo_dir)\n",
    "install_dependencies()\n",
    "\n",
    "#@markdown ##<br> `NEW` Dreambooth Config\n",
    "\n",
    "use_dreambooth_method = False #@param {type: 'boolean'}\n",
    "instance_token = \"sksfrog\" #@param {type: \"string\"}\n",
    "caption_extension = '.txt' #@param {'type':'string'}\n",
    "\n",
    "#@markdown ##<br> Download Pretrained Model\n",
    "modelUrl = \"https://huggingface.co/Linaqruf/anything-v3.0/resolve/main/anything-v3-fp32-pruned.safetensors\" #@param {'type': 'string'}\n",
    "#@markdown <small><font color=gray> **HINT**: useful for downloading pretrained model outside huggingface</small><br>\n",
    "modelName = \"anything-v3-fp32-pruned.safetensors\" #@param {'type': 'string'}\n",
    "if not modelName:\n",
    "  modelName = os.path.basename(modelUrl)\n",
    "modelPath = os.path.join(pretrained_model, modelName)\n",
    "\n",
    "#@markdown ##<br> Download VAE (Optional)\n",
    "vaeUrl = [\"\",\n",
    "          \"https://huggingface.co/Linaqruf/personal-backup/resolve/main/vae/animevae.pt\",\n",
    "          \"https://huggingface.co/hakurei/waifu-diffusion-v1-4/resolve/main/vae/kl-f8-anime.ckpt\",\n",
    "          \"https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt\"]\n",
    "vaeList = [\"none\",\n",
    "           \"anime.vae.pt\",\n",
    "           \"waifudiffusion.vae.pt\",\n",
    "           \"stablediffusion.vae.pt\"]\n",
    "vaeName = \"anime.vae.pt\" #@param [\"none\", \"anime.vae.pt\", \"waifudiffusion.vae.pt\", \"stablediffusion.vae.pt\"]\n",
    "vaePath = os.path.join(vae, vaeName)\n",
    "\n",
    "installVae.append((vaeName, vaeUrl[vaeList.index(vaeName)]))\n",
    "\n",
    "def install_model(url, name, is_vae):\n",
    "  ext = \"ckpt\" if url.endswith(\".ckpt\") else \"safetensors\"\n",
    "  if not is_vae:\n",
    "    if url.startswith(\"https://drive.google.com\"):\n",
    "      os.chdir(pretrained_model)\n",
    "      !gdown --fuzzy {url}\n",
    "    elif url.startswith(\"https://huggingface.co/\"):\n",
    "      if '/blob/' in url:\n",
    "        url = url.replace('/blob/', '/resolve/')\n",
    "      !aria2c --console-log-level=error --summary-interval=10 --header={user_header} -c -x 16 -k 1M -s 16 -d {pretrained_model} -o {name} {url}\n",
    "    else:\n",
    "      !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d {pretrained_model} -o {name} {url}\n",
    "  else:\n",
    "    !aria2c --console-log-level=error --summary-interval=10 --header={user_header} -c -x 16 -k 1M -s 16 -o vae/{name} \"{url}\"\n",
    "\n",
    "os.chdir(root_dir)\n",
    "install_model(modelUrl, modelName, False)\n",
    "if vaeName != \"none\":\n",
    "  for vae in installVae:\n",
    "      install_model(vae[1], vae[0], True)\n",
    "\n",
    "# Unzip Dataset if any\n",
    "def download_dataset(url):\n",
    "  if url.startswith(\"/content\"):\n",
    "    !unzip -j -o {url} -d \"{train_data_dir}\"\n",
    "  elif url.startswith(\"https://drive.google.com\"):\n",
    "    os.chdir(root_dir)\n",
    "    !gdown --fuzzy  {url}\n",
    "  elif url.startswith(\"https://huggingface.co/\"):\n",
    "    if '/blob/' in url:\n",
    "      url = url.replace('/blob/', '/resolve/')\n",
    "    !aria2c --console-log-level=error --summary-interval=10 --header={user_header} -c -x 16 -k 1M -s 16 -d {root_dir} -o {zipfile} {url}\n",
    "  else:\n",
    "    !aria2c --console-log-level=error --summary-interval=10 -c -x 16 -k 1M -s 16 -d {root_dir} -o {zipfile} {url}\n",
    "\n",
    "if dataset_zip_url:\n",
    "  download_dataset(dataset_zip_url)\n",
    "\n",
    "  !unzip -j -o {os.path.join(root_dir,zipfile)} -d \"{train_data_dir}\"\n",
    "  os.remove(os.path.join(root_dir,zipfile))\n",
    "\n",
    "  files_to_move = (\"meta_cap.json\",\n",
    "                    \"meta_cap_dd.json\",\n",
    "                    \"meta_lat.json\",\n",
    "                    \"meta_clean.json\")\n",
    "\n",
    "  for filename in os.listdir(train_data_dir):\n",
    "    file_path = os.path.join(train_data_dir, filename)\n",
    "    if filename in files_to_move:\n",
    "      if not os.path.exists(file_path):\n",
    "        shutil.move(file_path, training_dir)\n",
    "      else: \n",
    "        os.remove(file_path)\n",
    "\n",
    "# Clean Dataset\n",
    "for item in os.listdir(train_data_dir):\n",
    "    file_ext = os.path.splitext(item)[1]\n",
    "    item_path = os.path.join(train_data_dir, item)\n",
    "    if os.path.isfile(item_path):\n",
    "        if file_ext not in supported_types:\n",
    "            print(f\"Deleting unsupported file {item} from {train_data_dir}\")\n",
    "            os.remove(item_path)\n",
    "    else:\n",
    "        print(f\"Skipping directory {item}\")\n",
    "\n",
    "#@markdown ##<br> Auto-captioning\n",
    "#@markdown <small><font color=gray> **HINT**: this part will be skipped if you have `any` `.caption` or `.txt` inside your `train_data_dir` </small><br> \n",
    "use_wd_tagger = True #@param{type:\"boolean\"}\n",
    "use_blip = True #@param{type:\"boolean\"}\n",
    "\n",
    "#@markdown ##<br> Custom Tag\n",
    "extension = \"txt\" #@param [\"txt\", \"caption\"]\n",
    "custom_tag = \"\" #@param {type:\"string\"}\n",
    "#@markdown Tick this if you want to append custom tag at the end of lines instead\n",
    "append = False #@param {type:\"boolean\"}\n",
    "keep_tokens = 1 #@param {type:\"number\"}\n",
    "\n",
    "os.chdir(finetune_dir)\n",
    "\n",
    "if use_wd_tagger:\n",
    "  if not any([filename.endswith(\".txt\") for filename in os.listdir(train_data_dir)]):\n",
    "    !python tag_images_by_wd14_tagger.py \\\n",
    "      \"{train_data_dir}\" \\\n",
    "      --batch_size 8 \\\n",
    "      --repo_id \"SmilingWolf/wd-v1-4-swinv2-tagger-v2\" \\\n",
    "      --thresh 0.35 \\\n",
    "      --caption_extension .txt \\\n",
    "      --max_data_loader_n_workers 2\n",
    "if use_blip:\n",
    "  if not any([filename.endswith(\".caption\") for filename in os.listdir(train_data_dir)]):\n",
    "    !python make_captions.py \\\n",
    "      \"{train_data_dir}\" \\\n",
    "      --batch_size 8 \\\n",
    "      --beam_search \\\n",
    "      --caption_extension .caption \\\n",
    "      --max_data_loader_n_workers 2\n",
    "\n",
    "def add_tag(filename, tag, append):\n",
    "    with open(filename, \"r\") as f:\n",
    "        contents = f.read()\n",
    "        \t    \t\n",
    "    tag = \", \".join(tag.split())\n",
    "    tag = tag.replace(\"_\", \" \")\n",
    "    \n",
    "    if tag in contents:\n",
    "        return\n",
    "        \n",
    "    if not keep_tokens:\n",
    "      contents = contents.rstrip() + \", \" + tag if append else tag + \", \" + contents\n",
    "    else:\n",
    "      contents = tag + \", \" + contents\n",
    "    \n",
    "    with open(filename, \"w\") as f:\n",
    "        f.write(contents)\n",
    "\n",
    "if custom_tag:\n",
    "  if not any([filename.endswith(\".\" + extension) for filename in os.listdir(train_data_dir)]):\n",
    "      for filename in os.listdir(train_data_dir):\n",
    "          if filename.endswith((\".png\", \".jpg\", \".jpeg\", \".webp\", \".bmp\")):\n",
    "              open(os.path.join(train_data_dir, filename.split(\".\")[0] + \".\" + extension), \"w\").close()\n",
    "\n",
    "  tags = custom_tag.split()\n",
    "\n",
    "  for filename in os.listdir(train_data_dir):\n",
    "      if filename.endswith(\".\" + extension):\n",
    "          for tag in tags:\n",
    "              add_tag(os.path.join(train_data_dir, filename), tag, append)\n",
    "\n",
    "# Create JSON file for Finetuning\n",
    "if use_dreambooth_method:\n",
    "  files = [f for f in os.listdir(train_data_dir) if f.endswith(\".txt\")]\n",
    "\n",
    "  for file in files:\n",
    "      file_path = os.path.join(train_data_dir, file)\n",
    "\n",
    "      with open(file_path, \"r\") as f:\n",
    "          contents = f.read()\n",
    "\n",
    "      contents = html.unescape(contents)\n",
    "      contents = contents.replace(\"_\", \" \")\n",
    "      contents = \", \".join(contents.split(\"\\n\"))\n",
    "\n",
    "      with open(file_path, \"w\") as f:\n",
    "          f.write(contents)\n",
    "else:\n",
    "  if os.path.exists(train_data_dir):\n",
    "    if any(file.endswith('.caption') for file in os.listdir(train_data_dir)):\n",
    "      !python merge_captions_to_metadata.py \\\n",
    "        {train_data_dir} \\\n",
    "        {meta_cap}\n",
    "\n",
    "    if any(file.endswith('.txt') for file in os.listdir(train_data_dir)):\n",
    "      !python merge_dd_tags_to_metadata.py \\\n",
    "        {train_data_dir} \\\n",
    "        {meta_cap_dd}\n",
    "  else:\n",
    "    print(\"train_data_dir does not exist or is not a directory.\")\n",
    "\n",
    "  if os.path.exists(meta_cap):\n",
    "    !python merge_dd_tags_to_metadata.py \\\n",
    "      {train_data_dir} \\\n",
    "      --in_json {meta_cap} \\\n",
    "      {meta_cap_dd}\n",
    "\n",
    "  if os.path.exists(meta_cap_dd):\n",
    "    !python clean_captions_and_tags.py \\\n",
    "      {meta_cap_dd} \\\n",
    "      {meta_clean}\n",
    "  elif os.path.exists(meta_cap):\n",
    "    !python clean_captions_and_tags.py \\\n",
    "      {meta_cap} \\\n",
    "      {meta_clean}\n",
    "\n",
    "#@markdown ##<br> LoRA Config\n",
    "\n",
    "network_dim = 128 #@param {'type':'number'}\n",
    "network_alpha = 128 #@param {'type':'number'}\n",
    "#@markdown `network_weights` can be specified to resume training.\n",
    "network_weights = \"\" #@param {'type':'string'}\n",
    "unet_lr = 1e-4 #@param {'type':'number'}\n",
    "text_encoder_lr = 5e-5 #@param {'type':'number'}\n",
    "\n",
    "#@markdown ##<br> Optimizer Config\n",
    "#@markdown `AdamW8bit` was the old `--use_8bit_adam`. Use `DAdaptation` if you find it hard to set optimizer hyperparameter right.\n",
    "optimizer_type = \"AdamW8bit\" #@param [\"AdamW\", \"AdamW8bit\", \"Lion\", \"SGDNesterov\", \"SGDNesterov8bit\", \"DAdaptation\", \"AdaFactor\"]\n",
    "#@markdown Additional arguments for optimizer, e.g: `\"decouple=True weight_decay=0.01 betas=0.9,0.999 ...\"`\n",
    "optimizer_args = \"\" #@param {'type':'string'}\n",
    "\n",
    "#@markdown <small><font color=gray> **HINT**: for LoRA if you specify both `--unet_lr` and `--text_encoder_lr` you don't need this, however it's still recorded to metadata</small><br>  \n",
    "learning_rate = 2e-6 #@param {type:\"number\"}\n",
    "lr_scheduler = \"constant\" #@param [\"linear\", \"cosine\", \"cosine_with_restarts\", \"polynomial\", \"constant\", \"constant_with_warmup\"] {allow-input: false}\n",
    "lr_warmup_steps = 250 #@param {'type':'number'}\n",
    "#@markdown You can define `num_cycles` value for `cosine_with_restarts` or `power` value for `polynomial` in the field below.\n",
    "lr_scheduler_args = 1 #@param {'type':'number'}\n",
    "\n",
    "#@markdown ##<br> Training Config\n",
    "#@markdown <small><font color=gray> **HINT**: specify `v2` if you train on SDv2 base Model, with `v2_parameterization` for SDv2 768 Model</small><br>  \n",
    "v2 = False #@param{type:\"boolean\"}\n",
    "v_parameterization = False #@param {type:\"boolean\"}\n",
    "resolution = 512 #@param {type:\"slider\", min:512, max:768, step:128}\n",
    "flip_aug = False #@param{type:\"boolean\"}\n",
    "#@markdown Read [Diffusion With Offset Noise](https://www.crosslabs.org//blog/diffusion-with-offset-noise), in short, you can control and easily generating darker or light images by offset the noise when fine-tuning the model. Set to `0` by default, recommended value: `0.1`\n",
    "noise_offset = 0 #@param {type:\"number\"}\n",
    "#@markdown <small><font color=gray> **HINT**: try lowering `train_batch_size` if you do native training, around 4 batch sizes\n",
    "train_batch_size = 4 #@param {type:\"number\"}\n",
    "\n",
    "s = getoutput('nvidia-smi')\n",
    "if 'T4' in s:\n",
    "  if mode == \"native-training\" and train_batch_size > 4:\n",
    "    train_batch_size = 4\n",
    "  elif mode == \"LoRA\" and train_batch_size > 8:\n",
    "    train_batch_size = 8\n",
    "\n",
    "max_train_type = \"max_train_epochs\" #@param [\"max_train_steps\", \"max_train_epochs\"]\n",
    "max_train_type_value = 20 #@param {type:\"number\"}\n",
    "dataset_repeats = 10 #@param {type:\"number\"}\n",
    "mixed_precision = \"fp16\" #@param [\"no\",\"fp16\",\"bf16\"] {allow-input: false}\n",
    "save_precision = \"fp16\" #@param [\"float\", \"fp16\", \"bf16\"] {allow-input: false}\n",
    "save_n_epochs_type = \"save_n_epoch_ratio\" #@param [\"save_every_n_epochs\", \"save_n_epoch_ratio\"] {allow-input: false}\n",
    "save_n_epochs_type_value = 3 #@param {type:\"number\"}\n",
    "save_model_as = \"safetensors\" #@param [\"ckpt\", \"pt\", \"safetensors\"] {allow-input: false}\n",
    "clip_skip = 2 #@param {type:\"number\"}\n",
    "logging_dir = \"/content/training_dir/logs\"\n",
    "additional_argument = \"--shuffle_caption --xformers\" #@param {type:\"string\"}\n",
    "print_hyperparameter = True\n",
    "\n",
    "# Absolute value\n",
    "min_bucket_reso = 320 if resolution > 640 else 256\n",
    "max_bucket_reso = 1280 if resolution > 640 else 1024\n",
    "prior_loss_weight = 1.0\n",
    "\n",
    "# Dreambooth Config\n",
    "\n",
    "dreambooth_dir = f\"{dataset_repeats}_{instance_token}\"\n",
    "dreambooth_data_dir = os.path.join(train_data_dir, dreambooth_dir)\n",
    "reg_data_dir = os.path.join(os.path.dirname(train_data_dir), \"reg_data\")\n",
    "\n",
    "if use_dreambooth_method:\n",
    "  os.makedirs(dreambooth_data_dir, exist_ok=True)\n",
    "  os.makedirs(reg_data_dir, exist_ok=True)\n",
    "\n",
    "  for dataset in os.listdir(train_data_dir):\n",
    "      file_ext = os.path.splitext(dataset)[1]\n",
    "      if file_ext in supported_types:\n",
    "          source = os.path.join(train_data_dir, dataset)\n",
    "          destination = os.path.join(dreambooth_data_dir, dataset)\n",
    "          shutil.move(source, destination)\n",
    "else:\n",
    "  if os.path.exists(dreambooth_data_dir):\n",
    "    for dataset in os.listdir(dreambooth_data_dir):\n",
    "        source = os.path.join(dreambooth_data_dir, dataset)\n",
    "        destination = os.path.join(train_data_dir, dataset)\n",
    "        shutil.move(source, destination)\n",
    "    if not os.listdir(dreambooth_data_dir):\n",
    "        shutil.rmtree(dreambooth_data_dir)\n",
    "\n",
    "# V2 Config\n",
    "if v2 and not v_parameterization:\n",
    "  inference_url += \"v2-inference.yaml\"\n",
    "if v2 and v_parameterization:\n",
    "  inference_url += \"v2-inference-v.yaml\"\n",
    "\n",
    "# Download config\n",
    "try:\n",
    "  if v2:\n",
    "    !wget -c {inference_url} -O {training_dir}/{project_name}.yaml\n",
    "    print(\"File successfully downloaded\")\n",
    "except:\n",
    "  print(\"There was an error downloading the file. Please check the URL and try again.\")\n",
    "\n",
    "# Max Resolution\n",
    "if resolution == 512:\n",
    "  max_resolution = \"512,512\"\n",
    "elif resolution == 640:\n",
    "  max_resolution = \"640,640\"\n",
    "else:\n",
    "  max_resolution = \"768,768\"\n",
    "\n",
    "# Run script to prepare buckets and latent\n",
    "if not use_dreambooth_method:\n",
    "  if not os.path.exists(meta_lat):\n",
    "    bucket_latents=f\"\"\"\n",
    "    python prepare_buckets_latents.py \\\n",
    "      {train_data_dir} \\\n",
    "      {meta_clean} \\\n",
    "      {meta_lat} \\\n",
    "      {modelPath} \\\n",
    "      {\"--v2\" if v2 else \"\"} \\\n",
    "      {\"--flip_aug\" if flip_aug else \"\"} \\\n",
    "      {\"--min_bucket_reso \" + format(320) if resolution != 512 else \"--min_bucket_reso \" + format(256)} \\\n",
    "      {\"--max_bucket_reso \" + format(1280) if resolution != 512 else \"--max_bucket_reso \" + format(1024)} \\\n",
    "      {\"--batch_size \" + format(8)} \\\n",
    "      {\"--max_resolution \" + format(max_resolution)} \\\n",
    "      --mixed_precision no\n",
    "      \"\"\"\n",
    "    f = open(\"./bucket_latents.sh\", \"w\")\n",
    "    f.write(bucket_latents)\n",
    "    f.close()\n",
    "    !chmod +x ./bucket_latents.sh\n",
    "    !./bucket_latents.sh\n",
    "\n",
    "# Start Training\n",
    "os.chdir(repo_dir)\n",
    "train_command=f\"\"\"\n",
    "accelerate launch --config_file={accelerate_config} --num_cpu_threads_per_process=8 {f\"{repo_dir}/train_network.py\" if mode == \"LoRA\" else (f\"{repo_dir}/fine_tune.py\" if not use_dreambooth_method else f\"{repo_dir}/train_db.py\")} \\\n",
    "  {\"--v2\" if v2 else \"\"} \\\n",
    "  {\"--v_parameterization\" if v2 and v_parameterization else \"\"} \\\n",
    "  {\"--output_name=\" + project_name if project_name else \"\"} \\\n",
    "  --pretrained_model_name_or_path={modelPath} \\\n",
    "  {\"--vae=\" + vaePath if vaePath else \"\"} \\\n",
    "  --train_data_dir={train_data_dir} \\\n",
    "  {\"--reg_data_dir=\" + reg_data_dir if use_dreambooth_method else \"\"} \\\n",
    "  {\"--in_json=\" + meta_lat if not use_dreambooth_method else \"\"} \\\n",
    "  --output_dir={output_dir} \\\n",
    "  {\"--network_dim=\" + format(network_dim) if mode == \"LoRA\" else \"\"} \\\n",
    "  {\"--network_alpha=\" + format(network_alpha) if mode == \"LoRA\" else \"\"} \\\n",
    "  {\"--network_module=networks.lora\" if mode == \"LoRA\" else \"\"} \\\n",
    "  {(\"--network_weights=\" + network_weights if network_weights else \"\") if mode == \"LoRA\" else \"\"} \\\n",
    "  {(\"--unet_lr=\" + format(unet_lr) if unet_lr else \"\") if mode == \"LoRA\" else \"\"} \\\n",
    "  {(\"--text_encoder_lr=\" + format(text_encoder_lr) if text_encoder_lr else \"\") if mode == \"LoRA\" else \"\"} \\\n",
    "  --optimizer_type={optimizer_type} \\\n",
    "  {\"--optimizer_args=\" + optimizer_args if optimizer_args else \"\"} \\\n",
    "  --learning_rate={learning_rate} \\\n",
    "  --lr_scheduler={lr_scheduler} \\\n",
    "  {\"--lr_warmup_steps=\" + format(lr_warmup_steps) if lr_warmup_steps else \"\"} \\\n",
    "  {\"--lr_scheduler_num_cycles=\" + format(lr_scheduler_args) if lr_scheduler == \"cosine_with_restarts\" else \"\"} \\\n",
    "  {\"--lr_scheduler_power=\" + format(lr_scheduler_args) if lr_scheduler == \"polynomial\" else \"\"} \\\n",
    "  {\"--dataset_repeats=\" + format(dataset_repeats) if not use_dreambooth_method else \"\"} \\\n",
    "  --resolution={resolution} \\\n",
    "  {\"--enable_bucket\" if use_dreambooth_method else \"\"} \\\n",
    "  {\"--keep_tokens=\" + format(keep_tokens) if custom_tag else \"\"} \\\n",
    "  {\"--min_bucket_reso=\" + format(min_bucket_reso) if use_dreambooth_method else \"\"} \\\n",
    "  {\"--max_bucket_reso=\" + format(max_bucket_reso) if use_dreambooth_method else \"\"} \\\n",
    "  {\"--caption_extension=\" + caption_extension if use_dreambooth_method else \"\"} \\\n",
    "  {\"--cache_latents\" if use_dreambooth_method else \"\"} \\\n",
    "  {\"--prior_loss_weight=\" + format(prior_loss_weight) if use_dreambooth_method else \"\"} \\\n",
    "  {\"--lowram\" if lowram else \"\"} \\\n",
    "  {\"--noise_offset=\" + format(noise_offset) if noise_offset > 0 else \"\"} \\\n",
    "  --mixed_precision={mixed_precision} \\\n",
    "  --save_precision={save_precision} \\\n",
    "  {\"--save_every_n_epochs=\" + format(save_n_epochs_type_value) if save_n_epochs_type==\"save_every_n_epochs\" else \"\"} \\\n",
    "  {\"--save_n_epoch_ratio=\" + format(save_n_epochs_type_value) if save_n_epochs_type==\"save_n_epoch_ratio\" else \"\"} \\\n",
    "  --save_model_as={save_model_as} \\\n",
    "  --train_batch_size={train_batch_size} \\\n",
    "  {\"--max_token_length=\" + format(225)} \\\n",
    "  {\"--max_train_epochs=\" + format(max_train_type_value) if max_train_type == \"max_train_epochs\" else \"\"} \\\n",
    "  {\"--max_train_steps=\" + format(max_train_type_value) if max_train_type == \"max_train_steps\" else \"\"} \\\n",
    "  {\"--clip_skip=\" + format(clip_skip) if not v2 else \"\"} \\\n",
    "  --logging_dir={logging_dir} \\\n",
    "  --log_prefix={project_name} \\\n",
    "  {additional_argument}\n",
    "  \"\"\"\n",
    "\n",
    "debug_params = [\"mode\",\n",
    "                \"use_dreambooth_method\",\n",
    "                \"lowram\",\n",
    "                \"v2\",\n",
    "                \"v_parameterization\",\n",
    "                \"project_name\",\n",
    "                \"modelPath\",\n",
    "                \"vaePath\",\n",
    "                \"train_data_dir\",\n",
    "                \"reg_data_dir\" if use_dreambooth_method else \"\",\n",
    "                \"meta_lat\" if not use_dreambooth_method else \"\",\n",
    "                \"output_dir\",\n",
    "                \"network_dim\" if mode == \"LoRA\" else \"\" ,\n",
    "                \"network_alpha\" if mode == \"LoRA\" else \"\" ,\n",
    "                \"network_weights\" if mode == \"LoRA\" else \"\",\n",
    "                \"unet_lr\" if mode == \"LoRA\" else \"\",\n",
    "                \"text_encoder_lr\" if mode == \"LoRA\" else \"\",\n",
    "                \"optimizer_type\", \n",
    "                \"optimizer_args\",\n",
    "                \"learning_rate\",\n",
    "                \"lr_scheduler\",\n",
    "                \"lr_warmup_steps\", \n",
    "                \"lr_scheduler_args\",\n",
    "                \"keep_tokens\" if custom_tag else \"\",\n",
    "                \"dataset_repeats\" if not use_dreambooth_method else \"\",\n",
    "                \"min_bucket_reso\" if use_dreambooth_method else \"\",\n",
    "                \"max_bucket_reso\" if use_dreambooth_method else \"\",\n",
    "                \"resolution\",                \n",
    "                \"caption_extension\" if use_dreambooth_method else \"\",\n",
    "                \"noise_offset\",\n",
    "                \"prior_loss_weight\" if use_dreambooth_method else \"\",\n",
    "                \"mixed_precision\",\n",
    "                \"save_precision\",\n",
    "                \"save_n_epochs_type\",\n",
    "                \"save_n_epochs_type_value\",\n",
    "                \"save_model_as\",\n",
    "                \"train_batch_size\",\n",
    "                \"max_train_type\",\n",
    "                \"max_train_type_value\",\n",
    "                \"clip_skip\",\n",
    "                \"logging_dir\",\n",
    "                \"additional_argument\"]\n",
    "\n",
    "if print_hyperparameter:\n",
    "    table = PrettyTable()\n",
    "    table.field_names = [\"Hyperparameter\", \"Value\"]\n",
    "    for params in debug_params:\n",
    "        if params != \"\":\n",
    "            if globals()[params] == \"\":\n",
    "                value = \"False\"\n",
    "            else:\n",
    "                value = globals()[params]\n",
    "            table.add_row([params, value])\n",
    "    table.align = \"l\"\n",
    "    print(table)\n",
    "\n",
    "    arg_list = train_command.split()\n",
    "    mod_train_command = {'command': arg_list}\n",
    "    \n",
    "    if mode == \"LoRA\":\n",
    "      # save the YAML string to a file\n",
    "      with open((f'{training_dir}/dreambooth_lora_cmd.yaml' if use_dreambooth_method else f'{training_dir}/finetune_lora_cmd.yaml'), 'w') as f:\n",
    "          yaml.dump(mod_train_command, f)\n",
    "    else:\n",
    "      with open((f'{training_dir}/dreambooth_cmd.yaml' if use_dreambooth_method else f'{training_dir}/finetune_cmd.yaml'), 'w') as f:\n",
    "            yaml.dump(mod_train_command, f)\n",
    "\n",
    "f = open(\"./train.sh\", \"w\")\n",
    "f.write(train_command)\n",
    "f.close()\n",
    "!chmod +x ./train.sh\n",
    "!./train.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VQrywRr9UZVx"
   },
   "source": [
    "## Extras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "P5ZZ-xmPM8Gh"
   },
   "outputs": [],
   "source": [
    "#@title ## Inference\n",
    "import os\n",
    "\n",
    "#@markdown ### LoRA Config <br>\n",
    "#@markdown <small><font color=gray> **HINT**: leave this empty if doing native training\n",
    "network_weight = \"\" #@param {'type':'string'}\n",
    "if not network_weight and project_name:\n",
    "  network_weight = f\"{output_dir}/last.safetensors\"\n",
    "elif not network_weight and project_name:\n",
    "  network_weight = f\"{output_dir}/{project_name}.safetensors\"\n",
    "network_mul = 0.5 #@param {'type':'number'}\n",
    "\n",
    "#@markdown ### General Config\n",
    "v2 = False #@param {type:\"boolean\"}\n",
    "v_parameterization = True #@param {type:\"boolean\"}\n",
    "instance_prompt = \"\" #@param {type: \"string\"}\n",
    "prompt = \"masterpiece, 1girl, solo, long hair, looking at viewer, bangs, brown hair, dress, twintails, brown eyes, white shirt, upper body, outdoors, parted lips, day, bag, tree, ground vehicle, building, motor vehicle, realistic, car, road, street\" #@param {type: \"string\"}\n",
    "negative = \"lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry\" #@param {type: \"string\"}\n",
    "model = \"\" #@param {type: \"string\"}\n",
    "if not model:\n",
    "  model = modelPath\n",
    "vae = \"\" #@param {type: \"string\"}\n",
    "outdir = \"/content/images\" #@param {type: \"string\"}\n",
    "scale = 11 #@param {type: \"slider\", min: 1, max: 40}\n",
    "sampler = \"ddim\" #@param [\"ddim\", \"pndm\", \"lms\", \"euler\", \"euler_a\", \"heun\", \"dpm_2\", \"dpm_2_a\", \"dpmsolver\",\"dpmsolver++\", \"dpmsingle\", \"k_lms\", \"k_euler\", \"k_euler_a\", \"k_dpm_2\", \"k_dpm_2_a\"]\n",
    "steps = 28 #@param {type: \"slider\", min: 1, max: 100}\n",
    "precision = \"fp16\" #@param [\"fp16\", \"bf16\"] {allow-input: false}\n",
    "width = 512 #@param {type: \"integer\"}\n",
    "height = 768 #@param {type: \"integer\"}\n",
    "images_per_prompt = 4 #@param {type: \"integer\"}\n",
    "batch_size = 4 #@param {type: \"integer\"}\n",
    "clip_skip = 2 #@param {type: \"slider\", min: 1, max: 40}\n",
    "seed = -1 #@param {type: \"integer\"}\n",
    "\n",
    "final_prompt = f\"{instance_prompt}, {prompt} --n {negative}\" if instance_prompt else f\"{prompt} --n {negative}\"\n",
    "\n",
    "os.chdir(repo_dir)\n",
    "\n",
    "!python gen_img_diffusers.py \\\n",
    "  {\"--v2\" if v2 else \"\"} \\\n",
    "  {\"--v_parameterization\" if v2 and v_parameterization else \"\"} \\\n",
    "  {\"--network_module=networks.lora\" if network_weight else \"\"} \\\n",
    "  {\"--network_weight=\" + network_weight if network_weight else \"\"} \\\n",
    "  {\"--network_mul=\" + format(network_mul) if network_weight else \"\"} \\\n",
    "  --ckpt={model} \\\n",
    "  --outdir={outdir} \\\n",
    "  --xformers \\\n",
    "  {\"--vae=\" + vae if vae else \"\"} \\\n",
    "  --{precision} \\\n",
    "  --W={width} \\\n",
    "  --H={height} \\\n",
    "  {\"--seed=\" + format(seed) if seed > 0 else \"\"} \\\n",
    "  --scale={scale} \\\n",
    "  --sampler={sampler} \\\n",
    "  --steps={steps} \\\n",
    "  --max_embeddings_multiples=3 \\\n",
    "  --batch_size={batch_size} \\\n",
    "  --images_per_prompt={images_per_prompt} \\\n",
    "  {\"--clip_skip=\" + format(clip_skip) if v2 == False else \"\"} \\\n",
    "  --prompt=\"{final_prompt}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "QTXsM170GUpk"
   },
   "outputs": [],
   "source": [
    "# @title Upload to Huggingface hub\n",
    "from huggingface_hub import login\n",
    "from huggingface_hub import HfApi\n",
    "from huggingface_hub.utils import validate_repo_id, HfHubHTTPError\n",
    "\n",
    "# @markdown Login to Huggingface Hub\n",
    "# @markdown > Get **your** huggingface `WRITE` token [here](https://huggingface.co/settings/tokens)\n",
    "write_token = \"\"  # @param {type:\"string\"}\n",
    "login(write_token, add_to_git_credential=True)\n",
    "\n",
    "api = HfApi()\n",
    "user = api.whoami(write_token)\n",
    "\n",
    "# @markdown Fill this if you want to upload to your organization, or just leave it empty.\n",
    "\n",
    "orgs_name = \"\"  # @param{type:\"string\"}\n",
    "\n",
    "# @markdown If your model/dataset repo didn't exist, it will automatically create your repo.\n",
    "model_name = \"your-model-name\"  # @param{type:\"string\"}\n",
    "dataset_name = \"your-dataset-name\"  # @param{type:\"string\"}\n",
    "make_this_model_private = True  # @param{type:\"boolean\"}\n",
    "\n",
    "if orgs_name == \"\":\n",
    "    model_repo = user[\"name\"] + \"/\" + model_name.strip()\n",
    "    datasets_repo = user[\"name\"] + \"/\" + dataset_name.strip()\n",
    "else:\n",
    "    model_repo = orgs_name + \"/\" + model_name.strip()\n",
    "    datasets_repo = orgs_name + \"/\" + dataset_name.strip()\n",
    "\n",
    "if model_name != \"\":\n",
    "    try:\n",
    "        validate_repo_id(model_repo)\n",
    "        api.create_repo(repo_id=model_repo, private=make_this_model_private)\n",
    "        print(\"Model Repo didn't exists, creating repo\")\n",
    "        print(\"Model Repo: \", model_repo, \"created!\\n\")\n",
    "\n",
    "    except HfHubHTTPError as e:\n",
    "        print(f\"Model Repo: {model_repo} exists, skipping create repo\\n\")\n",
    "\n",
    "if dataset_name != \"\":\n",
    "    try:\n",
    "        validate_repo_id(datasets_repo)\n",
    "        api.create_repo(\n",
    "            repo_id=datasets_repo, repo_type=\"dataset\", private=make_this_model_private\n",
    "        )\n",
    "        print(\"Dataset Repo didn't exists, creating repo\")\n",
    "        print(\"Dataset Repo\", datasets_repo, \"created!\\n\")\n",
    "\n",
    "    except HfHubHTTPError as e:\n",
    "        print(f\"Dataset repo: {datasets_repo} exists, skipping create repo\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "CIeoJA-eO-8t"
   },
   "outputs": [],
   "source": [
    "# @title Upload Model\n",
    "from huggingface_hub import HfApi\n",
    "from pathlib import Path\n",
    "\n",
    "%store -r\n",
    "\n",
    "api = HfApi()\n",
    "\n",
    "# @markdown This will be uploaded to model repo\n",
    "model_path = \"/content/training_dir/output/last.safetensors\"  # @param {type :\"string\"}\n",
    "path_in_repo = \"\"  # @param {type :\"string\"}\n",
    "\n",
    "# @markdown Other Information\n",
    "commit_message = \"\"  # @param {type :\"string\"}\n",
    "\n",
    "if not commit_message:\n",
    "    commit_message = \"feat: upload \" + project_name + \" lora model\"\n",
    "\n",
    "\n",
    "def upload_model(model_paths, is_folder: bool):\n",
    "    path_obj = Path(model_paths)\n",
    "    trained_model = path_obj.parts[-1]\n",
    "\n",
    "    if path_in_repo:\n",
    "        trained_model = path_in_repo\n",
    "\n",
    "    if is_folder == True:\n",
    "        print(f\"Uploading {trained_model} to https://huggingface.co/\" + model_repo)\n",
    "        print(f\"Please wait...\")\n",
    "\n",
    "        api.upload_folder(\n",
    "            folder_path=model_paths,\n",
    "            path_in_repo=trained_model,\n",
    "            repo_id=model_repo,\n",
    "            commit_message=commit_message,\n",
    "            ignore_patterns=\".ipynb_checkpoints\",\n",
    "        )\n",
    "        print(\n",
    "            f\"Upload success, located at https://huggingface.co/\"\n",
    "            + model_repo\n",
    "            + \"/tree/main\\n\"\n",
    "        )\n",
    "    else:\n",
    "        print(f\"Uploading {trained_model} to https://huggingface.co/\" + model_repo)\n",
    "        print(f\"Please wait...\")\n",
    "\n",
    "        api.upload_file(\n",
    "            path_or_fileobj=model_paths,\n",
    "            path_in_repo=trained_model,\n",
    "            repo_id=model_repo,\n",
    "            commit_message=commit_message,\n",
    "        )\n",
    "\n",
    "        print(\n",
    "            f\"Upload success, located at https://huggingface.co/\"\n",
    "            + model_repo\n",
    "            + \"/blob/main/\"\n",
    "            + trained_model\n",
    "            + \"\\n\"\n",
    "        )\n",
    "\n",
    "\n",
    "def upload():\n",
    "    if model_path.endswith((\".ckpt\", \".safetensors\", \".pt\")):\n",
    "        upload_model(model_path, False)\n",
    "    else:\n",
    "        upload_model(model_path, True)\n",
    "\n",
    "\n",
    "upload()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "IW-hS9jnmf-E"
   },
   "outputs": [],
   "source": [
    "# @title Upload Dataset\n",
    "from huggingface_hub import HfApi\n",
    "from pathlib import Path\n",
    "import shutil\n",
    "import zipfile\n",
    "import os\n",
    "\n",
    "api = HfApi()\n",
    "\n",
    "# @markdown This will be compressed to zip and  uploaded to datasets repo, leave it empty if not necessary\n",
    "train_data_path = \"/content/training_dir/train_data\"  # @param {type :\"string\"}\n",
    "meta_lat_path = \"/content/training_dir/meta_lat.json\"  # @param {type :\"string\"}\n",
    "# @markdown `Nerd stuff, only if you want to save training logs`\n",
    "logs_path = \"/content/training_dir/logs\"  # @param {type :\"string\"}\n",
    "\n",
    "if project_name != \"\":\n",
    "    tmp_dataset = \"/content/training_dir/\" + project_name + \"_dataset\"\n",
    "else:\n",
    "    tmp_dataset = \"/content/training_dir/tmp_dataset\"\n",
    "\n",
    "tmp_train_data = tmp_dataset + \"/train_data\"\n",
    "dataset_zip = tmp_dataset + \".zip\"\n",
    "\n",
    "# @markdown Other Information\n",
    "commit_message = \"\"  # @param {type :\"string\"}\n",
    "\n",
    "if not commit_message:\n",
    "    commit_message = \"feat: upload \" + project_name + \" dataset and logs\"\n",
    "\n",
    "os.makedirs(tmp_dataset, exist_ok=True)\n",
    "os.makedirs(tmp_train_data, exist_ok=True)\n",
    "\n",
    "\n",
    "def upload_dataset(dataset_paths, is_zip: bool):\n",
    "    path_obj = Path(dataset_paths)\n",
    "    dataset_name = path_obj.parts[-1]\n",
    "\n",
    "    if is_zip:\n",
    "        print(\n",
    "            f\"Uploading {dataset_name} to https://huggingface.co/datasets/\"\n",
    "            + datasets_repo\n",
    "        )\n",
    "        print(f\"Please wait...\")\n",
    "\n",
    "        api.upload_file(\n",
    "            path_or_fileobj=dataset_paths,\n",
    "            path_in_repo=dataset_name,\n",
    "            repo_id=datasets_repo,\n",
    "            repo_type=\"dataset\",\n",
    "            commit_message=commit_message,\n",
    "        )\n",
    "        print(\n",
    "            f\"Upload success, located at https://huggingface.co/datasets/\"\n",
    "            + datasets_repo\n",
    "            + \"/blob/main/\"\n",
    "            + dataset_name\n",
    "            + \"\\n\"\n",
    "        )\n",
    "    else:\n",
    "        print(\n",
    "            f\"Uploading {dataset_name} to https://huggingface.co/datasets/\"\n",
    "            + datasets_repo\n",
    "        )\n",
    "        print(f\"Please wait...\")\n",
    "\n",
    "        api.upload_folder(\n",
    "            folder_path=dataset_paths,\n",
    "            path_in_repo=dataset_name,\n",
    "            repo_id=datasets_repo,\n",
    "            repo_type=\"dataset\",\n",
    "            commit_message=commit_message,\n",
    "            ignore_patterns=\".ipynb_checkpoints\",\n",
    "        )\n",
    "        print(\n",
    "            f\"Upload success, located at https://huggingface.co/datasets/\"\n",
    "            + datasets_repo\n",
    "            + \"/tree/main/\"\n",
    "            + dataset_name\n",
    "            + \"\\n\"\n",
    "        )\n",
    "\n",
    "\n",
    "def zip_file(tmp):\n",
    "    zipfiles = tmp + \".zip\"\n",
    "    with zipfile.ZipFile(zipfiles, \"w\") as zip:\n",
    "        for tmp, dirs, files in os.walk(tmp):\n",
    "            for file in files:\n",
    "                zip.write(os.path.join(tmp, file))\n",
    "\n",
    "\n",
    "def move(src_path, dst_path, is_metadata: bool):\n",
    "    files_to_move = [\n",
    "        \"meta_cap.json\",\n",
    "        \"meta_cap_dd.json\",\n",
    "        \"meta_lat.json\",\n",
    "        \"meta_clean.json\",\n",
    "        \"meta_final.json\",\n",
    "    ]\n",
    "\n",
    "    if os.path.exists(src_path):\n",
    "        shutil.move(src_path, dst_path)\n",
    "\n",
    "    if is_metadata:\n",
    "        parent_meta_path = os.path.dirname(src_path)\n",
    "\n",
    "        for filename in os.listdir(parent_meta_path):\n",
    "            file_path = os.path.join(parent_meta_path, filename)\n",
    "            if filename in files_to_move:\n",
    "                shutil.move(file_path, dst_path)\n",
    "\n",
    "\n",
    "def upload():\n",
    "    if train_data_path != \"\" and meta_lat_path != \"\":\n",
    "        move(train_data_path, tmp_train_data, False)\n",
    "        move(meta_lat_path, tmp_dataset, True)\n",
    "        zip_file(tmp_dataset)\n",
    "        upload_dataset(dataset_zip, True)\n",
    "    if logs_path != \"\":\n",
    "        upload_dataset(logs_path, False)\n",
    "\n",
    "\n",
    "upload()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "VQrywRr9UZVx"
   ],
   "include_colab_link": true,
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}