demo/builtin/rnnt/RNNT_DEMO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/main/demo/builtin/rnnt/RNNT_DEMO.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN-T Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Automatic speech recognition (ASR) systems convert audio into text representation. RNN-T is an end-to-end rnn based ASR model that directly output word transcripts given the input audio. This notebook contains step by step guide on how to optimize RNN-T model with Intel® End-to-End AI Optimization Kit, and detailed performance analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Content\n",
    "* [Overview](#Overview)\n",
    "    * [Model Architecture](#Model-Architecture)\n",
    "    * [Optimizations](#Optimizations)\n",
    "    * [Performance](#Performance)\n",
    "* [Getting Started](#Getting-Started)\n",
    "    * [1. Environment Setup](#1.-Environment-Setup)\n",
    "    * [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "    * [3. Data Prepare](#3.-Data-Prepare)\n",
    "    * [4. Train](#4.-Train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "<img src=\"./img/asr.png\" width=\"600\"/>\n",
    "\n",
    "* The traditional ASR system (top picture) contains acoustic, phonetic and language components that work together as in a pipeline system\n",
    "* The end-to-end ASR system is a single neural network that receives raw audio signal as input and provides a sequence of words at output"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Architecture\n",
    "<img src=\"./img/rnnt_structure.png\" width=\"250\"/>\n",
    "\n",
    "RNN-T is an end-to-end ASR model that directly converts audio into text representation.\n",
    "\n",
    "The encoder network is a RNN which maps input acoustic frames into a higher-level representation.\n",
    "The prediction network is a RNN that is explicitly conditioned on the history of previous non-blank targets predicted by the model.\n",
    "The joint network is a feed-forward network that combines the outputs of the prediction network and the encoder to produce logits followed by a softmax layer to produce a distribution over the next output symbol."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizations"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model architecture Intro\n",
    "\n",
    "For RNN-T model democratization, we enabled distributed training with pytorch DDP to scale out model training on multi nodes, added time stack layer and increased time stack factor to reduce input sequence lengh, added layer and batch normalization to speedup training converge, decreased layer size to get a lighter model.\n",
    "\n",
    "<img src=\"./img/model_base.png\" width=\"600\"/><figure>base model</figure>\n",
    "<img src=\"./img/model_opt.png\" width=\"800\"/><figure>democratized model</figure>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed training\n",
    "\n",
    "``` python\n",
    "# data parallel\n",
    "if world_size > 1:\n",
    "    model = DDP(model, find_unused_parameters=True)\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add time stack layer\n",
    "\n",
    "For ASR systems, the number of time frames for an audio input sequence is significantly higher than the number of output text labels. LSTM is sequential model which leads to much time cost in process long sequence data like audio data. The StackTime layer stacks audio frames to reduce sequence length and form a higher dimension input, which helps to speedup training process.\n",
    "\n",
    "```python\n",
    "class StackTime(nn.Module):\n",
    "    def __init__(self, factor):\n",
    "        super().__init__()\n",
    "        self.factor = int(factor)\n",
    "\n",
    "    def stack(self, x):\n",
    "        x = x.transpose(0, 1)\n",
    "        T = x.size(1)\n",
    "        padded = torch.nn.functional.pad(x, (0, 0, 0, (self.factor - (T % self.factor)) % self.factor))\n",
    "        B, T, H = padded.size()\n",
    "        x = padded.reshape(B, T // self.factor, -1)\n",
    "        x = x.transpose(0, 1)\n",
    "        return x\n",
    "\n",
    "    def forward(self, x, x_lens):\n",
    "        if type(x) is not list:\n",
    "            x = self.stack(x)\n",
    "            x_lens = (x_lens.int() + self.factor - 1) // self.factor\n",
    "            return x, x_lens\n",
    "        else:\n",
    "            if len(x) != 2:\n",
    "                raise NotImplementedError(\"Only number of seq segments equal to 2 is supported\")\n",
    "            assert x[0].size(1) % self.factor == 0, \"The length of the 1st seq segment should be multiple of stack factor\"\n",
    "            y0 = self.stack(x[0])\n",
    "            y1 = self.stack(x[1])\n",
    "            x_lens = (x_lens.int() + self.factor - 1) // self.factor\n",
    "            return [y0, y1], x_lens\n",
    "```\n",
    "\n",
    "About 4x speedup after increase time stack factor from 2 to 8.\n",
    "\n",
    "<img src=\"./img/time_stack_2.PNG\" width=\"800\"/><figure>time_stack = 2</figure>\n",
    "<img src=\"./img/time_stack_8.PNG\" width=\"800\"/><figure>time_stack = 8</figure>\n",
    "\n",
    "Profiling data proves that less time cost on forward/backward since input sequence reduced with time stack layer\n",
    "\n",
    "<img src=\"./img/stack_profile_base.png\" width=\"800\"/><figure>base model profiling</figure>\n",
    "<img src=\"./img/stack_profile_democratize.png\" width=\"800\"/><figure>democratized model profiling</figure>\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add layer normalization and batch normalization\n",
    "\n",
    "Layer normalization for LSTM is important to the success of RNN-T modeling. Add layer normalization for LSTM and batch normalization for input feature help to speedup training converge. It takes 52 epochs to converge without normalization, while only 49 epochs needed with normalization. \n",
    "\n",
    "```python\n",
    "enc_mod[\"batch_norm\"] = nn.BatchNorm1d(pre_rnn_input_size)\n",
    "```\n",
    "\n",
    "```python\n",
    "self.layer_norm = torch.nn.LayerNorm(hidden_size)\n",
    "```\n",
    "\n",
    "<img src=\"./img/no_norm.PNG\" width=\"800\"/><figure>without normalization</figure>\n",
    "<img src=\"./img/norm.PNG\" width=\"800\"/><figure>with normalization</figure>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HPO with SDA (Smart Democratization Advisor)\n",
    "\n",
    "SDA config\n",
    "\n",
    "```\n",
    "Parameters for SDA auto optimization:\n",
    "- learning_rate: 1.0e-3~1.0e-2 #training learning rate\n",
    "- warmup_epochs: 1~10 #epoch to warmup learning rate\n",
    "metrics:\n",
    "- name: training_time # training time threshold\n",
    "  objective: minimize\n",
    "  threshold: 43200\n",
    "- name: WER # training metric threshold\n",
    "  objective: minimize\n",
    "  threshold: 0.25\n",
    " ```\n",
    "\n",
    "request suggestions from SDA\n",
    "\n",
    "```python\n",
    "suggestion = self.conn.experiments(self.experiment.id).suggestions().create()\n",
    "```\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Framework related optimization\n",
    "\n",
    "leverage IPEX for distributed training and enable socket binding for training in two socket system\n",
    "\n",
    "```bash\n",
    "# Use IPEX launch to launch training, enable NUMA binding in two socket system.\n",
    "${CONDA_PREFIX}/bin/python -m intel_extension_for_pytorch.cpu.launch --distributed --nproc_per_node=2 --nnodes=4 --hostfile hosts train.py ${ARGS}\n",
    "```\n",
    "\n",
    "<img src=\"./img/no_numa_binding.png\" width=\"500\"/><figure>without numa binding</figure>\n",
    "<img src=\"./img/numa_binding.png\" width=\"500\"/><figure>enable numa binding</figure>\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance\n",
    "\n",
    "<img src=\"./img/rnnt_perf.png\" width=\"500\"/>\n",
    "\n",
    "* Distributed training with HW scaling delivered 3.83x speedup from 1 node to 4 nodes\n",
    "* HPO delivered 1.35x speedup, and 5.16x speedup over baseline\n",
    "* Time stacking + reduce LSTM layer size delivered 1.86x speedup, and 9.63x speedup over baseline\n",
    "* Add layer normalization in encoder and decoder, add batch normalization for input feature delivered 1.07x speedup, and 10.31x speedup over baseline\n",
    "* Reduce CCL worker number delivered 1.07x speedup, and 11.06x speedup over baseline\n",
    "* Thread affinity optimization delivered 1.28x speedup, and 14.19x speedup over baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "* [1. Environment Setup](#1.-Environment-Setup)\n",
    "* [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "* [3. Data Prepare](#3.-Data-Prepare)\n",
    "* [4. Train](#4.-Train)\n",
    "\n",
    "Notes: in order to run this demo, please follow `Environment Setup`, `Workflow Prepare` and `Data Prepare` section for pre-requirements."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Environment Setup\n",
    "\n",
    "### (Option1) Use Pip Install\n",
    "pre-work: move e2eAIOK source code to /home/vmagent/app/e2eaiok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install torchaudio==0.12.1 torch==1.12.1 --extra-index-url https://download.pytorch.org/whl/cpu\n",
    "pip install oneccl_bind_pt==1.12.100 intel-extension-for-pytorch==1.12.100 -f https://developer.intel.com/ipex-whl-stable\n",
    "pip install --extra-index-url https://developer.download.nvidia.com/compute/redist --upgrade nvidia-dali-cuda110==1.9.0\n",
    "pip install git+https://github.com/NVIDIA/dllogger#egg=dllogger\n",
    "pip install \"git+https://github.com/mlperf/logging.git@1.0.0\"\n",
    "pip install sentencepiece Unidecode tensorboard inflect soundfile librosa sox pandas pyyaml sigopt\n",
    "git clone https://github.com/HawkAaron/warp-transducer && cd warp-transducer \\\n",
    "    && mkdir build && cd build \\\n",
    "    && cmake .. && make && cd ../pytorch_binding \\\n",
    "    && python setup.py install\n",
    "pip install e2eAIOK-sda --pre\n",
    "apt install -y numactl"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Option2) Use Docker\n",
    "``` bash\n",
    "# Setup ENV\n",
    "git clone https://github.com/intel/e2eAIOK.git\n",
    "cd e2eAIOK\n",
    "git submodule update --init --recursive\n",
    "python3 scripts/start_e2eaiok_docker.py -b pytorch112 -w ${host0} ${host1} ${host2} ${host3} --proxy \"\"\n",
    "# Enter Docker\n",
    "sshpass -p docker ssh ${host0} -p 12347\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Workflow Prepare\n",
    "\n",
    "``` bash\n",
    "# prepare model codes\n",
    "bash workflow_prepare_rnnt.sh\n",
    "```\n",
    "\n",
    "a simple example of config file, one can refer to conf/e2eaiok_defaults_rnnt_example.conf for whole config file\n",
    "\n",
    "```yaml\n",
    "### GLOBAL SETTINGS ###\n",
    "observation_budget: 1\n",
    "save_path: /home/vmagent/app/e2eaiok/result/\n",
    "ppn: 2\n",
    "train_batch_size: 8\n",
    "eval_batch_size: 8\n",
    "iface: lo\n",
    "hosts:\n",
    "- localhost\n",
    "epochs: 2\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Data Prepare\n",
    "\n",
    "```bash\n",
    "# Download Dataset\n",
    "# Download and unzip dataset from https://www.openslr.org/12 to /home/vmagent/app/dataset/LibriSpeech\n",
    "\n",
    "cd /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch\n",
    "bash scripts/preprocess_librispeech.sh\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Train\n",
    "\n",
    "Edit config file to control SDA process\n",
    "> Note: Bellow training script is just for demonstration, and uses small sampled dataset and runs a small iterations. For actual performance result, please refer to [performance](#performance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-03-23 07:37:53,954 - E2EAIOK.SDA - INFO - ### Ready to submit current task  ###\n",
      "2023-03-23 07:37:53,956 - E2EAIOK.SDA - INFO - Model Advisor created\n",
      "2023-03-23 07:37:53,957 - E2EAIOK.SDA - INFO - model parameter initialized\n",
      "2023-03-23 07:37:53,958 - E2EAIOK.SDA - INFO - start to launch training\n",
      "2023-03-23 07:37:53,959 - sigopt - INFO - training launch command: /opt/intel/oneapi/intelpython/latest/bin/python -m intel_extension_for_pytorch.cpu.launch --distributed --nproc_per_node=2 --nnodes=1 --hostfile hosts /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py --output_dir /home/vmagent/app/e2eaiok/result/357dc3f8a3dfe894b3a3fcdd15fd1129f95f71cf887c8475679b1ff5b50674d8 --dist --dist_backend gloo --batch_size 8 --val_batch_size 8 --lr 0.007 --warmup_epochs 6 --beta1 0.9 --beta2 0.999 --max_duration 16.7 --target 0.25 --min_lr 1e-05 --lr_exp_gamma 0.939 --epochs 2 --epochs_this_job 0 --ema 0.999 --model_config modelzoo/rnnt/pytorch/configs/baseline_v3-1023sp.yaml --train_dataset_dir /home/vmagent/app/dataset/LibriSpeech/train --valid_dataset_dir /home/vmagent/app/dataset/LibriSpeech/valid --dali_device cpu --weight_decay 0.001 --grad_accumulation_steps 1 --weights_init_scale 0.5 --seed 2021 --train_manifests /home/vmagent/app/dataset/LibriSpeech/metadata/train-test.json --val_manifests /home/vmagent/app/dataset/LibriSpeech/metadata/dev-test.json --vectorized_sa --vectorized_sampler --multilayer_lstm --enable_prefetch --tokenized_transcript --dist_sampler --pre_sort_for_seq_split --jit_tensor_formation --max_symbol_per_sample 300 --data_cpu_threads 4 --min_seq_split_len 20 --log_frequency 1 --val_frequency 1 --prediction_frequency 1000000 --training_time_threshold 43200 --enc_n_hid 1024 --enc_pre_rnn_layers 2 --enc_stack_time_factor 8 --enc_post_rnn_layers 2 --pred_n_hid 512 --joint_n_hid 512 --fuse_relu_dropout --multi_tensor_ema --apex_transducer_loss fp16 --apex_transducer_joint pack             --buffer_pre_alloc --ema_update_type fp16 --apex_mlp --save_at_the_end \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'dataset_dir': '/home/vmagent/app/dataset/LibriSpeech', 'train_manifests': ['/home/vmagent/app/dataset/LibriSpeech/metadata/train-test.json'], 'val_manifests': ['/home/vmagent/app/dataset/LibriSpeech/metadata/dev-test.json']}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/intel/oneapi/intelpython/latest/lib/python3.9/runpy.py:127: RuntimeWarning: 'intel_extension_for_pytorch.cpu.launch' found in sys.modules after import of package 'intel_extension_for_pytorch.cpu', but prior to execution of 'intel_extension_for_pytorch.cpu.launch'; this may result in unpredictable behaviour\n",
      "  warn(RuntimeWarning(msg))\n",
      "2023-03-23 07:37:54,865 - __main__ - INFO - MASTER_ADDR=127.0.0.1\n",
      "2023-03-23 07:37:54,865 - __main__ - INFO - MASTER_PORT=29500\n",
      "2023-03-23 07:37:54,866 - __main__ - INFO - I_MPI_PIN_DOMAIN=[0x3fff0,0xfffc00000,]\n",
      "2023-03-23 07:37:54,866 - __main__ - WARNING - Neither TCMalloc nor JeMalloc is found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /root/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance\n",
      "2023-03-23 07:37:54,866 - __main__ - INFO - OMP_NUM_THREADS=14\n",
      "2023-03-23 07:37:54,866 - __main__ - INFO - Using Intel OpenMP\n",
      "2023-03-23 07:37:54,867 - __main__ - INFO - KMP_AFFINITY=granularity=fine,compact,1,0\n",
      "2023-03-23 07:37:54,867 - __main__ - INFO - KMP_BLOCKTIME=1\n",
      "2023-03-23 07:37:54,867 - __main__ - INFO - LD_PRELOAD=/opt/intel/oneapi/intelpython/latest/lib/libiomp5.so\n",
      "2023-03-23 07:37:54,867 - __main__ - INFO - CCL_WORKER_COUNT=4\n",
      "2023-03-23 07:37:54,867 - __main__ - INFO - CCL_WORKER_AFFINITY=0,1,2,3,18,19,20,21\n",
      "2023-03-23 07:37:54,867 - __main__ - INFO - mpiexec.hydra -l -np 2 -ppn 2 -genv I_MPI_PIN_DOMAIN=[0x3fff0,0xfffc00000,] -genv OMP_NUM_THREADS=14 /opt/intel/oneapi/intelpython/latest/bin/python -u /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py --output_dir /home/vmagent/app/e2eaiok/result/357dc3f8a3dfe894b3a3fcdd15fd1129f95f71cf887c8475679b1ff5b50674d8 --dist --dist_backend gloo --batch_size 8 --val_batch_size 8 --lr 0.007 --warmup_epochs 6 --beta1 0.9 --beta2 0.999 --max_duration 16.7 --target 0.25 --min_lr 1e-05 --lr_exp_gamma 0.939 --epochs 2 --epochs_this_job 0 --ema 0.999 --model_config modelzoo/rnnt/pytorch/configs/baseline_v3-1023sp.yaml --train_dataset_dir /home/vmagent/app/dataset/LibriSpeech/train --valid_dataset_dir /home/vmagent/app/dataset/LibriSpeech/valid --dali_device cpu --weight_decay 0.001 --grad_accumulation_steps 1 --weights_init_scale 0.5 --seed 2021 --train_manifests /home/vmagent/app/dataset/LibriSpeech/metadata/train-test.json --val_manifests /home/vmagent/app/dataset/LibriSpeech/metadata/dev-test.json --vectorized_sa --vectorized_sampler --multilayer_lstm --enable_prefetch --tokenized_transcript --dist_sampler --pre_sort_for_seq_split --jit_tensor_formation --max_symbol_per_sample 300 --data_cpu_threads 4 --min_seq_split_len 20 --log_frequency 1 --val_frequency 1 --prediction_frequency 1000000 --training_time_threshold 43200 --enc_n_hid 1024 --enc_pre_rnn_layers 2 --enc_stack_time_factor 8 --enc_post_rnn_layers 2 --pred_n_hid 512 --joint_n_hid 512 --fuse_relu_dropout --multi_tensor_ema --apex_transducer_loss fp16 --apex_transducer_joint pack --buffer_pre_alloc --ema_update_type fp16 --apex_mlp --save_at_the_end\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] No module named 'torch_ccl'\n",
      "[1] No module named 'torch_ccl'\n",
      "[0] world_size:2,rank:0\n",
      "[1] world_size:2,rank:1\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076109, \"event_type\": \"INTERVAL_START\", \"key\": \"init_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 357}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076192, \"event_type\": \"POINT_IN_TIME\", \"key\": \"seed\", \"value\": 2021, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 362}}\n",
      "[0] DLL 2023-03-23 07:37:56.194502 - PARAMETER | epochs :  2\n",
      "[0] DLL 2023-03-23 07:37:56.194556 - PARAMETER | warmup_epochs :  6\n",
      "[0] DLL 2023-03-23 07:37:56.194581 - PARAMETER | hold_epochs :  40\n",
      "[0] DLL 2023-03-23 07:37:56.194609 - PARAMETER | epochs_this_job :  0\n",
      "[0] DLL 2023-03-23 07:37:56.194632 - PARAMETER | cudnn_benchmark :  True\n",
      "[0] DLL 2023-03-23 07:37:56.194656 - PARAMETER | amp_level :  1\n",
      "[0] DLL 2023-03-23 07:37:56.194675 - PARAMETER | seed :  2021\n",
      "[0] DLL 2023-03-23 07:37:56.194701 - PARAMETER | local_rank :  0\n",
      "[0] DLL 2023-03-23 07:37:56.194720 - PARAMETER | target :  0.25\n",
      "[0] DLL 2023-03-23 07:37:56.194740 - PARAMETER | apex_transducer_loss :  fp16\n",
      "[0] DLL 2023-03-23 07:37:56.194766 - PARAMETER | fuse_relu_dropout :  True\n",
      "[0] DLL 2023-03-23 07:37:56.194789 - PARAMETER | weights_init_scale :  0.5\n",
      "[0] DLL 2023-03-23 07:37:56.194812 - PARAMETER | hidden_hidden_bias_scale : \n",
      "[0] DLL 2023-03-23 07:37:56.194831 - PARAMETER | batch_eval_mode : \n",
      "[0] DLL 2023-03-23 07:37:56.194857 - PARAMETER | cg_unroll_factor :  4\n",
      "[0] DLL 2023-03-23 07:37:56.194884 - PARAMETER | apex_transducer_joint :  pack\n",
      "[0] DLL 2023-03-23 07:37:56.194904 - PARAMETER | buffer_pre_alloc :  True\n",
      "[0] DLL 2023-03-23 07:37:56.194932 - PARAMETER | multilayer_lstm :  True\n",
      "[0] DLL 2023-03-23 07:37:56.194954 - PARAMETER | batch_split_factor :  1\n",
      "[0] DLL 2023-03-23 07:37:56.194972 - PARAMETER | apex_mlp :  True\n",
      "[0] DLL 2023-03-23 07:37:56.194991 - PARAMETER | num_cg :  0\n",
      "[0] DLL 2023-03-23 07:37:56.195018 - PARAMETER | min_seq_split_len :  20\n",
      "[0] DLL 2023-03-23 07:37:56.195037 - PARAMETER | pre_sort_for_seq_split :  True\n",
      "[0] DLL 2023-03-23 07:37:56.195057 - PARAMETER | dist :  True\n",
      "[0] DLL 2023-03-23 07:37:56.195075 - PARAMETER | dist_backend :  gloo\n",
      "[0] DLL 2023-03-23 07:37:56.195101 - PARAMETER | use_ipex :  False\n",
      "[0] DLL 2023-03-23 07:37:56.195127 - PARAMETER | batch_size :  8\n",
      "[0] DLL 2023-03-23 07:37:56.195148 - PARAMETER | val_batch_size :  8\n",
      "[0] DLL 2023-03-23 07:37:56.195195 - PARAMETER | lr :  0.007\n",
      "[0] DLL 2023-03-23 07:37:56.195216 - PARAMETER | min_lr :  1e-05\n",
      "[0] DLL 2023-03-23 07:37:56.195235 - PARAMETER | lr_exp_gamma :  0.939\n",
      "[0] DLL 2023-03-23 07:37:56.195253 - PARAMETER | weight_decay :  0.001\n",
      "[0] DLL 2023-03-23 07:37:56.195280 - PARAMETER | grad_accumulation_steps :  1\n",
      "[0] DLL 2023-03-23 07:37:56.195299 - PARAMETER | clip_norm :  1\n",
      "[0] DLL 2023-03-23 07:37:56.195321 - PARAMETER | beta1 :  0.9\n",
      "[0] DLL 2023-03-23 07:37:56.195339 - PARAMETER | beta2 :  0.999\n",
      "[0] DLL 2023-03-23 07:37:56.195367 - PARAMETER | ema :  0.999\n",
      "[0] DLL 2023-03-23 07:37:56.195389 - PARAMETER | multi_tensor_ema :  True\n",
      "[0] DLL 2023-03-23 07:37:56.195407 - PARAMETER | dist_lamb :  False\n",
      "[0] DLL 2023-03-23 07:37:56.195428 - PARAMETER | ema_update_type :  fp16\n",
      "[0] DLL 2023-03-23 07:37:56.195454 - PARAMETER | dwu_group_size :  8\n",
      "[0] DLL 2023-03-23 07:37:56.195479 - PARAMETER | dali_device :  cpu\n",
      "[0] DLL 2023-03-23 07:37:56.195496 - PARAMETER | resume :  False\n",
      "[0] DLL 2023-03-23 07:37:56.195514 - PARAMETER | ckpt : \n",
      "[0] DLL 2023-03-23 07:37:56.195542 - PARAMETER | save_at_the_end :  True\n",
      "[0] DLL 2023-03-23 07:37:56.195564 - PARAMETER | save_frequency : \n",
      "[0] DLL 2023-03-23 07:37:56.195580 - PARAMETER | keep_milestones :  []\n",
      "[0] DLL 2023-03-23 07:37:56.195601 - PARAMETER | save_best_from :  200\n",
      "[0] DLL 2023-03-23 07:37:56.195628 - PARAMETER | val_frequency :  1\n",
      "[0] DLL 2023-03-23 07:37:56.195650 - PARAMETER | log_frequency :  1\n",
      "[0] DLL 2023-03-23 07:37:56.195667 - PARAMETER | prediction_frequency :  1000000\n",
      "[0] DLL 2023-03-23 07:37:56.195687 - PARAMETER | model_config :  modelzoo/rnnt/pytorch/configs/baseline_v3-1023sp.yaml\n",
      "[0] DLL 2023-03-23 07:37:56.195714 - PARAMETER | num_buckets :  6\n",
      "[0] DLL 2023-03-23 07:37:56.195732 - PARAMETER | vectorized_sampler :  True\n",
      "[0] DLL 2023-03-23 07:37:56.195748 - PARAMETER | dist_sampler :  True\n",
      "[0] DLL 2023-03-23 07:37:56.195765 - PARAMETER | train_manifests :  ['/home/vmagent/app/dataset/LibriSpeech/metadata/train-test.json']\n",
      "[0] DLL 2023-03-23 07:37:56.195794 - PARAMETER | val_manifests :  ['/home/vmagent/app/dataset/LibriSpeech/metadata/dev-test.json']\n",
      "[0] DLL 2023-03-23 07:37:56.195817 - PARAMETER | max_duration :  16.7\n",
      "[0] DLL 2023-03-23 07:37:56.195839 - PARAMETER | max_txt_len :  125\n",
      "[0] DLL 2023-03-23 07:37:56.195864 - PARAMETER | max_eval_sample_duration :  32.7\n",
      "[0] DLL 2023-03-23 07:37:56.195892 - PARAMETER | train_dataset_dir :  /home/vmagent/app/dataset/LibriSpeech/train\n",
      "[0] DLL 2023-03-23 07:37:56.195914 - PARAMETER | valid_dataset_dir :  /home/vmagent/app/dataset/LibriSpeech/valid\n",
      "[0] DLL 2023-03-23 07:37:56.195933 - PARAMETER | output_dir :  /home/vmagent/app/e2eaiok/result/357dc3f8a3dfe894b3a3fcdd15fd1129f95f71cf887c8475679b1ff5b50674d8\n",
      "[0] DLL 2023-03-23 07:37:56.195960 - PARAMETER | log_file : \n",
      "[0] DLL 2023-03-23 07:37:56.195979 - PARAMETER | max_symbol_per_sample :  300\n",
      "[0] DLL 2023-03-23 07:37:56.195995 - PARAMETER | data_cpu_threads :  4\n",
      "[0] DLL 2023-03-23 07:37:56.196013 - PARAMETER | synthetic_audio_seq_len : \n",
      "[0] DLL 2023-03-23 07:37:56.196042 - PARAMETER | synthetic_text_seq_len : \n",
      "[0] DLL 2023-03-23 07:37:56.196066 - PARAMETER | enable_seq_len_stats :  False\n",
      "[0] DLL 2023-03-23 07:37:56.196085 - PARAMETER | vectorized_sa :  True\n",
      "[0] DLL 2023-03-23 07:37:56.196113 - PARAMETER | in_mem_file_list :  False\n",
      "[0] DLL 2023-03-23 07:37:56.196135 - PARAMETER | enable_prefetch :  True\n",
      "[0] DLL 2023-03-23 07:37:56.196169 - PARAMETER | tokenized_transcript :  True\n",
      "[0] DLL 2023-03-23 07:37:56.196187 - PARAMETER | jit_tensor_formation :  True\n",
      "[0] DLL 2023-03-23 07:37:56.196211 - PARAMETER | dali_dont_use_mmap :  False\n",
      "[0] DLL 2023-03-23 07:37:56.196233 - PARAMETER | training_time_threshold :  43200\n",
      "[0] DLL 2023-03-23 07:37:56.196251 - PARAMETER | enc_n_hid :  1024\n",
      "[0] DLL 2023-03-23 07:37:56.196271 - PARAMETER | enc_pre_rnn_layers :  2\n",
      "[0] DLL 2023-03-23 07:37:56.196298 - PARAMETER | enc_stack_time_factor :  8\n",
      "[0] DLL 2023-03-23 07:37:56.196322 - PARAMETER | enc_post_rnn_layers :  2\n",
      "[0] DLL 2023-03-23 07:37:56.196343 - PARAMETER | enc_dropout :  0.1\n",
      "[0] DLL 2023-03-23 07:37:56.196363 - PARAMETER | pred_n_hid :  512\n",
      "[0] DLL 2023-03-23 07:37:56.196389 - PARAMETER | pred_rnn_layers :  2\n",
      "[0] DLL 2023-03-23 07:37:56.196408 - PARAMETER | pred_dropout :  0.3\n",
      "[0] DLL 2023-03-23 07:37:56.196425 - PARAMETER | joint_n_hid :  512\n",
      "[0] DLL 2023-03-23 07:37:56.196443 - PARAMETER | joint_dropout :  0.3\n",
      "[0] DLL 2023-03-23 07:37:56.196525 - PARAMETER | rnn_type :  lstm\n",
      "[0] DLL 2023-03-23 07:37:56.196544 - PARAMETER | normalization :  False\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076213, \"event_type\": \"POINT_IN_TIME\", \"key\": \"gradient_accumulation_steps\", \"value\": 1, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 380}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076213, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_benchmark\", \"value\": \"rnnt\", \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 386}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076213, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_org\", \"value\": \"Intel\", \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 387}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076213, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_division\", \"value\": \"closed\", \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 388}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076213, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_status\", \"value\": \"onprem\", \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 389}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076213, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_platform\", \"value\": \"CPU\", \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 390}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076215, \"event_type\": \"POINT_IN_TIME\", \"key\": \"model_weights_initialization_scale\", \"value\": 0.5, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 397}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[0] 2023-03-23 07:37:56,098 - torch.distributed.distributed_c10d - INFO - Added key: store_based_barrier_key:1 to store for rank: 0\n",
      "[1] 2023-03-23 07:37:56,098 - torch.distributed.distributed_c10d - INFO - Added key: store_based_barrier_key:1 to store for rank: 1\n",
      "[0] 2023-03-23 07:37:56,098 - torch.distributed.distributed_c10d - INFO - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.\n",
      "[1] 2023-03-23 07:37:56,098 - torch.distributed.distributed_c10d - INFO - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076296, \"event_type\": \"POINT_IN_TIME\", \"key\": \"weights_initialization\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/rnn.py\", \"lineno\": 89, \"tensor\": \"pre_rnn\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076552, \"event_type\": \"POINT_IN_TIME\", \"key\": \"weights_initialization\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/rnn.py\", \"lineno\": 89, \"tensor\": \"post_rnn\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076556, \"event_type\": \"POINT_IN_TIME\", \"key\": \"weights_initialization\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py\", \"lineno\": 159, \"tensor\": \"pred_embed\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076582, \"event_type\": \"POINT_IN_TIME\", \"key\": \"weights_initialization\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/rnn.py\", \"lineno\": 89, \"tensor\": \"dec_rnn\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076584, \"event_type\": \"POINT_IN_TIME\", \"key\": \"weights_initialization\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py\", \"lineno\": 181, \"tensor\": \"joint_pred\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076588, \"event_type\": \"POINT_IN_TIME\", \"key\": \"weights_initialization\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py\", \"lineno\": 186, \"tensor\": \"joint_enc\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076591, \"event_type\": \"POINT_IN_TIME\", \"key\": \"weights_initialization\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py\", \"lineno\": 195, \"tensor\": \"joint_net\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076591, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_max_prediction_symbols\", \"value\": 300, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 418}}\n",
      "[0] Model size: 65.8M params\n",
      "[0] \n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557076616, \"event_type\": \"POINT_IN_TIME\", \"key\": \"model_eval_ema_factor\", \"value\": 0.999, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 432}}\n",
      "[0] Starting with LRs: 0.007\n",
      "[0] DistributedDataParallel(\n",
      "[0]   (module): RNNT(\n",
      "[0]     (encoder): ModuleDict(\n",
      "[0]       (pre_rnn): LSTM(\n",
      "[0]         (lstm): LSTM(256, 1024, num_layers=2, dropout=0.1)\n",
      "[0]         (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]       )\n",
      "[0]       (stack_time): StackTime()\n",
      "[0]       (post_rnn): LSTM(\n",
      "[0]         (lstm): LSTM(8192, 1024, num_layers=2, dropout=0.1)\n",
      "[0]         (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]       )\n",
      "[0]     )\n",
      "[0]     (prediction): ModuleDict(\n",
      "[0]       (embed): Embedding(1023, 512)\n",
      "[0]       (dec_rnn): LSTM(\n",
      "[0]         (lstm): LSTM(512, 512, num_layers=2, dropout=0.3)\n",
      "[0]         (dropout): Dropout(p=0.3, inplace=False)\n",
      "[0]       )\n",
      "[0]     )\n",
      "[0]     (joint_pred): Linear(in_features=512, out_features=512, bias=True)\n",
      "[0]     (joint_enc): Linear(in_features=1024, out_features=512, bias=True)\n",
      "[0]     (joint_net): Sequential(\n",
      "[0]       (0): FusedReluDropout()\n",
      "[0]       (1): Linear(in_features=512, out_features=1024, bias=True)\n",
      "[0]     )\n",
      "[0]   )\n",
      "[0] )\n",
      "[0] Setting up datasets...\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077226, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_train_max_duration\", \"value\": 16.7, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 472}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077226, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_speed_perturbaton_max\", \"value\": 1.15, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 474}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077226, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_speed_perturbaton_min\", \"value\": 0.85, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 476}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077226, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_spec_augment_freq_n\", \"value\": 2, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 478}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077226, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_spec_augment_freq_min\", \"value\": 0, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 480}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077226, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_spec_augment_freq_max\", \"value\": 20, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 482}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077227, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_spec_augment_time_n\", \"value\": 10, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 484}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077227, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_spec_augment_time_min\", \"value\": 0, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 486}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077227, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_spec_augment_time_max\", \"value\": 0.03, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 488}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077227, \"event_type\": \"POINT_IN_TIME\", \"key\": \"global_batch_size\", \"value\": 16, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 490}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077227, \"event_type\": \"INTERVAL_END\", \"key\": \"init_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 560}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077227, \"event_type\": \"INTERVAL_START\", \"key\": \"run_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 563}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077228, \"event_type\": \"POINT_IN_TIME\", \"key\": \"data_train_num_buckets\", \"value\": 6, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 569}}\n",
      "[0] Launching vectorized bucketing sampler\n",
      "[0] Launching simple sampler\n",
      "[0] Dataset read by DALI. Number of samples: 96\n",
      "[0] Initializing DALI with parameters:\n",
      "[0] \t           __class__ : <class 'common.data.dali.pipeline.DaliPipeline'>\n",
      "[0] \t          batch_size : 8\n",
      "[0] \t           device_id : None\n",
      "[0] \t        dither_coeff : 1e-05\n",
      "[0] \t       dont_use_mmap : False\n",
      "[0] \t           file_root : /home/vmagent/app/dataset/LibriSpeech/train\n",
      "[0] \t    in_mem_file_list : False\n",
      "[0] \t        max_duration : 16.7\n",
      "[0] \t           nfeatures : 80\n",
      "[0] \t                nfft : 512\n",
      "[0] \t         num_threads : 4[0] \n",
      "[0] \t       pipeline_type : train\n",
      "[0] \t            pre_sort : True\n",
      "[0] \t       preemph_coeff : 0.97\n",
      "[0] \tpreprocessing_device : cpu\n",
      "[0] \t      resample_range : [0.85, 1.15]\n",
      "[0] \t         sample_rate : 16000\n",
      "[0] \t             sampler : <common.data.dali.sampler.VectorizedBucketingSampler object at 0x7f39eec97bb0>\n",
      "[0] \t                seed : 2021\n",
      "[0] \t                self : <common.data.dali.pipeline.DaliPipeline object at 0x7f39eec97070>\n",
      "[0] \t   silence_threshold : -60\n",
      "[0] \t   synthetic_seq_len : None\n",
      "[0] \t         window_size : 0.02[0] \n",
      "[0] \t       window_stride : 0.01\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[0] /opt/intel/oneapi/intelpython/latest/lib/python3.9/site-packages/nvidia/dali/plugin/base_iterator.py:163: Warning: Please set `reader_name` and don't set last_batch_padded and size manually whenever possible. This may lead, in some situations, to missing some samples or returning duplicated ones. Check the Sharding section of the documentation for more details.\n",
      "[0]   _iterator_deprecation_warning()\n",
      "[1] /opt/intel/oneapi/intelpython/latest/lib/python3.9/site-packages/nvidia/dali/plugin/base_iterator.py:163: Warning: Please set `reader_name` and don't set last_batch_padded and size manually whenever possible. This may lead, in some situations, to missing some samples or returning duplicated ones. Check the Sharding section of the documentation for more details.\n",
      "[1]   _iterator_deprecation_warning()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] Dataset read by DALI. Number of samples: 73\n",
      "[0] Initializing DALI with parameters:\n",
      "[0] \t           __class__ : <class 'common.data.dali.pipeline.DaliPipeline'>\n",
      "[0] \t          batch_size : 8\n",
      "[0] \t           device_id : None[0] \n",
      "[0] \t        dither_coeff : 1e-05\n",
      "[0] \t       dont_use_mmap : False[0] \n",
      "[0] \t           file_root : /home/vmagent/app/dataset/LibriSpeech/valid\n",
      "[0] \t    in_mem_file_list : False\n",
      "[0] \t        max_duration : inf\n",
      "[0] \t           nfeatures : 80[0] \n",
      "[0] \t                nfft : 512\n",
      "[0] \t         num_threads : 4\n",
      "[0] \t       pipeline_type : val\n",
      "[0] \t            pre_sort : False\n",
      "[0] \t       preemph_coeff : 0.97\n",
      "[0] \tpreprocessing_device : cpu\n",
      "[0] \t      resample_range : None\n",
      "[0] \t         sample_rate : 16000\n",
      "[0] \t             sampler : <common.data.dali.sampler.SimpleSampler object at 0x7f39eec970d0>[0] \n",
      "[0] \t                seed : 2021\n",
      "[0] \t                self : <common.data.dali.pipeline.DaliPipeline object at 0x7f39eec97be0>\n",
      "[0] \t   silence_threshold : -60\n",
      "[0] \t   synthetic_seq_len : None\n",
      "[0] \t         window_size : 0.02[0] \n",
      "[0] \t       window_stride : 0.01\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077653, \"event_type\": \"POINT_IN_TIME\", \"key\": \"train_samples\", \"value\": 96, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 651}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077653, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_samples\", \"value\": 73, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 652}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077653, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_name\", \"value\": \"lamb\", \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 654}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077653, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_base_learning_rate\", \"value\": 0.007, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 655}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077653, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_lamb_epsilon\", \"value\": 1e-09, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 656}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077653, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_lamb_learning_rate_decay_poly_power\", \"value\": 0.939, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 657}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077653, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_learning_rate_warmup_epochs\", \"value\": 6, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 658}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_lamb_learning_rate_hold_epochs\", \"value\": 40, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 659}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_lamb_beta_1\", \"value\": 0.9, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 660}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_lamb_beta_2\", \"value\": 0.999, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 661}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_gradient_clip_norm\", \"value\": 1, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 662}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_learning_rate_alt_decay_func\", \"value\": true, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 663}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_learning_rate_alt_warmup_func\", \"value\": true, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 664}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_lamb_learning_rate_min\", \"value\": 1e-05, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 665}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077654, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_weight_decay\", \"value\": 0.001, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 666}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077655, \"event_type\": \"INTERVAL_START\", \"key\": \"block_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 697, \"first_epoch_num\": 1, \"epoch_count\": 1}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557077655, \"event_type\": \"INTERVAL_START\", \"key\": \"epoch_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 700, \"epoch_num\": 1}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[1] /opt/intel/oneapi/intelpython/latest/lib/python3.9/site-packages/torch/profiler/profiler.py:231: UserWarning: Profiler won't be using warmup, this can skew profiler results\n",
      "[1]   warn(\"Profiler won't be using warmup, this can skew profiler results\")\n",
      "[1] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/data/features.py:201: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[1]   x_lens = (x_lens.int() + stacking - 1) // stacking\n",
      "[1] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/data/dali/iterator.py:151: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[1]   pivot_len = (audio_shape_sorted[self.split_batch_size] + stack_factor-1) // stack_factor * stack_factor\n",
      "[1] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/helpers.py:276: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[1]   batch_offset = torch.cumsum(g_len * ((feat_lens+self.enc_stack_time_factor-1)//self.enc_stack_time_factor), dim=0)\n",
      "[0] /opt/intel/oneapi/intelpython/latest/lib/python3.9/site-packages/torch/profiler/profiler.py:231: UserWarning: Profiler won't be using warmup, this can skew profiler results\n",
      "[0]   warn(\"Profiler won't be using warmup, this can skew profiler results\")\n",
      "[0] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/data/features.py:201: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[0]   x_lens = (x_lens.int() + stacking - 1) // stacking\n",
      "[0] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/data/dali/iterator.py:151: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[0]   pivot_len = (audio_shape_sorted[self.split_batch_size] + stack_factor-1) // stack_factor * stack_factor\n",
      "[0] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/common/helpers.py:276: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[0]   batch_offset = torch.cumsum(g_len * ((feat_lens+self.enc_stack_time_factor-1)//self.enc_stack_time_factor), dim=0)\n",
      "[0] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py:64: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[0]   x_lens = (x_lens.int() + self.factor - 1) // self.factor\n",
      "[1] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py:64: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[1]   x_lens = (x_lens.int() + self.factor - 1) // self.factor\n",
      "[0] [W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())\n",
      "[1] [W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())\n",
      "[1] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n",
      "[0] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] DLL 2023-03-23 07:38:02.765200 - epoch    1 | iter    1/6 | loss  958.22 | utts/s     3 | took  5.08 s | lrate 3.78e-04\n",
      "[0] DLL 2023-03-23 07:38:06.465558 - epoch    1 | iter    2/6 | loss  906.90 | utts/s     4 | took  3.70 s | lrate 5.68e-04\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[0] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n",
      "[1] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] DLL 2023-03-23 07:38:09.720707 - epoch    1 | iter    3/6 | loss  801.17 | utts/s     5 | took  3.26 s | lrate 7.57e-04\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[0] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n",
      "[1] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n",
      "[1] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n",
      "[0] [W kineto_shim.cpp:337] Profiler is not initialized: skipping step() invocation\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] DLL 2023-03-23 07:38:12.573296 - epoch    1 | iter    4/6 | loss  535.04 | utts/s     6 | took  2.85 s | lrate 9.46e-04\n",
      "[0] DLL 2023-03-23 07:38:16.913737 - epoch    1 | iter    5/6 | loss 1013.65 | utts/s     4 | took  4.34 s | lrate 1.14e-03\n",
      "[0] DLL 2023-03-23 07:38:22.809516 - epoch    1 | iter    6/6 | loss  896.75 | utts/s     3 | took  5.90 s | lrate 1.32e-03\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557102810, \"event_type\": \"INTERVAL_END\", \"key\": \"epoch_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 786, \"epoch_num\": 1}}\n",
      "[0] DLL 2023-03-23 07:38:22.810912 - epoch    1 | avg train utts/s     4 | took 25.16 s\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557102810, \"event_type\": \"POINT_IN_TIME\", \"key\": \"throughput\", \"value\": 3.816295986713841, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 793}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557102811, \"event_type\": \"INTERVAL_START\", \"key\": \"eval_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 239, \"epoch_num\": 1}}\n",
      "[0] val evaluation:          0/5         \r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[1] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py:53: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[1]   x_lens = (x_lens.int() + self.factor - 1) // self.factor\n",
      "[0] /home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/rnnt/model.py:53: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[0]   x_lens = (x_lens.int() + self.factor - 1) // self.factor\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557114384, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_accuracy\", \"value\": 20.484347826086957, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 260, \"epoch_num\": 1}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557114384, \"event_type\": \"INTERVAL_END\", \"key\": \"eval_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 261, \"epoch_num\": 1}}\n",
      "[0] DLL 2023-03-23 07:38:34.385417 - epoch    1 |   dev ema wer 2048.43 | took 11.57 s\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557114386, \"event_type\": \"INTERVAL_END\", \"key\": \"block_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 811, \"first_epoch_num\": 1}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557114386, \"event_type\": \"INTERVAL_START\", \"key\": \"block_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 697, \"first_epoch_num\": 2, \"epoch_count\": 1}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557114386, \"event_type\": \"INTERVAL_START\", \"key\": \"epoch_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 700, \"epoch_num\": 2}}\n",
      "[0] DLL 2023-03-23 07:38:41.071182 - epoch    2 | iter    1/6 | loss  962.37 | utts/s     2 | took  6.59 s | lrate 1.51e-03\n",
      "[0] DLL 2023-03-23 07:40:28.400494 - epoch    2 | iter    2/6 | loss  512.79 | utts/s     0 | took 107.33 s | lrate 1.70e-03\n",
      "[0] DLL 2023-03-23 07:40:31.648097 - epoch    2 | iter    3/6 | loss  841.06 | utts/s     5 | took  3.25 s | lrate 1.89e-03\n",
      "[0] DLL 2023-03-23 07:40:35.097460 - epoch    2 | iter    4/6 | loss  894.31 | utts/s     5 | took  3.45 s | lrate 2.08e-03\n",
      "[0] DLL 2023-03-23 07:40:39.074445 - epoch    2 | iter    5/6 | loss 1000.28 | utts/s     4 | took  3.98 s | lrate 2.27e-03\n",
      "[0] DLL 2023-03-23 07:40:42.397943 - epoch    2 | iter    6/6 | loss  926.91 | utts/s     5 | took  3.32 s | lrate 2.46e-03\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557242398, \"event_type\": \"INTERVAL_END\", \"key\": \"epoch_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 786, \"epoch_num\": 2}}\n",
      "[0] DLL 2023-03-23 07:40:42.398739 - epoch    2 | avg train utts/s     1 | took 128.01 s\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557242398, \"event_type\": \"POINT_IN_TIME\", \"key\": \"throughput\", \"value\": 0.7499322663702139, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 793}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557242398, \"event_type\": \"INTERVAL_START\", \"key\": \"eval_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 239, \"epoch_num\": 2}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557251979, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_accuracy\", \"value\": 20.484347826086957, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 260, \"epoch_num\": 2}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557251979, \"event_type\": \"INTERVAL_END\", \"key\": \"eval_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 261, \"epoch_num\": 2}}\n",
      "[0] DLL 2023-03-23 07:40:51.980234 - epoch    2 |   dev ema wer 2048.43 | took  9.58 s\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557251980, \"event_type\": \"INTERVAL_END\", \"key\": \"block_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 811, \"first_epoch_num\": 2}}\n",
      "[0] -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n",
      "[0]                                                    Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  \n",
      "[0] -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n",
      "[0]                                                aten::mm        19.85%        4.794s        19.86%        4.797s     775.458us          6186  \n",
      "[0]                                             aten::addmm        16.37%        3.956s        19.82%        4.787s      62.369us         76753  \n",
      "[0]                                               aten::add        15.16%        3.663s        15.17%        3.664s     152.716us         23990  \n",
      "[0]                                           ProfilerStep*        14.28%        3.450s       100.00%       24.157s       12.078s             2  \n",
      "[0]                                             aten::copy_         3.89%     938.798ms         3.89%     938.798ms       9.300us        100946  \n",
      "[0]                                               aten::mul         3.39%     819.327ms         3.48%     839.973ms       7.668us        109536  \n",
      "[0]                                          aten::sigmoid_         2.77%     667.949ms         2.77%     667.949ms       7.352us         90852  \n",
      "[0]                                              aten::lstm         2.20%     530.277ms        33.49%        8.091s     695.661us         11630  \n",
      "[0]                                              aten::add_         2.07%     499.895ms         2.07%     499.895ms       8.217us         60838  \n",
      "[0]                                       fallback_function         1.74%     419.419ms         1.91%     460.248ms     230.124ms             2  \n",
      "[0]                                               aten::cat         1.52%     366.592ms         1.81%     438.358ms       8.835us         49616  \n",
      "[0]                                        aten::as_strided         0.84%     202.998ms         0.84%     202.998ms       0.399us        508563  \n",
      "[0]                                             aten::tanh_         0.83%     200.664ms         0.83%     200.664ms       6.626us         30284  \n",
      "[0]                                            aten::linear         0.78%     189.508ms        22.63%        5.467s      71.202us         76775  \n",
      "[0]                                                 aten::t         0.74%     178.310ms         1.30%     313.088ms       3.513us         89119  \n",
      "[0]                                              aten::view         0.73%     177.334ms         0.73%     177.334ms       1.150us        154182  \n",
      "[0]                                             aten::slice         0.72%     174.914ms         1.00%     240.542ms       1.335us        180218  \n",
      "[0]                                              aten::tanh         0.67%     161.211ms         0.67%     161.211ms       5.323us         30284  \n",
      "[0]                                         aten::transpose         0.59%     142.462ms         0.84%     202.870ms       1.805us        112396  \n",
      "[0]                                              aten::norm         0.51%     123.066ms         0.51%     123.254ms     655.606us           188  \n",
      "[0]                                            aten::select         0.48%     117.050ms         0.65%     157.108ms       1.361us        115434  \n",
      "[0]                                               aten::max         0.47%     113.605ms         0.64%     155.568ms      13.389us         11619  \n",
      "[0]                                             aten::empty         0.47%     113.253ms         0.47%     113.253ms       1.484us         76325  \n",
      "[0]                                      aten::unsafe_split         0.46%     111.597ms         1.33%     322.439ms      10.647us         30284  \n",
      "[0]                                            aten::unbind         0.44%     105.920ms         0.81%     196.554ms       4.222us         46558  \n",
      "[0]                                             aten::stack         0.43%     103.032ms         1.93%     465.877ms      10.008us         46550  \n",
      "[0]                                               aten::sum         0.42%     102.474ms         0.55%     132.781ms      42.888us          3096  \n",
      "[0]                                             aten::fill_         0.41%      98.251ms         0.41%      98.251ms       6.615us         14852  \n",
      "[0]                                              aten::mul_         0.39%      95.302ms         0.42%     101.011ms     531.637us           190  \n",
      "[0]                                            aten::narrow         0.36%      86.010ms         0.98%     236.685ms       1.782us        132830  \n",
      "[0]                                               aten::div         0.34%      82.127ms         0.35%      84.008ms     313.463us           268  \n",
      "[0]                                            aten::expand         0.27%      65.789ms         0.39%      93.134ms       1.213us         76774  \n",
      "[0]     autograd::engine::evaluate_function: AddmmBackward0         0.26%      63.350ms        20.16%        4.871s       1.583ms          3078  \n",
      "[0]                        aten::_log_softmax_backward_data         0.26%      62.837ms         0.26%      62.837ms      31.419ms             2  \n",
      "[0]                                  aten::sigmoid_backward         0.25%      60.725ms         0.25%      60.725ms       6.606us          9192  \n",
      "[0]                                   aten::constant_pad_nd         0.23%      56.473ms         0.84%     204.063ms      17.522us         11646  \n",
      "[0]                                      aten::index_select         0.23%      54.780ms         0.27%      65.486ms       5.657us         11576  \n",
      "[0]                                         aten::embedding         0.21%      49.961ms         0.60%     145.714ms      12.588us         11576  \n",
      "[0]       autograd::engine::evaluate_function: MulBackward0         0.20%      49.514ms         1.36%     329.026ms      35.717us          9212  \n",
      "[0]                                         aten::clamp_min         0.20%      47.144ms         0.20%      47.144ms       4.060us         11612  \n",
      "[0]                                          AddmmBackward0         0.19%      45.899ms        19.06%        4.604s       1.496ms          3078  \n",
      "[0]                                     aten::tanh_backward         0.19%      45.733ms         0.19%      45.733ms       7.463us          6128  \n",
      "[0]                                            MulBackward0         0.19%      45.327ms         1.16%     279.512ms      30.342us          9212  \n",
      "[0] autograd::engine::evaluate_function: torch::autograd...         0.16%      38.066ms         0.32%      77.696ms       1.253ms            62  \n",
      "[0]                                         aten::unsqueeze         0.16%      37.554ms         0.19%      46.795ms       1.973us         23723  \n",
      "[0]                                          aten::_to_copy         0.15%      35.827ms         0.50%     119.801ms       9.757us         12278  \n",
      "[0]         autograd::engine::evaluate_function: TBackward0         0.13%      31.386ms        14.47%        3.495s       1.131ms          3090  \n",
      "[0]                                      aten::_log_softmax         0.13%      30.411ms         0.13%      30.411ms      15.206ms             2  \n",
      "[0]                                      aten::unsafe_chunk         0.12%      28.321ms         1.45%     350.760ms      11.582us         30284  \n",
      "[0]       autograd::engine::evaluate_function: AddBackward0         0.12%      28.124ms         0.17%      40.828ms       6.647us          6142  \n",
      "[0]                                              aten::sqrt         0.11%      26.993ms         0.11%      26.993ms     435.371us            62  \n",
      "[0]                                              aten::relu         0.10%      24.207ms         0.30%      71.351ms       6.145us         11612  \n",
      "[0]      autograd::engine::evaluate_function: TanhBackward0         0.09%      22.711ms         0.49%     117.280ms      19.138us          6128  \n",
      "[0]                                        aten::bernoulli_         0.09%      22.315ms         0.09%      22.380ms       1.017ms            22  \n",
      "[0]                                           TanhBackward0         0.09%      21.905ms         0.28%      67.638ms      11.038us          6128  \n",
      "[0]                                                aten::ge         0.09%      21.064ms         0.09%      21.087ms       1.622ms            13  \n",
      "[0]                                     aten::empty_strided         0.08%      20.263ms         0.08%      20.263ms       1.647us         12302  \n",
      "[0]                                        SigmoidBackward0         0.08%      20.098ms         0.33%      80.823ms       8.793us          9192  \n",
      "[0]                                                aten::to         0.08%      20.002ms         0.58%     139.803ms       5.772us         24220  \n",
      "[0] autograd::engine::evaluate_function: SigmoidBackward...         0.08%      18.575ms         0.41%      99.398ms      10.814us          9192  \n",
      "[0]                                      aten::resolve_conj         0.07%      17.528ms         0.07%      17.528ms       0.107us        164490  \n",
      "[0]                                        aten::unsqueeze_         0.07%      17.241ms         0.10%      22.980ms       0.990us         23222  \n",
      "[0]                                               aten::pad         0.07%      16.944ms         0.88%     212.728ms      18.279us         11638  \n",
      "[0]                                           aten::reshape         0.06%      15.168ms         0.14%      34.134ms       2.928us         11659  \n",
      "[0]                                          aten::squeeze_         0.06%      15.104ms         0.08%      18.842ms       0.811us         23220  \n",
      "[0]                                    aten::_reshape_alias         0.06%      13.832ms         0.06%      13.832ms       1.187us         11656  \n",
      "[0]                                          aten::addcmul_         0.06%      13.388ms         0.06%      13.388ms     215.935us            62  \n",
      "[0]                                Optimizer.step#Lamb.step         0.05%      13.183ms         1.27%     307.516ms     153.758ms             2  \n",
      "[0]                                                aten::gt         0.05%      12.540ms         0.05%      12.589ms       6.295ms             2  \n",
      "[0] torch.distributed.ddp.reducer::search_unused_paramet...         0.05%      12.511ms         0.05%      12.511ms       6.255ms             2  \n",
      "[0]                                                   _RNNT         0.05%      12.167ms         0.16%      37.920ms      18.960ms             2  \n",
      "[0]                                              aten::item         0.05%      11.957ms         0.08%      20.166ms       1.705us         11831  \n",
      "[0] autograd::engine::evaluate_function: UnsafeSplitBack...         0.05%      11.578ms         0.50%     121.183ms      39.551us          3064  \n",
      "[0]                                              aten::div_         0.05%      11.228ms         0.05%      11.582ms     445.462us            26  \n",
      "[0]                                       aten::as_strided_         0.04%       9.477ms         0.04%       9.477ms       0.204us         46442  \n",
      "[0]                                    UnsafeSplitBackward0         0.03%       8.422ms         0.45%     109.605ms      35.772us          3064  \n",
      "[0]                               aten::_local_scalar_dense         0.03%       8.209ms         0.03%       8.209ms       0.694us         11831  \n",
      "[0]                         DistributedDataParallel.forward         0.03%       7.098ms         5.72%        1.383s     691.408ms             2  \n",
      "[0]                                           aten::dropout         0.02%       5.552ms         0.06%      13.909ms       1.195us         11640  \n",
      "[0]                               aten::cudnn_is_acceptable         0.02%       5.190ms         0.02%       5.190ms       0.446us         11630  \n",
      "[0]                                      aten::masked_fill_         0.02%       4.740ms         0.02%       4.740ms     526.667us             9  \n",
      "[0]                                              TBackward0         0.02%       4.675ms         0.07%      17.124ms       5.542us          3090  \n",
      "[0]                                            AddBackward0         0.01%       3.252ms         0.01%       3.252ms       0.529us          6142  \n",
      "[0]                                               aten::any         0.01%       3.196ms         0.01%       3.222ms     537.000us             6  \n",
      "[0] autograd::engine::evaluate_function: UnbindBackward0...         0.01%       2.598ms         0.08%      18.518ms     661.357us            28  \n",
      "[0]                                          StackBackward0         0.01%       2.455ms         0.04%       9.718ms     347.071us            28  \n",
      "[0]     autograd::engine::evaluate_function: StackBackward0         0.01%       2.233ms         0.05%      12.100ms     432.143us            28  \n",
      "[0] autograd::engine::evaluate_function: UnsafeViewBackw...         0.01%       1.438ms         0.01%       2.519ms     179.929us            14  \n",
      "[0]                                           <backward op>         0.01%       1.393ms         2.06%     497.258ms     248.629ms             2  \n",
      "[0] autograd::engine::evaluate_function: LogSoftmaxBackw...         0.01%       1.327ms         0.27%      64.197ms      32.099ms             2  \n",
      "[0] autograd::engine::evaluate_function: torch::jit::(an...         0.00%       1.002ms         2.06%     498.324ms     249.162ms             2  \n",
      "[0]                           aten::native_dropout_backward         0.00%     976.000us         0.15%      35.617ms      17.808ms             2  \n",
      "[0]                                             aten::zero_         0.00%     970.000us         0.18%      43.540ms     187.672us           232  \n",
      "[0]                                                _DDPSink         0.00%     670.000us         0.10%      23.044ms      11.522ms             2  \n",
      "[0]                                             aten::index         0.00%     536.000us         0.00%     590.000us      39.333us            15  \n",
      "[0]                                            aten::matmul         0.00%     501.000us         0.64%     155.562ms       7.071ms            22  \n",
      "[0]                                             aten::zeros         0.00%     492.000us         0.01%       1.707ms      10.161us           168  \n",
      "[0]                          aten::embedding_dense_backward         0.00%     450.000us         0.00%     655.000us     327.500us             2  \n",
      "[0]                                                aten::le         0.00%     431.000us         0.00%     431.000us     107.750us             4  \n",
      "[0]                                         UnbindBackward0         0.00%     428.000us         0.07%      15.920ms     568.571us            28  \n",
      "[0] -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  \n",
      "[0] Self CPU time total: 24.157s\n",
      "[0] \n",
      "[0] DLL 2023-03-23 07:42:06.064895 -  | avg train utts/s     1\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557326064, \"event_type\": \"INTERVAL_END\", \"key\": \"run_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 832, \"status\": \"aborted\"}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557326065, \"event_type\": \"INTERVAL_START\", \"key\": \"eval_start\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 239, \"epoch_num\": 2}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557337509, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_accuracy\", \"value\": 20.484347826086957, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 260, \"epoch_num\": 2}}\n",
      "[0] :::MLLOG {\"namespace\": \"\", \"time_ms\": 1679557337510, \"event_type\": \"INTERVAL_END\", \"key\": \"eval_stop\", \"value\": null, \"metadata\": {\"file\": \"/home/vmagent/app/e2eaiok/modelzoo/rnnt/pytorch/train.py\", \"lineno\": 261, \"epoch_num\": 2}}\n",
      "[0] DLL 2023-03-23 07:42:17.510989 - epoch    2 |   dev ema wer 2048.43 | took 11.45 s\n",
      "[0] Saving /home/vmagent/app/e2eaiok/result/357dc3f8a3dfe894b3a3fcdd15fd1129f95f71cf887c8475679b1ff5b50674d8/RNN-T_epoch2_checkpoint.pt...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-03-23 07:42:28,613 - sigopt - INFO - Training completed based in sigopt suggestion, took 274.6536593437195 secs\n",
      "2023-03-23 07:42:28,616 - E2EAIOK.SDA - INFO - training script completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('/home/vmagent/app/e2eaiok/result/357dc3f8a3dfe894b3a3fcdd15fd1129f95f71cf887c8475679b1ff5b50674d8',\n",
       " [{'name': 'WER', 'value': 20.484347826086957},\n",
       "  {'name': 'training_time', 'value': 274.6536593437195}])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from e2eAIOK.SDA.SDA import SDA\n",
    "import yaml\n",
    "\n",
    "# create SDA settings\n",
    "settings = {}\n",
    "settings[\"data_path\"] = \"/home/vmagent/app/dataset/LibriSpeech/\"\n",
    "settings[\"enable_sigopt\"] = False\n",
    "settings[\"python_path\"] = \"/opt/intel/oneapi/intelpython/latest/bin/python\"\n",
    "settings[\"train_path\"] = \"e2eaiok/modelzoo/rnnt/pytorch/train.py\"\n",
    "settings[\"model_config\"] = \"e2eaiok/modelzoo/rnnt/pytorch/configs/baseline_v3-1023sp.yaml\"\n",
    "# load RNN-T settings\n",
    "with open(\"e2eaiok/tests/cicd/conf/e2eaiok_defaults_rnnt_example.conf\") as f:\n",
    "    conf = yaml.load(f, Loader=yaml.FullLoader)\n",
    "settings.update(conf)\n",
    "\n",
    "sda = SDA(model=\"rnnt\", settings=settings)\n",
    "sda.launch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}