demo/builtin/resnet/RESNET_DEMO.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/main/demo/builtin/resnet/RESNET_DEMO.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RESNET Demo\n",
    "ResNet, short for Residual Network is a specific type of neural network that was introduced in paper “Deep Residual Learning for Image Recognition”.\n",
    "\n",
    "* original source\n",
    "    * Source repo: https://github.com/mlcommons/training_results_v1.0/tree/master/Intel/benchmarks/resnet/2-nodes-16s-8376H-tensorflow"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Content\n",
    "* [Overview](#Overview)\n",
    "    * [Model Architecture](#Model-Architecture)\n",
    "    * [Optimizations](#Optimizations)\n",
    "    * [Performance](#Performance)\n",
    "* [Getting Started](#Getting-Started)\n",
    "    * [1. Environment Setup](#1.-Environment-Setup)\n",
    "    * [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "    * [3. Data Prepare](#3.-Data-Prepare)\n",
    "    * [4. Train](#4.-Train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "## Model Architecture\n",
    "\n",
    "### Image Classification\n",
    "* Classify different images into different categories to achieve the smallest classification error. Image classification is a supervised learning problem: define a set of target classes (objects to identify in images), and train a model to recognize them using labeled example photos.\n",
    "<div><img src=\"./img/image_classification.png\" alt=\"image_classification.png\" width=\"600\"></div>\n",
    "\n",
    "### VGG Neural Networks\n",
    "* VGG stands for Visual Geometry Group; it is a standard deep Convolutional Neural Network (CNN) architecture with multiple layers. The “deep” refers to the number of layers with VGG-16 or VGG-19 consisting of 16 and 19 convolutional layers. \n",
    "* The VGG architecture is the basis of ground-breaking object recognition models. Developed as a deep neural network, the VGGNet also surpasses baselines on many tasks and datasets beyond ImageNet. Moreover, it is now still one of the most popular image recognition architectures.\n",
    "\n",
    "<div><img src=\"./img/resnet.png\" alt=\"resnet.png\"></div>\n",
    "### Deep Residual Learning for Image Recognition(ResNet)\n",
    "Deep residual networks like the popular ResNet-50 model is a convolutional neural network (CNN) that is 50 layers deep. A Residual Neural Network (ResNet) is an Artificial Neural Network (ANN) of a kind that stacks residual(shown as below image) blocks on top of each other to form a network.\n",
    "\n",
    "A residual network is a stack of many residual blocks. Regular design, like VGG: each residual block has two 3x3 conv. \n",
    "The Network is divided into stages: the first block of each stage halves the resolution (with stride-2 conv) and doubles the number of channels\n",
    "\n",
    "<div><img src=\"./img/residual.png\" alt=\"residual.png\"></div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizations\n",
    "## HPO with SDA (Smart Democratization Advisor)\n",
    "\n",
    "\n",
    "\n",
    "SDA config\n",
    "\n",
    "```\n",
    "Parameters for SDA auto optimization:\n",
    "- resnet_size:[18, 34, 50] #number of residual blocks \n",
    "- num_filters:[2,4,8,16,32,64] #number filters of convolution layer\n",
    "- kernel_size:[1,3,5,7,9] #number kernel size of convolution layer\n",
    "- label_smoothing:0.05~0.5 #parameter for optimizer\n",
    "- momentum:0.6~0.99 #parameter for optimizer\n",
    "- weight_decay:1e-5~5e-5 #weight decay coefficiant for l2 regularization\n",
    "- base_lr:0.01~0.2 #base learning rate for optimizer\n",
    "metrics:\n",
    "- name: accuracy\n",
    "  objective: maximize\n",
    "- name: training_time\n",
    "  strategy: optimize\n",
    "  objective: minimize\n",
    "```\n",
    "\n",
    "\n",
    "request suggestions from SDA\n",
    "\n",
    "```python\n",
    "suggestion = self.conn.experiments(self.experiment.id).suggestions().create()\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model related optimization\n",
    "### Add attention block\n",
    "* The attention mechanism is generated by imitating human attention. When human are looking at an image, some parts of the image  will attract human attention, while others may be ignored. This is equivalent to adding a visual weight to the entire image. Attention mechanisms can be divided into channel attention and spatial attention. In simple terms, the attention mechanism is to multiply each channel or the entire spatial feature map by a weight to represent their importance.\n",
    "* Add attention block to 34 layers ResNet model. The implemented attention block layer to ResNet model is a middle layer between two residual blocks, so that the output of the middle layer becomes outputs = (1 + masks) * maps.\n",
    "### Disable BF16\n",
    "* MLCommons training v1.0 code optimized on CPX and with BF16 enabled by default, while ICX doesn't support BF16\n",
    "* Disable BF16 for 50 layers ResNet model on ICX \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance\n",
    "\n",
    "<img src=\"./img/resnet_perf.png\" width=\"600\"/>\n",
    "\n",
    "* Distributed training with HW scaling delivered 3.84x speedup from 1 node to 4 nodes\n",
    "* HPO (hyper parameter optimization) with SDA (a component of AIOK, smart democratization advisor) delivered 1.21x speedup, and 4.63x speedup over baseline\n",
    "* Input size tuning delivered 1.98x speedup, and 8.77x speedup over baseline\n",
    "    * reduce input size from 224 to 112 after 8 epoch\n",
    "    * within 5% acc gap (0.722 vs. 0.759)\n",
    "* baseline converged and stopped at 37 epoch, optimized model converged and stopped at 31 epoch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "* [1. Environment Setup](#1.-Environment-Setup)\n",
    "* [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "* [3. Data Prepare](#3.-Data-Prepare)\n",
    "* [4. Train](#4.-Train)\n",
    "\n",
    "Notes: This demo cannot run directly on colab, since it require you to download dataset manually, and store all files according to the specified directory hierarchy. In order to run this demo, please follow `Environment Setup`, `Workflow Prepare` and `Data Prepare` section for pre-requirements."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Environment Setup\n",
    "### (Option 1) Use Pip Install\n",
    "pre-work: move e2eAIOK source code to /home/vmagent/app/e2eaiok."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install sigopt future pydot dill pyyaml\n",
    "pip install --no-cache-dir intel-tensorflow==2.10\n",
    "HOROVOD_WITHOUT_MPI=1 HOROVOD_CPU_OPERATIONS=CCL \\\n",
    "    HOROVOD_WITHOUT_MXNET=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITH_TENSORFLOW=1 \\\n",
    "    pip install --no-cache-dir horovod\n",
    "pip install --no-cache-dir --no-deps tensorflow-transform==0.24.1 tensorflow-metadata==0.14.0\n",
    "pip install \"git+https://github.com/mlperf/logging.git@1.0.0\"\n",
    "pip install e2eAIOK-sda --pre --no-deps --ignore-installed"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Option 2) Use Docker\n",
    "\n",
    "Step1. prepare code\n",
    "``` bash\n",
    "git clone https://github.com/intel/e2eAIOK.git\n",
    "cd e2eAIOK\n",
    "git submodule update --init --recursive\n",
    "```\n",
    "\n",
    "Step2. build docker image\n",
    "```bash\n",
    "python3 scripts/start_e2eaiok_docker.py -b tensorflow -w ${host0} ${host1} ${host2} ${host3} --proxy \"\"\n",
    "```\n",
    "\n",
    "Step3. run docker and start conda env\n",
    "```bash\n",
    "sshpass -p docker ssh ${host0} -p 12344\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Workflow Prepare\n",
    "\n",
    "\n",
    "\n",
    "``` bash\n",
    "# prepare model codes\n",
    "bash workflow_prepare_resnet.sh\n",
    "```\n",
    "\n",
    "a simple example of config file, one can refer to conf/e2eaiok_defaults_resnet_example.conf for whole config file\n",
    "\n",
    "```yaml\n",
    "observation_budget: 1\n",
    "interface: env1\n",
    "save_path: ./result/\n",
    "ppn: 2\n",
    "ccl_worker_num: 4\n",
    "batch_size: 204\n",
    "num_epochs: 1\n",
    "cores: None\n",
    "training_time_threshold: 5000\n",
    "metric: accuracy\n",
    "metric_objective: maximize\n",
    "metric_threshold: 0.11656\n",
    "train_dataset_path: /home/vmagent/app/dataset/resnet/\n",
    "use_synthetic_data: True\n",
    "project: sda\n",
    "experiment: resnet\n",
    "hosts:\n",
    "- localhost\n",
    "```\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Data Prepare\n",
    "```\n",
    "The TF ResNet50-v1.5 model is trained with ImageNet 1K, a popular image classification dataset from ILSVRC challenge. The dataset can be downloaded from:\n",
    "\n",
    "http://image-net.org/download-images\n",
    "\n",
    "More dataset requirements can be found at:\n",
    "\n",
    "https://github.com/mlperf/training/tree/master/image_classification#3-datasetenvironment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notes: ResNet training is based on ImageNet1K dataset, we evaluated Top-1 Accuracy with stock model (based on MLPerf submission) at ImageNet1K dataset, and final Top-1 Accuracy is 0.757\n",
    "\n",
    "public reference paper: https://arxiv.org/abs/1512.03385"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Train\n",
    "Noted: Below performance result is using sample dataset and small iterations to demonstrate its function. The actual performance result please refers to the [performance section](#performance).\n",
    "\n",
    "Edit config file to control SDA process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===============================================\n",
      "***    Best Trained Model    ***\n",
      "===============================================\n",
      "  Model Type: resnet\n",
      "  Model Saved Path: /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa\n",
      "  Sigopt Experiment id is None\n",
      "  === Result Metrics ===\n",
      "    accuracy: 1.0\n",
      "    training_time: 0.010071277618408203\n",
      "===============================================\n",
      "2022-10-31 17:54:48,376 - E2EAIOK - INFO - Above info is history record of this model\n",
      "2022-10-31 17:54:48,376 - E2EAIOK.SDA - INFO - ### Ready to submit current task  ###\n",
      "2022-10-31 17:54:48,376 - E2EAIOK.SDA - INFO - Model Advisor created\n",
      "2022-10-31 17:54:48,376 - E2EAIOK.SDA - INFO - model parameter initialized\n",
      "2022-10-31 17:54:48,376 - E2EAIOK.SDA - INFO - start to launch training\n",
      "2022-10-31 17:54:48,376 - sigopt - INFO - training launch command: /opt/intel/oneapi/intelpython/latest/envs/tensorflow/bin//horovodrun -n 2 -H localhost:2 HOROVOD_CPU_OPERATIONS=CCL CCL_ATL_TRANSPORT=mpi /opt/intel/oneapi/intelpython/latest/envs/tensorflow/bin//python -u /home/vmagent/app/e2eaiok/modelzoo/resnet/mlperf_resnet/imagenet_main.py '123456' --label_smoothing '0.0' --num_filters '64' --data_dir '/home/vmagent/app/dataset/resnet/' --model_dir '/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa' --train_epochs '1' --stop_threshold '0.11656' --batch_size '204' --version 1 --resnet_size '50' --epochs_between_evals 1 --inter_op_parallelism_threads 2 --intra_op_parallelism_threads 2 --enable_lars --weight_decay '0.0001' --kernel_size '7' --base_lr '0.128' --momentum '0.9' --use_synthetic_data\n",
      "training launch command: /opt/intel/oneapi/intelpython/latest/envs/tensorflow/bin//horovodrun -n 2 -H localhost:2 HOROVOD_CPU_OPERATIONS=CCL CCL_ATL_TRANSPORT=mpi /opt/intel/oneapi/intelpython/latest/envs/tensorflow/bin//python -u /home/vmagent/app/e2eaiok/modelzoo/resnet/mlperf_resnet/imagenet_main.py '123456' --label_smoothing '0.0' --num_filters '64' --data_dir '/home/vmagent/app/dataset/resnet/' --model_dir '/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa' --train_epochs '1' --stop_threshold '0.11656' --batch_size '204' --version 1 --resnet_size '50' --epochs_between_evals 1 --inter_op_parallelism_threads 2 --intra_op_parallelism_threads 2 --enable_lars --weight_decay '0.0001' --kernel_size '7' --base_lr '0.128' --momentum '0.9' --use_synthetic_data\n",
      "[1]<stdout>:horovod size:2\n",
      "[0]<stdout>:horovod size:2\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896408, \"event_type\": \"POINT_IN_TIME\", \"key\": \"cache_clear\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 64}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896785, \"event_type\": \"INTERVAL_START\", \"key\": \"run_start\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 65}}\n",
      "[1]<stdout>:horovod size:2\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896408, \"event_type\": \"POINT_IN_TIME\", \"key\": \"cache_clear\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 64}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896789, \"event_type\": \"INTERVAL_START\", \"key\": \"run_start\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 65}}\n",
      "[0]<stdout>:horovod size:2\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896786, \"event_type\": \"POINT_IN_TIME\", \"key\": \"cache_clear\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop_eval.py\", \"lineno\": 64}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896791, \"event_type\": \"INTERVAL_START\", \"key\": \"run_start\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop_eval.py\", \"lineno\": 65}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896790, \"event_type\": \"POINT_IN_TIME\", \"key\": \"cache_clear\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop_eval.py\", \"lineno\": 64}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896794, \"event_type\": \"INTERVAL_START\", \"key\": \"run_start\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop_eval.py\", \"lineno\": 65}}\n",
      "[1]<stdout>:flags:Namespace(base_lr=0.128, batch_size=204, benchmark_log_dir=None, bigquery_data_set='test_benchmark', bigquery_metric_table='benchmark_metric', bigquery_run_table='benchmark_run', data_dir='/home/vmagent/app/dataset/resnet/', data_format=None, dtype=tf.float32, enable_lars=True, epochs_between_evals=1, eval_mode=False, export_dir=None, fine_tune=False, gcp_project=None, hooks=['LoggingTensorHook'], inter_op_parallelism_threads=2, intra_op_parallelism_threads=2, kernel_size=7, label_smoothing=0.0, loss_scale=1, max_train_steps=None, model_dir='/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa', momentum=0.9, num_filters=64, num_gpus=0, resnet_size=50, stop_threshold=0.11656, train_epochs=1, use_bfloat16=False, use_synthetic_data=True, version=1, weight_decay=0.0001)\n",
      "[1]<stdout>:Setting random seed =  123456\n",
      "[1]<stdout>:special seeding\n",
      "[1]<stderr>:2022-10-31 17:54:56.796487: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "[1]<stderr>:To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "[0]<stdout>:flags:Namespace(base_lr=0.128, batch_size=204, benchmark_log_dir=None, bigquery_data_set='test_benchmark', bigquery_metric_table='benchmark_metric', bigquery_run_table='benchmark_run', data_dir='/home/vmagent/app/dataset/resnet/', data_format=None, dtype=tf.float32, enable_lars=True, epochs_between_evals=1, eval_mode=False, export_dir=None, fine_tune=False, gcp_project=None, hooks=['LoggingTensorHook'], inter_op_parallelism_threads=2, intra_op_parallelism_threads=2, kernel_size=7, label_smoothing=0.0, loss_scale=1, max_train_steps=None, model_dir='/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa', momentum=0.9, num_filters=64, num_gpus=0, resnet_size=50, stop_threshold=0.11656, train_epochs=1, use_bfloat16=False, use_synthetic_data=True, version=1, weight_decay=0.0001)\n",
      "[0]<stdout>:Setting random seed =  123456\n",
      "[0]<stdout>:special seeding\n",
      "[0]<stderr>:2022-10-31 17:54:56.797658: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "[0]<stderr>:To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "[0]<stderr>:2022-10-31 17:54:56.806260: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      "[1]<stderr>:2022-10-31 17:54:56.806261: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896808, \"event_type\": \"POINT_IN_TIME\", \"key\": \"seed\", \"value\": 123456, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 522}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896808, \"event_type\": \"POINT_IN_TIME\", \"key\": \"seed\", \"value\": 123456, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 522}}\n",
      "[0]<stderr>:INFO:tensorflow:Initializing RunConfig with distribution strategies.\n",
      "[0]<stderr>:INFO:tensorflow:Not using Distribute Coordinator.\n",
      "[1]<stderr>:INFO:tensorflow:Initializing RunConfig with distribution strategies.\n",
      "[1]<stderr>:INFO:tensorflow:Not using Distribute Coordinator.\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896809, \"event_type\": \"POINT_IN_TIME\", \"key\": \"global_batch_size\", \"value\": 408, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 531}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896809, \"event_type\": \"POINT_IN_TIME\", \"key\": \"global_batch_size\", \"value\": 408, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 531}}\n",
      "[0]<stderr>:INFO:tensorflow:Using config: {'_model_dir': '/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main', '_tf_random_seed': 123456, '_save_summary_steps': 100, '_save_checkpoints_steps': 2502, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 2\n",
      "[0]<stderr>:inter_op_parallelism_threads: 2\n",
      "[0]<stderr>:allow_soft_placement: true\n",
      "[0]<stderr>:, '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 20, '_train_distribute': <tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy object at 0x7fc554149c50>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n",
      "[1]<stderr>:INFO:tensorflow:Using config: {'_model_dir': '/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/tmp1', '_tf_random_seed': 123456, '_save_summary_steps': 100, '_save_checkpoints_steps': 2502, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 2\n",
      "[1]<stderr>:inter_op_parallelism_threads: 2\n",
      "[1]<stderr>:allow_soft_placement: true\n",
      "[1]<stderr>:, '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 20, '_train_distribute': <tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy object at 0x7f75df1d4b10>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n",
      "[0]<stderr>:INFO:tensorflow:Using config: {'_model_dir': '/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main', '_tf_random_seed': 123456, '_save_summary_steps': 100, '_save_checkpoints_steps': 2502, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 2\n",
      "[0]<stderr>:inter_op_parallelism_threads: 2\n",
      "[0]<stderr>:allow_soft_placement: true\n",
      "[0]<stderr>:, '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 20, '_train_distribute': <tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy object at 0x7fc554149a10>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n",
      "[0]<stdout>:-----flags:Namespace(base_lr=0.128, batch_size=204, benchmark_log_dir=None, bigquery_data_set='test_benchmark', bigquery_metric_table='benchmark_metric', bigquery_run_table='benchmark_run', data_dir='/home/vmagent/app/dataset/resnet/', data_format=None, dtype=tf.float32, enable_lars=True, epochs_between_evals=1, eval_mode=False, export_dir=None, fine_tune=False, gcp_project=None, hooks=['LoggingTensorHook'], inter_op_parallelism_threads=2, intra_op_parallelism_threads=2, kernel_size=7, label_smoothing=0.0, loss_scale=1, max_train_steps=None, model_dir='/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa', momentum=0.9, num_filters=64, num_gpus=0, resnet_size=50, stop_threshold=0.11656, train_epochs=1, use_bfloat16=False, use_synthetic_data=True, version=1, weight_decay=0.0001)\n",
      "[0]<stdout>:flags.train_epochs // flags.epochs_between_evals:1\n",
      "[0]<stdout>:flags.train_epochs:1\n",
      "[0]<stdout>:flags.epochs_between_evals:1\n",
      "[1]<stderr>:INFO:tensorflow:Using config: {'_model_dir': '/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main', '_tf_random_seed': 123456, '_save_summary_steps': 100, '_save_checkpoints_steps': 2502, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 2\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896810, \"event_type\": \"INTERVAL_START\", \"key\": \"block_start\", \"value\": 1, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 609}}\n",
      "[1]<stderr>:inter_op_parallelism_threads: 2\n",
      "[1]<stderr>:allow_soft_placement: true\n",
      "[1]<stderr>:, '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 20, '_train_distribute': <tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy object at 0x7f75df1d4810>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}\n",
      "[1]<stdout>:-----flags:Namespace(base_lr=0.128, batch_size=204, benchmark_log_dir=None, bigquery_data_set='test_benchmark', bigquery_metric_table='benchmark_metric', bigquery_run_table='benchmark_run', data_dir='/home/vmagent/app/dataset/resnet/', data_format=None, dtype=tf.float32, enable_lars=True, epochs_between_evals=1, eval_mode=False, export_dir=None, fine_tune=False, gcp_project=None, hooks=['LoggingTensorHook'], inter_op_parallelism_threads=2, intra_op_parallelism_threads=2, kernel_size=7, label_smoothing=0.0, loss_scale=1, max_train_steps=None, model_dir='/home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa', momentum=0.9, num_filters=64, num_gpus=0, resnet_size=50, stop_threshold=0.11656, train_epochs=1, use_bfloat16=False, use_synthetic_data=True, version=1, weight_decay=0.0001)\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896810, \"event_type\": \"POINT_IN_TIME\", \"key\": \"first_epoch_num\", \"value\": 0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 610}}\n",
      "[1]<stdout>:flags.train_epochs // flags.epochs_between_evals:1\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896810, \"event_type\": \"POINT_IN_TIME\", \"key\": \"epoch_count\", \"value\": 1, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 611}}\n",
      "[1]<stdout>:flags.train_epochs:1\n",
      "[1]<stdout>:flags.epochs_between_evals:1\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896810, \"event_type\": \"INTERVAL_START\", \"key\": \"block_start\", \"value\": 1, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 609}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256896810, \"event_type\": \"POINT_IN_TIME\", \"key\": \"epoch_num\", \"value\": 0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 615}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896810, \"event_type\": \"POINT_IN_TIME\", \"key\": \"first_epoch_num\", \"value\": 0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 610}}\n",
      "[0]<stdout>:Starting a training cycle.\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896810, \"event_type\": \"POINT_IN_TIME\", \"key\": \"epoch_count\", \"value\": 1, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 611}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256896811, \"event_type\": \"POINT_IN_TIME\", \"key\": \"epoch_num\", \"value\": 0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 615}}\n",
      "[1]<stdout>:Starting a training cycle.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]<stderr>:INFO:tensorflow:Calling model_fn.\n",
      "[1]<stderr>:INFO:tensorflow:Calling model_fn.\n",
      "[0]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:414: UserWarning: `tf.layers.conv2d` is deprecated and will be removed in a future version. Please Use `tf.keras.layers.Conv2D` instead.\n",
      "[0]<stderr>:  warnings.warn('`tf.layers.conv2d` is deprecated and '\n",
      "[1]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:414: UserWarning: `tf.layers.conv2d` is deprecated and will be removed in a future version. Please Use `tf.keras.layers.Conv2D` instead.\n",
      "[1]<stderr>:  warnings.warn('`tf.layers.conv2d` is deprecated and '\n",
      "[0]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1692: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.\n",
      "[0]<stderr>:  warnings.warn('`layer.apply` is deprecated and '\n",
      "[1]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1692: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.\n",
      "[1]<stderr>:  warnings.warn('`layer.apply` is deprecated and '\n",
      "[1]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/normalization.py:308: UserWarning: `tf.layers.batch_normalization` is deprecated and will be removed in a future version. Please use `tf.keras.layers.BatchNormalization` instead. In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "[1]<stderr>:  '`tf.layers.batch_normalization` is deprecated and '\n",
      "[0]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/normalization.py:308: UserWarning: `tf.layers.batch_normalization` is deprecated and will be removed in a future version. Please use `tf.keras.layers.BatchNormalization` instead. In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "[0]<stderr>:  '`tf.layers.batch_normalization` is deprecated and '\n",
      "[1]<stderr>:WARNING:tensorflow:From /opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:534: _colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "[1]<stderr>:Instructions for updating:\n",
      "[1]<stderr>:Colocations handled automatically by placer.\n",
      "[1]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/pooling.py:310: UserWarning: `tf.layers.max_pooling2d` is deprecated and will be removed in a future version. Please use `tf.keras.layers.MaxPooling2D` instead.\n",
      "[1]<stderr>:  warnings.warn('`tf.layers.max_pooling2d` is deprecated and '\n",
      "[0]<stderr>:WARNING:tensorflow:From /opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:534: _colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "[0]<stderr>:Instructions for updating:\n",
      "[0]<stderr>:Colocations handled automatically by placer.\n",
      "[0]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/pooling.py:310: UserWarning: `tf.layers.max_pooling2d` is deprecated and will be removed in a future version. Please use `tf.keras.layers.MaxPooling2D` instead.\n",
      "[0]<stderr>:  warnings.warn('`tf.layers.max_pooling2d` is deprecated and '\n",
      "[0]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/core.py:171: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "[0]<stderr>:  warnings.warn('`tf.layers.dense` is deprecated and '\n",
      "[1]<stderr>:/opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/core.py:171: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "[1]<stderr>:  warnings.warn('`tf.layers.dense` is deprecated and '\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899489, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_learning_rate_decay_steps\", \"value\": 13756, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 246}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899494, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_base_learning_rate\", \"value\": 10.5, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 255}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899494, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_learning_rate_decay_poly_power\", \"value\": 2, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 256}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899494, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_end_learning_rate\", \"value\": 0.0001, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 257}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899495, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_learning_rate_warmup_epochs\", \"value\": 2, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 258}}\n",
      "[0]<stdout>::::MLPv0.5.0 resnet 1667256899.500624657 (/home/vmagent/app/e2eaiok/modelzoo/resnet/mlperf_resnet/resnet_run_loop.py:385) opt_learning_rate: \"DEFERRED: 70870cea-5245-4104-a6a2-520ec74d7c04\"\n",
      "[0]<stderr>:WARNING:tensorflow:From /home/vmagent/app/e2eaiok/modelzoo/resnet/mlperf_compliance/tf_mlperf_log.py:60: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20.\n",
      "[0]<stderr>:Instructions for updating:\n",
      "[0]<stderr>:Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:\n",
      "[0]<stderr>:\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899515, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_learning_rate_decay_steps\", \"value\": 13756, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 246}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899516, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_name\", \"value\": \"lars\", \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 400}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899516, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_epsilon\", \"value\": 0.0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 401}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256899517, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_weight_decay\", \"value\": 0.0001, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 402}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899522, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_base_learning_rate\", \"value\": 10.5, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 255}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899523, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_learning_rate_decay_poly_power\", \"value\": 2, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 256}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899523, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_end_learning_rate\", \"value\": 0.0001, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 257}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899523, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_learning_rate_warmup_epochs\", \"value\": 2, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 258}}\n",
      "[1]<stdout>::::MLPv0.5.0 resnet 1667256899.526709795 (/home/vmagent/app/e2eaiok/modelzoo/resnet/mlperf_resnet/resnet_run_loop.py:385) opt_learning_rate: \"DEFERRED: 2047fad9-4556-4b24-94eb-8b99f14447c1\"\n",
      "[1]<stderr>:WARNING:tensorflow:From /home/vmagent/app/e2eaiok/modelzoo/resnet/mlperf_compliance/tf_mlperf_log.py:60: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20.\n",
      "[1]<stderr>:Instructions for updating:\n",
      "[1]<stderr>:Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:\n",
      "[1]<stderr>:\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899540, \"event_type\": \"POINT_IN_TIME\", \"key\": \"opt_name\", \"value\": \"lars\", \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 400}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899541, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_epsilon\", \"value\": 0.0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 401}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256899541, \"event_type\": \"POINT_IN_TIME\", \"key\": \"lars_opt_weight_decay\", \"value\": 0.0001, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 402}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]<stderr>:INFO:tensorflow:Done calling model_fn.\n",
      "[0]<stderr>:INFO:tensorflow:Create CheckpointSaverHook.\n",
      "[1]<stderr>:INFO:tensorflow:Done calling model_fn.\n",
      "[1]<stderr>:INFO:tensorflow:Create CheckpointSaverHook.\n",
      "[0]<stderr>:WARNING:tensorflow:From /opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/util.py:96: DistributedIteratorV1.initialize (from tensorflow.python.distribute.input_lib) is deprecated and will be removed in a future version.\n",
      "[0]<stderr>:Instructions for updating:\n",
      "[0]<stderr>:Use the iterator's `initializer` property instead.\n",
      "[1]<stderr>:WARNING:tensorflow:From /opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/util.py:96: DistributedIteratorV1.initialize (from tensorflow.python.distribute.input_lib) is deprecated and will be removed in a future version.\n",
      "[1]<stderr>:Instructions for updating:\n",
      "[1]<stderr>:Use the iterator's `initializer` property instead.\n",
      "[0]<stderr>:INFO:tensorflow:Graph was finalized.\n",
      "[1]<stderr>:INFO:tensorflow:Graph was finalized.\n",
      "[0]<stderr>:2022-10-31 17:55:04.590556: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2600000000 Hz\n",
      "[1]<stderr>:2022-10-31 17:55:04.693731: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2600000000 Hz\n",
      "[0]<stderr>:INFO:tensorflow:Running local_init_op.\n",
      "[1]<stderr>:INFO:tensorflow:Running local_init_op.\n",
      "[0]<stderr>:INFO:tensorflow:Done running local_init_op.\n",
      "[1]<stderr>:INFO:tensorflow:Done running local_init_op.\n",
      "[0]<stderr>:INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "[0]<stderr>:INFO:tensorflow:Saving checkpoints for 0 into /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main/model.ckpt.\n",
      "[1]<stderr>:INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "[1]<stderr>:INFO:tensorflow:Saving checkpoints for 0 into /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/tmp1/model.ckpt.\n",
      "[0]<stderr>:INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "[1]<stderr>:INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "[0]<stderr>:2022-10-31 17:55:10.899049: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)\n",
      "[1]<stderr>:2022-10-31 17:55:10.959241: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)\n",
      "[1]<stderr>::::MLPv0.5.0 [2047fad9-4556-4b24-94eb-8b99f14447c1][1667256917.191329][0]\n",
      "[0]<stderr>::::MLPv0.5.0 [70870cea-5245-4104-a6a2-520ec74d7c04][1667256917.1847949][0]\n",
      "[0]<stderr>:INFO:tensorflow:cross_entropy = 6.908755, learning_rate = 0.0, train_accuracy = 1.0\n",
      "[0]<stderr>:INFO:tensorflow:{'num_examples': 0}\n",
      "[0]<stderr>:INFO:tensorflow:loss = 8.246834, step = 0\n",
      "[1]<stderr>:INFO:tensorflow:cross_entropy = 6.908755, learning_rate = 0.0, train_accuracy = 1.0\n",
      "[1]<stderr>:INFO:tensorflow:{'num_examples': 0}\n",
      "[1]<stderr>:INFO:tensorflow:loss = 8.246834, step = 0\n",
      "[1]<stderr>:INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1...\n",
      "[1]<stderr>:INFO:tensorflow:Saving checkpoints for 1 into /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/tmp1/model.ckpt.\n",
      "[1]<stderr>:WARNING:tensorflow:From /opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/training/saver.py:971: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "[1]<stderr>:Instructions for updating:\n",
      "[1]<stderr>:Use standard file APIs to delete files with this prefix.\n",
      "[0]<stderr>:INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1...\n",
      "[0]<stderr>:INFO:tensorflow:Saving checkpoints for 1 into /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main/model.ckpt.\n",
      "[0]<stderr>:WARNING:tensorflow:From /opt/intel/oneapi/intelpython/latest/envs/tensorflow/lib/python3.7/site-packages/tensorflow/python/training/saver.py:971: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "[0]<stderr>:Instructions for updating:\n",
      "[0]<stderr>:Use standard file APIs to delete files with this prefix.\n",
      "[1]<stderr>:INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1...\n",
      "[1]<stderr>:INFO:tensorflow:{'num_examples': 408}\n",
      "[0]<stderr>:INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1...\n",
      "[1]<stderr>:INFO:tensorflow:Loss for final step: 8.246834.\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256958114, \"event_type\": \"INTERVAL_END\", \"key\": \"block_stop\", \"value\": 1, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 672}}\n",
      "[1]<stdout>:Starting to evaluate.\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256958115, \"event_type\": \"INTERVAL_START\", \"key\": \"eval_start\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 687}}\n",
      "[1]<stderr>:INFO:tensorflow:Calling model_fn.\n",
      "[0]<stderr>:INFO:tensorflow:{'num_examples': 408}\n",
      "[0]<stderr>:INFO:tensorflow:Loss for final step: 8.246834.\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256958596, \"event_type\": \"INTERVAL_END\", \"key\": \"block_stop\", \"value\": 1, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 672}}\n",
      "[0]<stdout>:Starting to evaluate.\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256958596, \"event_type\": \"INTERVAL_START\", \"key\": \"eval_start\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 687}}\n",
      "[0]<stderr>:INFO:tensorflow:Calling model_fn.\n",
      "[1]<stderr>:INFO:tensorflow:Done calling model_fn.\n",
      "[1]<stderr>:INFO:tensorflow:Starting evaluation at 2022-10-31T17:56:00\n",
      "[0]<stderr>:INFO:tensorflow:Done calling model_fn.\n",
      "[0]<stderr>:INFO:tensorflow:Starting evaluation at 2022-10-31T17:56:00\n",
      "[1]<stderr>:INFO:tensorflow:Graph was finalized.\n",
      "[1]<stderr>:INFO:tensorflow:Restoring parameters from /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main/model.ckpt-1\n",
      "[1]<stderr>:INFO:tensorflow:Running local_init_op.\n",
      "[1]<stderr>:INFO:tensorflow:Done running local_init_op.\n",
      "[0]<stderr>:INFO:tensorflow:Graph was finalized.\n",
      "[0]<stderr>:INFO:tensorflow:Restoring parameters from /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main/model.ckpt-1\n",
      "[0]<stderr>:INFO:tensorflow:Running local_init_op.\n",
      "[0]<stderr>:INFO:tensorflow:Done running local_init_op.\n",
      "[1]<stderr>:INFO:tensorflow:Inference Time : 7.03990s\n",
      "[1]<stderr>:INFO:tensorflow:Finished evaluation at 2022-10-31-17:56:07\n",
      "[1]<stderr>:INFO:tensorflow:Saving dict for global step 1: accuracy = 1.0, accuracy_top_5 = 1.0, global_step = 1, loss = 8.246834, num_examples = 200\n",
      "[0]<stderr>:INFO:tensorflow:Inference Time : 6.54582s\n",
      "[0]<stderr>:INFO:tensorflow:Finished evaluation at 2022-10-31-17:56:07\n",
      "[0]<stderr>:INFO:tensorflow:Saving dict for global step 1: accuracy = 1.0, accuracy_top_5 = 1.0, global_step = 1, loss = 8.246834, num_examples = 200\n",
      "[1]<stderr>:INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1: /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main/model.ckpt-1\n",
      "[0]<stderr>:INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1: /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa/main/model.ckpt-1\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256967977, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_samples\", \"value\": 200, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 703}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256967977, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_samples\", \"value\": 200, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 703}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256967978, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_accuracy\", \"value\": 1.0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 705}}\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256967978, \"event_type\": \"INTERVAL_END\", \"key\": \"eval_stop\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 706}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256967978, \"event_type\": \"POINT_IN_TIME\", \"key\": \"eval_accuracy\", \"value\": 1.0, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 705}}\n",
      "[1]<stdout>:allreduced_results:1.0\n",
      "[1]<stderr>:INFO:tensorflow:Stop threshold of 0.11656 was passed with metric value 1.0.\n",
      "[1]<stdout>:train total time:71.16807818412781\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256967978, \"event_type\": \"INTERVAL_END\", \"key\": \"eval_stop\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 706}}\n",
      "[0]<stdout>:allreduced_results:1.0\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256967978, \"event_type\": \"POINT_IN_TIME\", \"key\": \"run_stop\", \"value\": {\"success\": true}, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 719}}\n",
      "[0]<stderr>:INFO:tensorflow:Stop threshold of 0.11656 was passed with metric value 1.0.\n",
      "[1]<stdout>::::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1667256967978, \"event_type\": \"INTERVAL_END\", \"key\": \"run_stop\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 720}}\n",
      "[0]<stdout>:train total time:71.16870665550232\n",
      "[1]<stdout>:Total time:71.18707275390625\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256967979, \"event_type\": \"POINT_IN_TIME\", \"key\": \"run_stop\", \"value\": {\"success\": true}, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 719}}\n",
      "[0]<stdout>::::MLLOG {\"namespace\": \"worker0\", \"time_ms\": 1667256967979, \"event_type\": \"INTERVAL_END\", \"key\": \"run_stop\", \"value\": null, \"metadata\": {\"file\": \"resnet/mlperf_resnet/resnet_run_loop.py\", \"lineno\": 720}}\n",
      "[0]<stdout>:Total time:71.18531727790833\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-10-31 17:56:09,346 - sigopt - INFO - Training completed based in sigopt suggestion, took 80.96904230117798 secs\r\n",
      "2022-10-31 17:56:09,347 - E2EAIOK.SDA - INFO - training script completed\r\n",
      "\r\n",
      "We found the best model! Here is the model explaination\r\n",
      "\r\n",
      "===============================================\r\n",
      "***    Best Trained Model    ***\r\n",
      "===============================================\r\n",
      "  Model Type: resnet\r\n",
      "  Model Saved Path: /home/vmagent/app/e2eaiok/result/159938443786e7f5538c8ba256dbfefa\r\n",
      "  Sigopt Experiment id is None\r\n",
      "  === Result Metrics ===\r\n",
      "    accuracy: 1.0\r\n",
      "    training_time: 80.96904230117798\r\n",
      "===============================================\r\n"
     ]
    }
   ],
   "source": [
    "# Train (single node)\n",
    "from e2eAIOK.SDA.SDA import SDA\n",
    "import yaml\n",
    "\n",
    "# create SDA settings\n",
    "settings = {}\n",
    "settings[\"enable_sigopt\"] = False\n",
    "# load ResNet settings\n",
    "with open(\"/home/vmagent/app/e2eaiok/conf/e2eaiok_defaults_resnet_example.conf\") as f:\n",
    "    conf = yaml.load(f, Loader=yaml.FullLoader)\n",
    "settings.update(conf)\n",
    "\n",
    "sda = SDA(model=\"resnet\", settings=settings)\n",
    "sda.launch()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}