mala_tutorial_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Machine learning for computational materials science and chemistry with MALA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## What is MALA?\n",
    "\n",
    "A short summary about MALA and how it works is given in `mala_background.pdf`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Setting up MALA\n",
    "\n",
    "For this tutorial, a Google Collab enviroment will be provided that includes all necessary packages. Generally, MALA is an open source framework that can be obtained [here](https://github.com/mala-project/mala). Detailled (installation) instructions can be found [here](https://mala-project.github.io/mala/).\n",
    "\n",
    "A few examples at the end of the notebook tackle advanced applications. The necessary backends are, for the ease of installation, not bundled with the Google Collab environment. Interested readers may install them themselves on their machines, for in presence workshops, they will be demonstrated by the host, as are sosme aspects of data generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Loading the modules\n",
    "\n",
    "These modules will be necessary for the tutorials discussed here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# MALA itself.\n",
    "\n",
    "import mala\n",
    "\n",
    "# We would like to visualize simple plots.\n",
    "# The font size can sometimes be a bit small for Jupyter Notebooks.\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "font = {'size'   : 22}\n",
    "matplotlib.rc('font', **font)\n",
    "\n",
    "# For the data paths.\n",
    "from os.path import join as pj\n",
    "\n",
    "# Only for the prediction down below.\n",
    "from ase.io import read\n",
    "\n",
    "# To do some timings down below.\n",
    "from time import time\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Data I: Performing simulations (presentation only)\n",
    "\n",
    "Data is the backbone of each ML application. For MALA, this includes target data (some electronic structure quantity to be learned, usually the local density of states, LDOS) and some descriptor data (a vectorial field that encodes the atomic density at each point in space, usually via so called bispectrum components).\n",
    "\n",
    "MALA data generation can be performed with the Quantum ESPRESSO package. Some changes to this open source package were necessary to enable the correct sampling of the LDOS. The current development branch of Quantum ESPRESSO includes those - beginning with Quantum ESPRESSO version 7.2 (to be released in ~June 2023) users can simply download the latest QE version and perform data generation.\n",
    "\n",
    "Data generation is two-fold: First, one creates a set of atomic position via a regular DFT-MD simulation at the conditions of interest. This can be done with any suitable code, such as VASP, QE, etc. Secondly, one performs DFT simulations to access the LDOS.\n",
    "\n",
    "The test system for our investigation here will be a simply beryllium system at room temperature consisting of 2 beryllium atoms. Atomic configurations have been sampled beforehand. We will start with the DFT simulation. Note: For actual data generation, the simulation output needs to be saved (usually as \".out\" file) and is of course done on HPC infrastructure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "MALA data generation thus comes down to a simply DFT + Postprocessing calculation. There is one drawback though: The LDOS has to be sampled with a very high fidelity in the k-space (phase space, i.e., how the Fourier components of the basis set are sampled). The fidelity of the MALA calculation has to be higher then for standard DFT calculations, which has to be kept in mind. A good way to visualize the problem is through the density of states (LDOS integrated on real space grid) which shows unphysical oscillations for low fidelity calculations. These oscillations vanish as one moves to higher fidelity calculations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "After we have performed the actual simulation, we are halfway done. We now have simulation outputs, both the DFT output as well as the LDOS. From this we need to do two things:\n",
    "\n",
    "1. We need to convert the LDOS into a format we actually want to work with. Cube files are unnecessarily huge, complicated both disk usage as well as speed.\n",
    "2. We need to calculate the atomic density descriptors from the atomic positions.\n",
    "\n",
    "So it's finally time to fire up MALA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# The MALA interface (hands-on)\n",
    "\n",
    "Within extended ML frameworks, a big problem is reproducibility. Models often depend on a lot of so called hyperparameters, that characterize model behavior. This may include, but in no way be limited to, neural network layer sizes and number, training procedures, description of data specifics (e.g.: how is the local density of states sampled?), etc.\n",
    "There is a number of ways to efficiently handle this problem. A lot of frameworks rely on command line arguments to deal with this, i.e., the user provides a, potentially extensive, list of command line arguments upon runtime. Another good way to handle all this is the usage of input files. This is consistent with the way standard computational science simulation codes work.\n",
    "An obvious downside to this is that one has to have a framework at hand to prepare these input files.\n",
    "\n",
    "MALA follows a route sort of in between the two approaches. The central quantity is the `Parameters()` object. It holds ALL (hyper)parameters one could use in the course of a MALA run. It is structured by the subtasks of MALA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mala.common.parameters.ParametersHyperparameterOptimization at 0x7f9fb69c24d0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "\n",
    "# All parameters related to how data is handled in general.\n",
    "parameters.data\n",
    "\n",
    "# \"Targets\" always refers to the quantity being learned.\n",
    "parameters.targets\n",
    "\n",
    "# \"Descriptors\" always refers to the quantity from which we learn.\n",
    "parameters.descriptors\n",
    "\n",
    "# \"Data generation\" refers to useful routines for creating training data.\n",
    "# These routines are mostly experimental at the moment and will not be discussed here in detail\n",
    "parameters.datageneration\n",
    "\n",
    "# \"Network\" refers to everything related to neural network creation and training.\n",
    "# In the future, support for more models is planned, and this collection of parameters\n",
    "# will be updated to reflect this.\n",
    "parameters.network\n",
    "\n",
    "# \"Hyperparameters\" means hyperparameter optimization. This is the process of finding the optimal\n",
    "# hyperparameters for MALA model training, and we will come back to this process later.\n",
    "parameters.hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Individual parameter objects can be printed to see what's hidden inside. This also works on the main object as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "snapshot_directories_list: []\n",
      "data_splitting_type: by_snapshot\n",
      "input_rescaling_type: None\n",
      "output_rescaling_type: None\n",
      "use_lazy_loading: False\n",
      "use_lazy_loading_prefetch: False\n",
      "use_fast_tensor_data_set: False\n",
      "shuffling_seed : None\n"
     ]
    }
   ],
   "source": [
    "parameters.data.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Finally, there are some high-level parameters one needs when performing ML at scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Whether or not to use a GPU for model training and inference.\n",
    "parameters.use_gpu\n",
    "\n",
    "# Whether or not to use MPI parallel CPU inference (no training supported).\n",
    "# This option is either for pre-processing or production runs of trained models.\n",
    "# More on that later.\n",
    "parameters.use_mpi\n",
    "\n",
    "# Manual seeds can be used to fix the Pseudo RNG to re-create models with the exact\n",
    "# same model weights.\n",
    "parameters.manual_seed\n",
    "\n",
    "# A comment may be useful to distinguish between sets of parameters.\n",
    "parameters.comment\n",
    "\n",
    "# This is useful for adjusting the output level of MALA.\n",
    "parameters.verbosity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "All of this does not explain how MALA handles reproducibility. Write hundreds of lines of parameter statement in each python script is not exactly maintainable.\n",
    "Therefore MALA provides a .json interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "parameters.save(\"mala_parameters_01.json\")\n",
    "new_parameters = mala.Parameters.load_from_file(\"mala_parameters_01.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Have a look at the .json file that was just created. You will see that it is structured in the same ways as the python object, allowing fast access. You will see some parameters that are not dicussed here since they exceed the scope of this tutorial. For a first excercise, try to modify parameters both in python and in json and see whether loading will recover those changes. The comment and manual seed are good first examples for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Set a comment\n",
    "parameters.comment = \"My first parameters.\"\n",
    "\n",
    "# Save.\n",
    "parameters.save(\"mala_parameters_01.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---     All parameter MALA needs to perform its various tasks. ---\n",
      "comment        : My first parameters.\n",
      "manual_seed    : None\n",
      "use_gpu        : False\n",
      "device         : cpu\n",
      "use_horovod    : False\n",
      "use_mpi        : False\n",
      "verbosity      : 1\n",
      "openpmd_configuration: {}\n",
      "openpmd_granularity: 1\n",
      "---     Parameters necessary for constructing a neural network. ---\n",
      "\tnn_type        : feed-forward\n",
      "\tlayer_sizes    : [10, 10, 10]\n",
      "\tlayer_activations: ['Sigmoid']\n",
      "\tloss_function_type: mse\n",
      "\tnum_hidden_layers: 1\n",
      "\tno_hidden_state: False\n",
      "\tbidirection    : False\n",
      "\tdropout        : 0.1\n",
      "\tnum_heads      : 10\n",
      "---     Parameters necessary for calculating/parsing input descriptors. ---\n",
      "\tdescriptor_type: Bispectrum\n",
      "\tlammps_compute_file: \n",
      "\tdescriptors_contain_xyz: True\n",
      "\tuse_z_splitting: True\n",
      "\tnumber_y_planes: 0\n",
      "\tbispectrum_twojmax: 10\n",
      "\trcutfac        : 4.67637\n",
      "\tatomic_density_cutoff: 4.67637\n",
      "\tsnap_switchflag: 1\n",
      "\tuse_atomic_density_energy_formula: False\n",
      "\tatomic_density_sigma: None\n",
      "\tminterpy_point_list: []\n",
      "\tminterpy_cutoff_cube_size: 0.0\n",
      "\tminterpy_polynomial_degree: 4\n",
      "\tminterpy_lp_norm: 2\n",
      "---     Parameters necessary for calculating/parsing output quantites. ---\n",
      "\ttarget_type    : LDOS\n",
      "\tldos_gridsize  : 0\n",
      "\tldos_gridspacing_ev: 0\n",
      "\tldos_gridoffset_ev: 0\n",
      "\trestrict_targets: zero_out_negative\n",
      "\tpseudopotential_path: None\n",
      "\trdf_parameters : {'number_of_bins': 500, 'rMax': 'mic'}\n",
      "\ttpcf_parameters: {'number_of_bins': 20, 'rMax': 'mic'}\n",
      "\tssf_parameters : {'number_of_bins': 100, 'kMax': 12.0}\n",
      "---     Parameters necessary for loading and preprocessing data. ---\n",
      "\tsnapshot_directories_list: []\n",
      "\tdata_splitting_type: by_snapshot\n",
      "\tinput_rescaling_type: None\n",
      "\toutput_rescaling_type: None\n",
      "\tuse_lazy_loading: False\n",
      "\tuse_lazy_loading_prefetch: False\n",
      "\tuse_fast_tensor_data_set: False\n",
      "\tshuffling_seed : None\n",
      "---     Parameters needed for network runs (train, test or inference). ---\n",
      "\ttrainingtype   : SGD\n",
      "\tlearning_rate  : 0.5\n",
      "\tmax_number_epochs: 100\n",
      "\tverbosity      : True\n",
      "\tmini_batch_size: 10\n",
      "\tweight_decay   : 0\n",
      "\tearly_stopping_epochs: 0\n",
      "\tearly_stopping_threshold: 0\n",
      "\tlearning_rate_scheduler: None\n",
      "\tlearning_rate_decay: 0.1\n",
      "\tlearning_rate_patience: 0\n",
      "\tuse_compression: False\n",
      "\tnum_workers    : 0\n",
      "\tuse_shuffling_for_samplers: True\n",
      "\tcheckpoints_each_epoch: 0\n",
      "\tcheckpoint_name: checkpoint_mala\n",
      "\tvisualisation  : 0\n",
      "\tvisualisation_dir: ./mala_logging\n",
      "\tvisualisation_dir_append_date: True\n",
      "\tduring_training_metric: ldos\n",
      "\tafter_before_training_metric: ldos\n",
      "\tinference_data_grid: [0, 0, 0]\n",
      "\tuse_mixed_precision: False\n",
      "\tuse_graphs     : False\n",
      "\ttraining_report_frequency: 1000\n",
      "\tprofiler_range : [1000, 2000]\n",
      "---     Hyperparameter optimization parameters. ---\n",
      "\tdirection      : minimize\n",
      "\tn_trials       : 100\n",
      "\thyper_opt_method: optuna\n",
      "\tcheckpoints_each_trial: 0\n",
      "\tcheckpoint_name: checkpoint_mala_ho\n",
      "\tstudy_name     : None\n",
      "\trdb_storage    : None\n",
      "\trdb_storage_heartbeat: None\n",
      "\tnumber_training_per_trial: 1\n",
      "\ttrial_ensemble_evaluation: mean\n",
      "\tuse_multivariate: True\n",
      "\tnaswot_pruner_cutoff: 0\n",
      "\tpruner         : None\n",
      "\tnaswot_pruner_batch_size: 0\n",
      "\tnumber_bad_trials_before_stopping: None\n",
      "\tsqlite_timeout : 600\n",
      "\tacsd_points    : 100\n",
      "---     All parameters to help with data generation. ---\n",
      "\ttrajectory_analysis_denoising_width: 100\n",
      "\ttrajectory_analysis_below_average_counter: 50\n",
      "\ttrajectory_analysis_estimated_equilibrium: 0.1\n",
      "\ttrajectory_analysis_correlation_metric_cutoff: -0.1\n",
      "\ttrajectory_analysis_temperature_tolerance_percent: 1\n",
      "\tlocal_psp_path : None\n",
      "\tlocal_psp_name : None\n",
      "\tofdft_timestep : 0\n",
      "\tofdft_number_of_timesteps: 0\n",
      "\tofdft_temperature: 0\n",
      "\tofdft_kedf     : WT\n",
      "\tofdft_friction : 0.1\n"
     ]
    }
   ],
   "source": [
    "# Edit something in the file (e.g. the manual_seed) and reload.\n",
    "parameters = mala.Parameters.load_from_file(\"mala_parameters_01.json\")\n",
    "parameters.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "For extended experiments, it is very useful to operate with such input files and only use the in-python parameter editing when absolutely necessary. Further, concluded experiments can be saved in this way for future reference.\n",
    "In the following, we will use a combination of both approaches for the sake of transparency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Data II: Data preprocessing (presentation-only)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We can now start using MALA to prepare our data. MALA directly takes in calculation outputs and transforms it into formats with which we can easily work.\n",
    "First, we have to decide which descriptors to calculate and how to correctly process the LDOS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "parameters = mala.Parameters()\n",
    "\n",
    "# These values we will take for granted now. In the hyperparameter section we will find out\n",
    "# how they are determined.\n",
    "parameters.descriptors.descriptor_type = \"Bispectrum\"\n",
    "parameters.descriptors.bispectrum_twojmax = 10\n",
    "parameters.descriptors.bispectrum_cutoff = 4.67637\n",
    "parameters.descriptors.descriptors_contain_xyz = True\n",
    "\n",
    "# These values need to correspond to the ones used in the DFT simulation.\n",
    "parameters.targets.target_type = \"LDOS\"\n",
    "parameters.targets.ldos_gridsize = 11\n",
    "parameters.targets.ldos_gridspacing_ev = 2.5\n",
    "parameters.targets.ldos_gridoffset_ev = -5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we can use the `DataConverter` class to convert simulation outputs. File format labels follow the ASE package wherever possible!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: RDMAV_FORK_SAFE=1\n",
      "Disabling z-splitting for preprocessing.\n",
      "Calculating descriptors from ./data_generation/dft.out\n",
      "Reading 11 LDOS files from./data_generation/tmp.pp0*Be_ldos.cube.\n"
     ]
    }
   ],
   "source": [
    "%env RDMAV_FORK_SAFE=1\n",
    "\n",
    "data_converter = mala.DataConverter(parameters)\n",
    "data_converter.add_snapshot(descriptor_input_type=\"espresso-out\",\n",
    "                            descriptor_input_path=\"./data_generation/dft.out\",\n",
    "                            target_input_type=\".cube\",\n",
    "                            target_input_path=\"./data_generation/tmp.pp0*Be_ldos.cube\")\n",
    "data_converter.convert_snapshots(\"./data_generation/\", naming_scheme=\"Be_snapshot*.npy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In this example we will use the [MALA test data set](https://github.com/mala-project/test-data), which contains 4 atomic configurations, including simulation output, bispectrum components and LDOS - i.e., this preprocessing has already been done for all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "data_path = \"/home/fiedlerl/data/mala_data_repo/Be2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Visualizing and reproducing output data (hands-on)\n",
    "\n",
    "Before we train a model, it is a good idea to think about which metric is important, i.e., how do we test if a model is good?\n",
    "\n",
    "In essence, the advantage of MALA is the access to multiple observables. Two easily accesible metrics are the density of states (DOS) and the band energy. We will now learn how to calculate them from the LDOS (the actual DFT LDOS in this case) so we can do the same after model training to test our models.\n",
    "\n",
    "For this, we first have to make sure the correct LDOS parameters are used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "parameters.targets.target_type = \"LDOS\"\n",
    "parameters.targets.ldos_gridsize = 11\n",
    "parameters.targets.ldos_gridspacing_ev = 2.5\n",
    "parameters.targets.ldos_gridoffset_ev = -5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we can create an LDOS calculator and directly populate it with the LDOS data from the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ldos_calculator = mala.LDOS.from_numpy_file(parameters, pj(data_path, \"Be_snapshot0.out.npy\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Afterwards, we have to read in some additional information from the simulation data (size of the real space grid, temperature, etc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ldos_calculator.read_additional_calculation_data(pj(data_path, \"Be_snapshot0.out\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we can access the DOS and the band energy as properties of the calculator object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.447077138958328\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAJNCAYAAACIkPmLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABcpElEQVR4nO3dd3yVZZ7+8eubTgstAUIKvUgvARsWGHWsKNLEcYrj7BTL/nacnR1np3d3ZnfcnaKrs+o0K0EFy4wFsaOS0HuHnAAhEEpISL9/f+TEiUhCgJzcp3zer9d5kTyncGViyDXPc+7vbc45AQAAIPzF+Q4AAACA1qG4AQAARAiKGwAAQISguAEAAEQIihsAAECEoLgBAABEiATfAdpDWlqa69+/v+8YAAAAp1RQUHDAOZd+svtiorj1799f+fn5vmMAAACckpntau4+LpUCAABECIobAABAhKC4AQAARAiKGwAAQISguAEAAEQIihsAAECEoLgBAABECIobAABAhKC4AQAARAiKGwAAQISguAEAAEQIihsAAECEoLgBAABECIobAABAhKC4AQAARAiKGwAAQISguAEAAEQIihsAAECEoLgBAABECIobAABAhKC4AQAARAiKGwAAQIRI8B0AACLNhr1H9V+vbFbn5HjNnJilCwalKT7OfMcCEAMobgDQSseqavXfr27Wo+/tVGpKgurqnZ5buUcZXVM0Y3ymZk7M0qD0zr5jAohiFDcAOAXnnP6+dp9+9Px67TtaqXmTc/StK4cpJTFeizfs14LlAT341nbd/8Y2jcvuppkTszR9TF917ZjoOzqAKGPOOd8ZQi43N9fl5+f7jgEgAu0+WKEfLFqrJZtKdE5Gqn42Y5Qm5HT/xOP2l1Vq0co9yisIaOO+MiXFx+nyEb01c2KmLh6SroR43lIMoHXMrMA5l3vS+yhuAPBJVbV1eujN7frdkq1KiDPdfcUwff78fqcsYM45rdtzVAuWB7Rw5R6VllcrrXOyZozvq5kTszS8T2o7fQUAIhXFjeIG4DS8t/WAvrtwrbaXlOua0Rn63rUj1Kdrymm/TnVtvd7Y1HAp9fWN+1VT5zSyb6pmTsjS9eP6qmfn5BCkBxDpKG4UNwCtsL+sUj97cYMWrtyjfj076kfTR+rSYb3a5LVLy6u1aGWRFiwv0pqiI0qIM00d3kszJ2Rp2vBeSkrgUiqABhQ3ihuAFtTVOz32wS796uVNqqqp11cvHaTbLx2klMT4kPx9m4vLtKAgoGdWFKmkrErdOybq+nGZmjkhS6MyU2XGaBEgllHcKG4AmrE6cFjffW6tVgeOaMrgNP34+pEa2E4jPWrr6vX21gNaUBDQK+uLVV1br6G9O2vmhCzNGJ+pXqmnf3kWQOSjuFHcAJzgyPEa/dcrm/SX93cprXOyvnftCF03JsPb2a4jx2v0wuo9WlAQ0PLdhxVn0sVD0zVzQpYuH9E7ZGf/AIQfihvFDUCQc06LVu3RT17YoNLyKn3u/P66+4qhSk0Jn5lr20uO6ZnlRXpmeUB7jlSqS0qCrhvbVzMnZGlCTjcupQJRjuJGcQMgaVvJMX1/4Vq9u/WgxmZ11c9mjNaozK6+YzWrvt5p6faDWlAQ0N/W7tPxmjoNTOukGydkasaELGV26+A7IoAQoLhR3ICYVllTp98v2aoH39yu5MQ4/duVw3Xz5JyI2l/0WFWtXlqzVwsKAvpgR6nMpAsG9dTMCVm6clQfdUxiIxwgWlDcKG5AzFqyab9+sHCddpdWaMb4TP371ecovUtkz08rLK3QM8uLtGB5QLtLK9QpKV5Xj87QzIlZmty/h+IiqJAC+CSKG8UNiDl7jxzXj59fr7+t3adB6Z30kxtG6YJBab5jtSnnnJbtPKQFBQG9uGavjlXVKqt7B82ckKWZE7KU07Oj74gAzgDFjeIGxIzaunr98b2duu/Vzaqtd/rnTw3RP100MOoH3B6vrtPL6/ZpwfKA3tl6QM5Jk/v30MyJmbp6dIa6hNHiCwAt81bczOxKSf8jKV7S/znn7j3h/q9KukNSnaRjkr7snFsfvO/bkm4L3vfPzrmXW/OaJ0NxA2JDwa5SfefZtdq4r0zThvfSj6aPVHaP2DvrtPfIcT27okh5BQFtLylXSmKcrhzZRzMnZumCQWkR9d4+IBZ5KW5mFi9ps6TLJQUkLZM0r7GYBR+T6pw7Gvx4uqTbnXNXmtkISU9Imiypr6TXJA0NPq3F1zwZihsQ3Q6VV+s//r5RTy4rVEbXFP1w+khdMaJ3zI/NcM5pZeFhLVge0KKVe3S0slYZXVM0Y3ymZk7M0qB2GjQM4PS0VNxCuQxpsqStzrntwRBPSrpe0kclq7G0BXWS1Ngir5f0pHOuStIOM9safD2d6jUBxI76eqe85QH94qUNKqus1VcuHqh//tQQdUpmhaUkmZnG53TX+Jzu+u41I7R4Q8OG9w++tV33v7FN47K7aebELE0f01ddO3IpFYgEofzXLVNSYZPPA5LOPfFBZnaHpLslJUma1uS575/w3Mzgx6d8TQDRb9O+Mn33uTVatvOQJvXvrp/eMFrD+nTxHStspSTG65oxGbpmTIb2l1Vq0co9yisI6HvPrdVPX1ivBz87UZcO6+U7JoBT8P5uXefc751zgyR9S9J32+p1zezLZpZvZvklJSVt9bIAPCuvqtUvXtqgq3/ztrbuP6Zfzhqjp758PqXtNPTqkqIvXTRQf/t/F+mFu6ZoQFon/ctTKxU4VOE7GoBTCGVxK5KU3eTzrOCx5jwp6YZTPLfVr+mce8g5l+ucy01PTz+95ADCjnNOf1+7T5f/+k09+NZ2zZ6Ypde/canm5GYzt+wMmZlGZXbV/94yUXV1Tnc8vkJVtXW+YwFoQSiL2zJJQ8xsgJklSbpJ0qKmDzCzIU0+vUbSluDHiyTdZGbJZjZA0hBJH7bmNQFEn8LSCt32p3x99a8FSu2QqAVfO1/3zhyj7p2SfEeLCv3TOulXs8dqVeFh/fzFDb7jAGhByN7j5pyrNbM7Jb2shtEdjzjn1pnZjyXlO+cWSbrTzC6TVCPpkKTPB5+7zsyeVsOig1pJdzjn6iTpZK8Zqq8BgF/VtfX6w9vb9dvXtyjOTN+95hx94YL+Soj3/i6PqHPlqD76p4sG6A9v79DE/j00fWxf35EAnAQDeAGEpaXbDuq7z63RtpJyXTWqj75/3QhldGVT9VCqqavXvIfe1/q9R7Xozika3ItxIYAPLY0D4f+2AggrJWVV+vpTKzXvD++ruq5ej946SQ/cMpHS1g4S4+P0u5snqENivG5/rEAV1bW+IwE4AcUNQFioq3f6y/u79Kn/ekMvrN6ju6YN1qtfv0RTGVHRrvp0TdH/3DReW/Yf03eeXatYuCoDRBKmVALwbm3REX3nubVaVXhYFwzqqZ/cMIqp/h5NGZKmr182VL9+dbMm9e+hm8/N8R0JQBDFDYA3Rytr9OtXNuvPS3eqR6dk/c9N4zR9bN+Y36oqHNw5dbDydx3SDxet0+jMrhqd1dV3JADiUikAD5xzen7VHl32X2/qT0t36rPn9dPib1yi68dlUtrCRFyc6b/njlNa5yTd/niBjlTU+I4EQBQ3AO1sx4Fyfe6RD3XXEyvUOzVFC++4UD+6fpS6dmCvzHDTo1OSfveZCdp3pFLfmL+K97sBYYDiBqDdPL2sUJ++7y2tLDysn1w/Us/dcaHGZHXzHQstmJDTXf9+9Tl6bUOxHnpru+84QMzjPW4A2sX2kmP67sK1yu3fXf990zj16pLiOxJa6QsX9Ff+zkP65cubNC67m84d2NN3JCBmccYNQMjV1zvds2CNUhLiKG0RyMx078zR6tejo+58YoX2l1X6jgTELIobgJB77INd+nBnqb537QhKW4TqkpKo+2+ZoLLKGv2/J1aqtq7edyQgJlHcAIRU0eHjuvdvG3XRkDTNmpjlOw7OwvA+qfrZDaO1dPtB3ffaZt9xgJhEcQMQMs45fefZNXKSfj5jNKM+osDMiVmaNzlbv1+yTa9vLPYdB4g5FDcAIfPcyiK9salE//bpYcru0dF3HLSRH1w3UiP7purrT61SYWmF7zhATKG4AQiJA8eq9KPn12tiv+767Pn9fcdBG0pJjNf9n5mgeud05+PLVVVb5zsSEDMobgBC4oeL1qmiqk7/MXO04uO4RBpt+vXspP+cPVarAkf0sxc3+I4DxAyKG4A298q6fXph9V7986cGa3CvLr7jIEQ+PbKPvnzxQP156S4tXFnkOw4QEyhuANrUkeM1+u5za3VORqq+cskg33EQYt/89DBN6t9d335mjbbuL/MdB4h6FDcAbeoXL23QwfJq/WrWGCXG809MtEuMj9Pvbp6gjknx+upfl6u8qtZ3JCCq8a8qgDbz7tYDenJZof7pooEaldnVdxy0k96pKfqfm8Zre8mxhvEvbEYPhAzFDUCbqKiu1befWaMBaZ30L5cN8R0H7ezCwWn6+mVD9dzKPXrsg92+4wBRi+IGoE38+pXN2l1aoXtvHK2UxHjfceDBHVMH69Jh6frx8+u1JnDEdxwgKlHcAJy1FbsP6ZF3d+iW83J07sCevuPAk7g4031zximtc5K+9liBjlTU+I4ERB2KG4CzUlVbp3/LW60+qSn61pXDfceBZ907Jen3n5mg4qOVuvvplaqv5/1uQFuiuAE4K/cv2aYt+4/pZzNGq0tKou84CAPjc7rru9eM0OKN+/XgW9t9xwGiCsUNwBnbuO+o7n9jq2aMz9TU4b18x0EY+dz5/XTtmAz96uWNWrrtoO84QNSguAE4I3X1Tt/KW63UlER979oRvuMgzJiZ7p05Rv3TOumuJ1Zo/9FK35GAqEBxA3BGHn13h1YFjuiH00eqR6ck33EQhjonJ+iBz0zUsaoa3fXECtXW1fuOBEQ8ihuA07bzQLn+85VNuuyc3rp2TIbvOAhjw/p00c9njNYHO0r161c3+44DRDyKG4DT4pzTPc+sVmJ8nH42Y5TMzHckhLkbJ2Rp3uQc3f/GNi3eUOw7DhDRKG4ATsuTywr1/vZSfefqc9Q7NcV3HESIH1w3QiP7purrT61UYWmF7zhAxKK4AWi1fUcq9fMXN+j8gT01d1K27ziIICmJ8XrgMxPlJN3+2HJV1db5jgREJIobgFZxzum7z61RTX297p05mkukOG05PTvq13PGaU3REf3khfW+4wARieIGoFWeX71Xr23Yr3+9Ypj69ezkOw4i1OUjeusrlwzUX9/fredWFPmOA0QcihuAUyotr9YPF63T2OxuuvXCAb7jIMJ984phmty/h779zBptKS7zHQeIKBQ3AKf04+fXqayyRr+cOUbxcVwixdlJiI/Tb28er07J8fraY8tVXlXrOxIQMShuAFr0+sZiPbdyj+6YOljD+nTxHQdRondqin4zb7y2lxzTt59ZI+fYjB5oDYobgGaVVdboO8+u1bDeXXT7pYN9x0GUuWBQmr5xxTAtWrVHf/1gt+84QESguAFo1n/8faOKj1bqP2aNUVIC/1yg7X3tkkGaOixdP3l+vVYVHvYdBwh7/EsM4KTe335Qf31/t7544QCNy+7mOw6iVFyc6b6545TeJVm3P7ZchyuqfUcCwhrFDcAnVNbU6Z4Fq5XTo6O+ccUw33EQ5bp1TNL9n5mg/WWVuvvpVaqv5/1uQHMobgA+4b7XNmvnwQrdO3O0OiTF+46DGDA2u5u+d+0Ivb5xv/73rW2+4wBhi+IG4GNWBw7rD29t17zJ2bpgUJrvOIghnz2vn64b21f/+fImLd120HccICxR3AB8pKauXv+Wt1rpXZJ1z1Xn+I6DGGNm+sWNozUgrZPuemKF9h+t9B0JCDsUNwAfefDNbdq4r0w/vWG0unZI9B0HMahzcoIeuGWiyqtqdecTK1RbV+87EhBWKG4AJElbisv0m8Vbde2YDF0+orfvOIhhQ3t30S9uHK0Pd5TqP1/Z7DsOEFYobgBUV+/0rQWr1Sk5Xj+cPtJ3HEA3jM/UZ87N0f++uU2vri/2HQcIGxQ3APrz0p1avvuwfnDdSKV1TvYdB5Akfe/aERqVmapvPL1ShaUVvuMAYYHiBsS4wtIK/fLvmzR1WLquH9fXdxzgIymJ8XrgMxMlSV97rECVNXWeEwH+UdyAGOac078/u0ZxJv1sxmiZme9IwMdk9+io/5ozTmuLjuonL6z3HQfwjuIGxLC8goDe3nJA91x9jvp26+A7DnBSl4/ora9eMkiPfbBbz64I+I4DeEVxA2LU/qOV+skL6zW5fw99ZnKO7zhAi/71iqGaPKCH/v2ZtdpcXOY7DuANxQ2IUd9fuE5VtfW6d+ZoxcVxiRThLSE+Tr+bN16dkhP01b8W6FhVre9IgBcUNyAG/W3NXv193T59/fKhGpje2XccoFV6pabot/PGa+eBcn37mTVyjs3oEXsobkCMOVxRre8tXKdRman60pQBvuMAp+X8QT31jSuG6flVe/SX93f5jgO0O4obEGN++uIGHa6o1i9njlVCPP8EIPJ87ZJBmja8l37ywnqtLDzsOw7QrvhXG4ghb20uUV5BQF+9ZJBG9E31HQc4I3Fxpl/PGateXVJ0x2PLdai82nckoN1Q3IAYUV5Vq28/s0aD0jvpzmmDfccBzkq3jkl64JYJKimr0t1Pr1R9Pe93Q2yguAEx4lcvb9KeI8f1y1ljlJIY7zsOcNbGZHXT964boSWbSvTAm9t8xwHaBcUNiAH5O0v1p6U79fnz+2tivx6+4wBt5pZzc3T9uL76r1c26b2tB3zHAUKO4gZEucqaOn1rwWr17dpB3/z0MN9xgDZlZvr5jNEamN5Z//zkChUfrfQdCQgpihsQ5X73+lZtKynXL24crU7JCb7jAG2uU3KCHvjMBJVX1emux1fwfjdENYobEMXW7Tmi/31zm2ZNzNLFQ9N9xwFCZkjvLvrBdSP04c5Svbqh2HccIGQobkCUqq2r17cWrFa3jkn67jXn+I4DhNysiVnq17Ojfvf6VnZVQNSiuAFR6g9v79DaoqP6yfUj1a1jku84QMglxMfp9ksHaU3REb25ucR3HCAkQlrczOxKM9tkZlvN7J6T3H+3ma03s9VmttjM+gWPTzWzlU1ulWZ2Q/C+P5rZjib3jQvl1wBEou0lx3Tfa5t11ag+ump0hu84QLuZMT5Lfbum6LecdUOUCllxM7N4Sb+XdJWkEZLmmdmIEx62QlKuc26MpDxJv5Qk59wS59w459w4SdMkVUh6pcnzvtl4v3NuZai+BiAS1dc73bNgjVIS4vSj60f6jgO0q6SEOH310kEq2HVI728v9R0HaHOhPOM2WdJW59x251y1pCclXd/0AcGCVhH89H1JWSd5nVmS/tbkcQBa8NiHu/XhzlJ979oR6tUlxXccoN3Nyc1Wepdk/fb1Lb6jAG0ulMUtU1Jhk88DwWPNuU3S305y/CZJT5xw7GfBy6v3mVny2cUEokfR4eO696UNumhImmZNPNn/DwKiX0pivL5y8UC9t+2gCnZx1g3RJSwWJ5jZLZJyJf3qhOMZkkZLernJ4W9LGi5pkqQekr7VzGt+2czyzSy/pIQ3qSL6Oef0nWfXyEn6+YzRMjPfkQBvbj43R907Jup3r2/1HQVoU6EsbkWSspt8nhU89jFmdpmk70ia7pyrOuHuOZKedc7VNB5wzu11DaokPaqGS7Kf4Jx7yDmX65zLTU9nfhWi33Mri/TGphJ989PDlN2jo+84gFcdkxL0pYsGasmmEq0tOuI7DtBmQlnclkkaYmYDzCxJDZc8FzV9gJmNl/SgGkrb/pO8xjydcJk0eBZO1nA64QZJa9s+OhBZDhyr0o+eX6+J/brrc+f39x0HCAufPb+fuqQkcNYNUSVkxc05VyvpTjVc5twg6Wnn3Doz+7GZTQ8+7FeSOkuaHxzt8VGxM7P+ajhj9+YJL/2Yma2RtEZSmqSfhuprACLFDxetU0VVnf5j5mjFx3GJFJCk1JRE3XpBf/193T5tLi7zHQdoEyHduNA595Kkl0449v0mH1/WwnN36iSLGZxz09owIhDxXlm3Ty+s3qt/vWKoBvfq4jsOEFZuvXCAHn5nh373+lb9Zt5433GAsxYWixMAnJkjx2v03efWanifLvrKJYN8xwHCTvdOSbrl/H56YfUebS855jsOcNYobkAE+8VLG3TgWJV+NWusEuP5cQZO5ktTBioxPk4PvLHNdxTgrPEvPRCh3t16QE8uK9Q/XTxQo7O6+o4DhK30LsmaNzlHz64oUmEps9wR2ShuQASqqK7Vt59ZowFpnfT1y4b6jgOEva9cMlBm0oNvcdYNkY3iBkSgX7+yWbtLK3TvjaOVkhjvOw4Q9jK6dtCsidl6ellAxUcrfccBzhjFDYgwK3Yf0iPv7tAt5+Xo3IE9fccBIsbtlw5SnXN66K3tvqMAZ4ziBkSQqto6fWvBavVOTdG3rhzuOw4QUbJ7dNQN4zL12Ae7dODYiRv1AJGB4gZEkPuXbNPm4mP62YxR6pKS6DsOEHFunzpIVbX1evidHb6jAGeE4gZEiI37jur+N7bqhnF9NW14b99xgIg0KL2zrhmdoT+/t1OHK6p9xwFOG8UNiAB19U7fylut1JREff+6kb7jABHtjqmDVV5dpz++t9N3FOC0UdyACPDouzu0KnBEP5w+Uj06JfmOA0S0czJSdfmI3nr03Z0qq6zxHQc4LRQ3IMztPlih/3xlky47p7euHZPhOw4QFe6aNlhHjtfor+/v9h0FOC0UNyDMPfDmVjkn/fSGUTIz33GAqDAmq5suGZqu/3t7uyqqa33HAVqN4gaEsdLyaj2zvEg3TshSn64pvuMAUeWuaYN1sLxaT3xY6DsK0GoUNyCMPf7BLlXV1uuLF/b3HQWIOrn9e+i8gT300FvbVFlT5zsO0CoUNyBMVdfW689Ld+nioeka0ruL7zhAVLpr2hAVH61SXkHAdxSgVShuQJh6cc0e7S+r4mwbEEIXDOqp8Tnd9MAb21RTV+87DnBKFDcgDDnn9PA7OzS4V2ddMjTddxwgapmZ/nnaEBUdPq7nVhT5jgOcEsUNCEPLdh7S2qKj+uKFA1hJCoTYpcPSNbJvqu5/Y5vq6p3vOECLKG5AGHr4ne3q1jFRM8Zn+o4CRD0z013TBmvHgXK9sHqP7zhAiyhuQJjZfbBCr6wv1mfOzVGHpHjfcYCYcMWIPhrSq7N+v2Sr6jnrhjBGcQPCzB/f26l4M332vP6+owAxIy7OdOe0wdpcfEyvrC/2HQdoFsUNCCNllTV6Or9Q147JYOAu0M6uGZ2h/j076ndLtsg5zrohPFHcgDDydH5Ax6pq9cUpA3xHAWJOQnycbp86WGuLjuqNzSW+4wAnRXEDwkRdvdMf39uhSf27a0xWN99xgJg0Y3ymMrt10G8Xc9YN4YniBoSJV9fvU2Hpcd3G2TbAm8T4OH310kFavvuwlm476DsO8AkUNyBMPPLOTmV176DLR/TxHQWIabMnZqlXl2T99vWtvqMAn0BxA8LAmsARfbizVF+4oL/i4xi4C/iUkhivL188UEu3H1T+zlLfcYCPobgBYeCRd3eoU1K85kzK9h0FgKSbz81Rj05J+t0SzrohvFDcAM+Kj1bq+VV7NGdStlJTEn3HASCpY1KCvnTRAL2xqURrAkd8xwE+QnEDPPvL0l2qc063XsCiBCCcfPa8fkpNSdBvX9/iOwrwEYob4FFlTZ0e+2CXLj+nt3J6dvQdB0ATXVISdeuFA/TK+mJt3HfUdxxAEsUN8OqZ5UU6VFHDCBAgTN16YX91SorX75ds8x0FkERxA7xxzumRd3doZN9UTR7Qw3ccACfRrWOSPnt+f72weo+2lRzzHQeguAG+vLXlgLbuP6bbpgyQGSNAgHD1pYsGKDkhTg+8wVk3+EdxAzx55J0dSu+SrGvH9PUdBUAL0jona97kHD27okiFpRW+4yDGUdwAD7buL9Obm0v0ufP6KSmBH0Mg3H3l4kGKN9MDb3LWDX7xGwPw4OF3dio5IU43n5vjOwqAVujTNUWzc7OUlx/Q3iPHfcdBDKO4Ae3sUHm1nlke0IzxmerZOdl3HACt9NVLBqnOOT301nbfURDDKG5AO3v8w92qqq3XFxkBAkSU7B4dNWN8pp74cLdKyqp8x0GMorgB7ai6tl5/XrpTFw1J09DeXXzHAXCabr90kKpq6/XwOzt8R0GMorgB7eilNXtVfLSKs21AhBqY3lnXjumrvyzdqcMV1b7jIAZR3IB24pzTw+/s0KD0TrpkSLrvOADO0J1TB6u8uk6PvrvTdxTEIIob0E7ydx3SmqIjuvXCAYqLY+AuEKmG9emiT4/srUff3aGyyhrfcRBjKG5AO3n47R3q2iFRMydk+Y4C4CzdOXWIjlbW6s9Ld/mOghhDcQPaQWFphV5Zv083n5ujDknxvuMAOEujs7rq0mHpevidHaqorvUdBzGE4ga0gz++t1NxZvrc+f18RwHQRu6aNlil5dV6/IPdvqMghlDcgBArq6zRU8sKdfXoDGV07eA7DoA2MrFfD50/sKceemu7KmvqfMdBjKC4ASE2Pz+gY1W1uo0RIEDUuetTg7W/rErzCwK+oyBGUNyAEKqrd3r0vR3K7dddY7O7+Y4DoI2dP7CnJvbrrv99Y5uqa+t9x0EMoLgBIfTahmIVlh5n4C4QpcxMd04brKLDx/XciiLfcRADKG5ACD38zg5lduugK0b09h0FQIhcOjRdozJTdf8bW1Vbx1k3hBbFDQiRtUVH9OGOUn3hgv5KiOdHDYhWZqY7pw7RzoMVenHNXt9xEOX4bQKEyCPv7FCnpHjNnZztOwqAELtiRG8N691Fv3t9q+rrne84iGIUNyAE9h+t1POr92h2brZSUxJ9xwEQYnFxpjumDdaW/cf0yvp9vuMgilHcgBD4y/u7VFvvdOuF/X1HAdBOrhmdoQFpnfTb17fKOc66ITQobkAbq6yp02Mf7NZl5/RWv56dfMcB0E7i40y3XzpI6/Yc1ZJN+33HQZSiuAFt7LkVRSotr9YXL2QECBBrbhifqcxuHfSbxZx1Q2hQ3IA25JzTI+/u0IiMVJ03sIfvOADaWWJ8nL526SCtLDys97Yd9B0HUYjiBrSht7cc0ObiY/rilAEyM99xAHgwa2KWeqcm67evb/EdBVGI4ga0oUfe3aG0zsm6bmyG7ygAPElJjNdXLh6k97eXatnOUt9xEGUobkAb2bq/TG9sKtHnzu+n5IR433EAeDRvco56dkrSb1/f6jsKogzFDWgjj767U0kJcbr53BzfUQB41iEpXl+6aKDe2lyiVYWHfcdBFKG4AW3gUHm1FiwPaMa4TKV1TvYdB0AYuOW8HHXtkKjfLeGsG9pOSIubmV1pZpvMbKuZ3XOS++82s/VmttrMFptZvyb31ZnZyuBtUZPjA8zsg+BrPmVmSaH8GoDWeGLZblXW1OvWKf19RwEQJrqkJOrWC/vr1fXF2rD3qO84iBIhK25mFi/p95KukjRC0jwzG3HCw1ZIynXOjZGUJ+mXTe477pwbF7xNb3L8PyTd55wbLOmQpNtC9TUArVFTV68/v7dLUwanaXifVN9xAISRWy8YoM7JCfo9Z93QRkJ5xm2ypK3Oue3OuWpJT0q6vukDnHNLnHMVwU/fl5TV0gtaw3yFaWooeZL0J0k3tGVo4HS9tGav9h2t1G1TGLgL4OO6dkzU587vpxfX7NW2kmO+4yAKhLK4ZUoqbPJ5IHisObdJ+luTz1PMLN/M3jezG4LHeko67JyrbeVrAiHlnNPD7+zQwPROumRouu84AMLQbVMGKDkhjrNuaBNhsTjBzG6RlCvpV00O93PO5Uq6WdJ/m9mg03zNLweLX35JSUkbpgX+oWDXIa0OHNGtFw5QXBwDdwF8Us/OyfrMuf20cOUe7T5YceonAC0IZXErkpTd5POs4LGPMbPLJH1H0nTnXFXjcedcUfDP7ZLekDRe0kFJ3cwsoaXXDD7vIedcrnMuNz2dMyEIjUfe3aGuHRI1cwInfgE078sXD1S8mR54c5vvKIhwoSxuyyQNCa4CTZJ0k6RFTR9gZuMlPaiG0ra/yfHuZpYc/DhN0oWS1ruGHXuXSJoVfOjnJS0M4dcANKuwtEJ/X7tP8ybnqGNSwqmfACBm9U5N0ZxJWcorKNTeI8d9x0EEC1lxC74P7U5JL0vaIOlp59w6M/uxmTWuEv2VpM6S5p8w9uMcSflmtkoNRe1e59z64H3fknS3mW1Vw3veHg7V1wC05E/v7ZSZ6XPn9zv1gwHEvK9eMkjOSQ++ud13FESwkJ4mcM69JOmlE459v8nHlzXzvPckjW7mvu1qWLEKeHOsqlZPLSvU1aMz1LdbB99xAESArO4ddeOETD3x4W7dMXWw0rswrBunLywWJwCRZn5+ocqqahkBAuC0fO3Swaqpq9f/vc1ZN5wZihtwmurqnf743k5NyOmmcdndfMcBEEEGpHXSdWP76i/v79Kh8mrfcRCBKG7AaVq8oVi7DlbotikDfUcBEIHumDpYFdV1evTdHb6jIAJR3IDT9PA7O5TZrYM+PbK37ygAItDQ3l105cg+evS9nTpaWeM7DiIMxQ04DWuLjuiDHaX6/AX9lBDPjw+AM3PntMEqq6zVX5bu8h0FEYbfPMBpeOTdHeqYFK+5k3J8RwEQwUZldtXUYen6v7e3q6K69tRPAIIobkAr7S+r1POr9mj2xCx17ZDoOw6ACHfntCE6VFGjxz/Y7TsKIgjFDWilvy7dpdp6py9cyAgQAGdvYr/uunBwTz341nZV1tT5joMIQXEDWqGypk5//WC3PjW8lwakdfIdB0CUuHPqEJWUVenp/ELfURAhKG5AKyxcWaTS8mp9kYG7ANrQeQN7KLdfd/3vG9tUXVvvOw4iAMUNOAXnnB5+Z4eG9+mi8wf29B0HQBQxM905bbD2HKnUsysCvuMgAlDcgFN4d+tBbS4+ptumDJCZ+Y4DIMpcMjRdY7K66v43tqm2jrNuaBnFDTiFh9/ZrrTOSZo+rq/vKACikJnpzqmDtetghV5Yvdd3HIQ5ihvQgm0lx7RkU4luOa+fkhPifccBEKUuO6e3hvfpot8t2ar6euc7DsIYxQ1owaPv7lBSfJxuOa+f7ygAolhcnOmOqYO1df8x/X3dPt9xEMYobkAzDldUa0FBka4f11dpnZN9xwEQ5a4enaGBaZ3029e3yjnOuuHkKG5AM574sFDHa+p020WMAAEQevFxptunDtaGvUf1+sb9vuMgTFHcgJOoqavXn97bqQsH99TwPqm+4wCIEdeP66us7h0464ZmUdyAk/jb2n3ad7RSX2R7KwDtKDE+TrdfOlgrCw/r3a0HfcdBGEpo7g4zu7sVzy93zj3YhnkA7xoH7g5I66Spw3r5jgMgxsycmKnfLN6i+9/YqilD0nzHQZhp6YzbNyV1ltSlhds3Qh0QaG/Ldx/WqsLDuvXC/oqLY+AugPaVnBCvz57fT+9tO6jtJcd8x0GYafaMm6S/OOd+3NKTzYzdthF1Hnlnh1JTEjRzQpbvKABi1OzcLN336mY9uaxQ/371Ob7jIIy0dMbt16d6snPu39owC+Bd4FCF/rZ2r+ZNzlGn5Jb+fw0AhE6vLim67JzeyisIqKq2zncchJGWittKM3vNzG4zs27tFQjw6c9Ld8nM9PkL+vuOAiDG3XxujkrLq/XKumLfURBGWipumZJ+JWmKpE1mttDMbjKzDu0TDWhf5VW1euLD3bpqVB/17cZ/5gD8mjI4TVndO+jxD3b7joIw0mxxc87VOededs7dKilb0iOSrpe0w8wea6+AQHvJKwiorLJWX5zCCBAA/sXFmeZNztHS7SxSwD+0ao6bc65a0npJGyQdlcQ7JRFV6uudHn13h8bndNOEnO6+4wCApIZFCglxpqeWFfqOgjDRYnEzs2wz+6aZLZf0QvDx051zE9olHdBOFm/cr50HKxi4CyCsNC5SmM8iBQQ1W9zM7D1J70jqJemfnHPDnHM/dM5tbLd0QDt55J0d6ts1RVeN6uM7CgB8zDwWKaCJls643SOpv3Pum865gvYKBLS3dXuOaOn2g/r8Bf2VEM8ucADCy0XBRQpPfMgiBbS8OOEt55wzs6FmttjM1kqSmY0xs++2X0QgtB59d6c6JMbrpkk5vqMAwCc0LlJ4b9tB7ThQ7jsOPGvN6YU/SPq2pBpJcs6tlnRTKEMB7WV/WaUWrdyj2blZ6tox0XccADip2RMbFik8yVm3mNea4tbROffhCcdqQxEGaG9/fX+3quvq9QUG7gIIY71SWaSABq0pbgfMbJAkJ0lmNkvS3pCmAtpBZU2dHnt/lz41vJcGpnf2HQcAWsQiBUitK253SHpQ0nAzK5L0L5K+GspQQHtYtHKPDpZXM3AXQERgkQKklseBnG9m5pzb7py7TFK6pOHOuSnOuV3tFxFoe845PfLuDg3v00UXDOrpOw4AnBKLFCC1fMbtc5IKzOxJM/uCpC7OubL2iQWE1nvbDmrjvjJ9ccoAmZnvOADQKrMnZimeRQoxraVxIF8L7pDwQ0ndJf3RzJaa2c/N7GIzi2+vkEBbe+SdHUrrnKTpY/v6jgIArdawSKEXixRi2Cnf4+ac2+icu885d6WkaWrYTWG2pA9CHQ4Ihe0lx7R443595tx+Sknk/38AiCw3n9uPRQox7LTGxDvnjjvnXpL0bedcbogyASH16Ls7lRQfp1vO6+c7CgCctosGpymzG4sUYtWZ7u+zvk1TAO3kSEWN8goCmj6ur9K7JPuOAwCnrWGRQjaLFGJUQnN3mNndzd0liaFXiEhPLNut4zV1+uKFjAABELnm5Gbrvte26MkPd+vbV5/jOw7aUUtn3H6uhkUJXU64dT7F84CwVFNXrz+9t1PnD+ypEX1TfccBgDPGIoXY1ewZN0nLJT3nnCs48Q4z+1LoIgGh8fe1+7T3SKV+cv0o31EA4KzNm5yjl9cV69X1xbp2DCvkY0VLZ85uldTcoF0WJiDiPPzODvXv2VHThvfyHQUAztrFQ9KV2a2DHv+ARQqxpKU5bpuccweauY81yIgoy3cf0srCw7r1wgGKi2PgLoDIxyKF2NTSllc/PNWTW/MYIBw8/M4OdUlJ0KyJWb6jAECbmZ2b3bCTwjLOusWKlt7j9iUzO9rC/SbpJjXsrACEraLDx/X3tft025QB6pTc0n/yABBZegcXKeTlB/SNy4cpKYG1g9Gupe/wH/TJFaUnri79Q6gDAmfrz+/tlCR9/oL+XnMAQCjMm5yjg+XVemX9Pt9R0A6aPf3gnPtRewYBQqG8qlaPf7hbV47so8xuHXzHAYA2d1FwkcITH+5mdWkM4JwqotqC5QGVVdbqi1MYuAsgOsUHFym8u/WgdrJIIepR3BC16uudHn13p8Zmd9OEnG6+4wBAyDQuUniCRQpRj+KGqLVk037tOFCu26YMkBkjQABEr96pKfrU8IZFCtW19b7jIIROWdzM7JdmlmpmiWa22MxKzOyW9ggHnI2H39mhjK4pumpUH99RACDkbj6XRQqxoDVn3K5wzh2VdK2knZIGS/pmKEMBZ2vr/jK9t+2gPnt+PyXGc2IZQPRrukgB0as1v9EaV55eI2m+c+5ICPMAbWJ+QUDxcabZE7N9RwGAdhEfZ7ppEosUol1ritsLZrZR0kRJi80sXVJlaGMBZ662rl7PLi/S1GHpSu+S7DsOALSbOZNYpBDtTlncnHP3SLpAUq5zrkZShaTrQx0MOFNvbz2g/WVVbG8FIOawSCH6tWZxQkdJt0t6IHior6TcUIYCzkZeQUDdOyZq2vDevqMAQLubxyKFqNaaS6WPSqpWw1k3SSqS9NOQJQLOwpGKGr26rljXj8tkzz4AMeliFilEtdb8ZhvknPulpBpJcs5VqGGDeSDsLFq9R9V19VwmBRCzWKQQ3VpT3KrNrIMkJ0lmNkhSVUhTAWcoryCg4X26aGTfVN9RAMAbFilEr9YUtx9K+rukbDN7TNJiSd8KZSjgTGwpLtOqwsOaNTGLnRIAxDQWKUSv1qwqfUXSjZK+IOkJNawuXRLiXMBpy1seUEKc6Ybxmb6jAIB3jYsUXl1f7DsK2lBrVpUuds4ddM696Jx7wTl3wMwWt+bFzexKM9tkZlvN7J6T3H+3ma03s9XB7bT6BY+PM7OlZrYueN/cJs/5o5ntMLOVwdu40/h6EaUaZ7ddOqyX0jozuw0AGhcpPP7hLt9R0IaaLW5mlmJmPSSlmVl3M+sRvPWXdMpTGmYWL+n3kq6SNELSPDMbccLDVqjhDN4YSXmSfhk8XiHpc865kZKulPTfZtatyfO+6ZwbF7ytbM0XiujG7DYA+DgWKUSnls64fUVSgaThwT8bbwsl/a4Vrz1Z0lbn3HbnXLWkJ3XC4F7n3JLgKlVJel9SVvD4ZufcluDHeyTtl5Te2i8KsScvv3F2Wy/fUQAgbDQuUnhyWaHvKGgjzRY359z/OOcGSPpX59xA59yA4G2sc641xS1TUtP/UgJq+UzdbZL+duJBM5ssKUnStiaHfxa8hHqfmXFdLMYdrmh4Dwez2wDg4z5apFBQyCKFKNGaxQm/NbNRZjbHzD7XeGvLEGZ2ixp2Y/jVCcczJP1F0q3Oucb/4r6thrOAkyT1UDMrXM3sy2aWb2b5JSUlbRkXYeb5VcxuA4DmzDs3RweOsUghWrRmccIPJP02eJuqhvehTW/FaxdJym7yeVbw2Imvf5mk70ia7pyranI8VdKLkr7jnHu/8bhzbq9rUKWGXR0mn+wvd8495JzLdc7lpqdzlTWaMbsNAJrHTgrRpTXXlWZJ+pSkfc65WyWNldS1Fc9bJmmImQ0wsyRJN0la1PQBZjZe0oNqKG37mxxPkvSspD875/JOeE5G8E+TdIOkta3Igii1ubhMqwJHmN0GAM1oXKTwztYDLFKIAq0pbseDlylrg2fB9uvjZ9JOyjlXK+lOSS9L2iDpaefcOjP7sZk1nrH7laTOkuYHR3s0Frs5ki6W9IWTjP14zMzWSFojKU3smxrTFhQwuw0ATmV2LosUokVCKx6THxzF8Qc1rCo9Jmlpa17cOfeSpJdOOPb9Jh9f1szz/irpr83cN601fzeiX21dvZ5ZUaSpw5ndBgAt6dM1RdOCixTuvnwoC7kiWGsWJ9zunDvsnPtfSZdL+nzwking1dtbDqiE2W0A0Co3s0ghKrRq54TGj51zO51zq1u7cwIQSnkFAfXolKSpw5jdBgCnwiKF6BCynROAUPrH7La+nPIHgFaIjzPNDS5S2HWQRQqRKpQ7JwAhw+w2ADh9c4KLFJ74kEUKkSqUOycAIZNXENA5Gaka2bc1k2kAANLHFymwk0JkaulS6SQz6+Oc+23w88+Z2UIz+03wEirgRdPZbQCA03Pz5IZFCq9tYJFCJGrpUumDkqolycwulnSvpD9LOiLpodBHA04uLzi77fpxfX1HAYCIc/HQhkUKj3/AIoVI1FJxi3fOlQY/nivpIefcAufc9yQNDn004JNq6+r1zHJmtwHAmWKRQmRrsbiZWeOA3k9Jer3Jfa0Z3Au0ube2lOjAMWa3AcDZYJFC5GqpuD0h6U0zWyjpuKS3JcnMBqvhcinQ7vIKAurZKUnThjO7DQDOFIsUIldLq0p/Jukbkv4oaYpzzjV5zl2hjwZ83KHyar22fr+uH5epxHhmtwHA2WCRQmRq8ZKnc+79kxzbHLo4QPOeX83sNgBoK00XKVw9OsN3HLQSpy0QMfIKAhqRkaoRfVN9RwGAiMcihcjU0hw3luwhbGzaV6bVzG4DgDY1JzdbcSY9uYxFCpGipTNuSyXJzP7STlmAZi1Yzuw2AGhrDYsUemt+PosUIkVL73FLMrObJV1gZjeeeKdz7pnQxQL+oXF227ThvdST2W0A0KY+c26OXttQrNc2FPNetwjQUnH7qqTPSOom6boT7nOSKG5oF8xuA4DQuXhouvp2TdETH7JIIRI0W9ycc+9IesfM8p1zD7djJuBj5uc3zG6byuw2AGhzDYsUcnTfa5u162C5+vXs5DsSWtCaVaV/MbN/NrO84O0uM0sMeTJAwdltG4qZ3QYAITR3EosUIkVrfhPeL2li8M/7JU2Q9EAoQwGNFq3ao5o6x2VSAAghFilEjtYUt0nOuc87514P3m6VNCnUwQCJ2W0A0F5uPjebnRQiQGuKW52ZDWr8xMwGSqoLXSSgwcZ9R7Wm6Ihm53K2DQBC7ZKhvT5apIDw1Zri9k1JS8zsDTN7U9LratjDFAipBQUBJcabrh+X6TsKAES9xkUKb285oN0HK3zHQTNOWdycc4slDZH0z2rYXH6Yc25JqIMhttXU1evZFXs0bXgv9eiU5DsOAMSEOZOyFGfSE8s46xauWrVMzzlX5ZxbHbxVhToU8Nbmxtlt2b6jAEDMyOjagUUKYY75CghLeQUNs9suHZbuOwoAxJTGRQqLWaQQlihuCDuNs9tuGM/sNgBob42LFB5nkUJYOuVvRTN7xsyuMTN+g6JdMLsNAPxhkUJ4a+0A3pslbTGze81sWIgzIcblFQQ0sm+qzslgdhsA+MAihfDVmlWlrznnPqOGHRN2SnrNzN4zs1vZ+gptrXF2G2fbAMAfFimEr1Zd/jSznpK+IOlLklZI+h81FLlXQ5YMMSkvn9ltABAOWKQQnlrzHrdnJb0tqaOk65xz051zTznn7pLUOdQBETtq6ur13MoiZrcBQBhgkUJ4as0Ztz8450Y4537hnNsrSWaWLEnOudyQpkNMeXNTiQ4cq9ZsZrcBgHcsUghPrSluPz3JsaVtHQTIKwgorXOSLmF2GwCEhcZFCk+ySCFsNFvczKyPmU2U1MHMxpvZhODtUjVcNgXaTGl5tRZvLNYN45jdBgDhomGRQi89nR9QTR2LFMJBQgv3fVoNCxKyJP26yfEySf8ewkyIQYtWFqmmzmkmq0kBIKzcfG6OXtuQr9fWF+uq0Rm+48S8Zoubc+5Pkv5kZjOdcwvaMRNiUN7ygEZlMrsNAMLNJUN7KSO4SIHi5l+zxc3MbnHO/VVSfzO7+8T7nXO/PsnTgNO2Ye9RrS06qh9eN8J3FADACRoWKWTrv1/bot0HK5TTk3dL+dTSm4k6Bf/sLKnLSW5Am1hQ0DC7bTqz2wAgLM2dlM0ihTDR0qXSB4N//qj94iDWNM5u+9Tw3sxuA4Aw1XSRwtcvH8oiMo9aM4D3l2aWamaJZrbYzErM7Jb2CIfo1zi7jS2uACC8zZucowPHqvTaenZS8Kk1lfkK59xRSdeqYa/SwZK+GcpQiB3zCwqZ3QYAEeDSYf9YpAB/WlPcGi+nXiNpvnPuSAjzIIYcPFalxRv2M7sNACJA4yKFt7ccUGEpOyn40prfli+Y2UZJEyUtNrN0SZWhjYVYsGjVHtXWO83K5TIpAESCxkUKT3DWzZtTFjfn3D2SLpCU65yrkVQu6fpQB0P0yysIaHRmVw3vw+w2AIgE7KTgX2uvTw2XNNfMPidplqQrQhcJsWD9nqNat+coixIAIMI0LlJYvIFFCj60tOWVJMnM/iJpkKSVkuqCh52kP4cuFqLdguXB2W1j+/qOAgA4DZcMTVdG1xQ99sFuXTmKnRTa2ymLm6RcSSOccy7UYRAbaurq9dyKIl12Tm91Z3YbAESUhPi4j3ZSKCytUHYPdlJoT625VLpWUp9QB0HseGNTiQ6WM7sNACLVnFwWKfjSmuKWJmm9mb1sZosab6EOhuiVV1CotM7Jungos9sAIBL17dZBU4exSMGH1lwq/WGoQyB2NM5uu/XC/sxuA4AIdvO5OVq8MV+LNxTzXrd21JpxIG+qYceExODHyyQtD3EuRKnG2W0zuUwKABGt6SIFtJ/W7FX6T5LyJD0YPJQp6bkQZkIUY3YbAESHhPg4zcllJ4X21pprVXdIulDSUUlyzm2R1CuUoRCdmN0GANGlcSeFJ5dx1q29tKa4VTnnqhs/MbMENcxxA05LXkFASfFxzG4DgCjBIoX215ri9qaZ/bukDmZ2uaT5kp4PbSxEm+raej23skiXjejF7DYAiCLzJueopIydFNpLa4rbPZJKJK2R9BVJL0n6bihDIfq8sWm/SpndBgBR59JhDYsUHv+w0HeUmHDKcSDOuXoze07Sc865ktBHQjTKKwg0zG4bwuw2AIgmjYsUfvM6Oym0h2bPuFmDH5rZAUmbJG0ysxIz+377xUM0OHisSq9v3K8bJ2QqgdltABB15k7KlolFCu2hpd+iX1fDatJJzrkezrkeks6VdKGZfb1d0iEqLFwZnN02gcukABCNWKTQfloqbp+VNM85t6PxgHNuu6RbJH0u1MEQPfIKAhqT1VXD+nTxHQUAECIsUmgfLRW3ROfcgRMPBt/nlhi6SIgm6/Yc0fq9zG4DgGh36bB09UllkUKotVTcqs/wPuAjCwqKlBQfp+vGMLsNAKJZQnyc5k7K1ttbSthJIYRaKm5jzezoSW5lkka35sXN7Eoz22RmW83snpPcf7eZrTez1Wa22Mz6Nbnv82a2JXj7fJPjE81sTfA1f2NmdjpfMNoPs9sAILbMYZFCyDVb3Jxz8c651JPcujjnTnmp1MziJf1e0lWSRkiaZ2YjTnjYCkm5zrkxatgP9ZfB5/aQ9AM1LIaYLOkHZtY9+JwHJP2TpCHB25Wn8fWiHTG7DQBiS2a3DrqURQohFcrZDJMlbXXObQ9umfWkpOubPsA5t8Q513g+9X1Jjb/hPy3pVedcqXPukKRXJV1pZhmSUp1z7zvnnKQ/S7ohhF8DzkJeQUDpXZjdBgCx5OaPFins9x0lKoWyuGVKavoOxUDwWHNuk/S3Uzw3M/hxa18TnhxonN02ntltABBL/rFIgculoRAWv1HN7BZJuZJ+1Yav+WUzyzez/JISNnxobx/NbuMyKQDElIT4OM1hkULIhLK4FUnKbvJ5VvDYx5jZZZK+I2m6c67qFM8t0j8upzb7mpLknHvIOZfrnMtNT+dSXXvLKwhobFZXDe3N7DYAiDXspBA6oSxuyyQNMbMBZpYk6SZJi5o+wMzGS3pQDaWt6cXwlyVdYWbdg4sSrpD0snNur6SjZnZecDXp5yQtDOHXgDOwbs8RbWB2GwDELBYphE7IiptzrlbSnWooYRskPe2cW2dmPzaz6cGH/UpSZ0nzzWylmS0KPrdU0k/UUP6WSfpx8Jgk3S7p/yRtlbRN/3hfHMJEXkGgYXbbWGa3AUCsYpFCaCSE8sWdcy9JeumEY99v8vFlLTz3EUmPnOR4vqRRbRgTbai6tl4LV+7R5SN6q1tHZrcBQKxqukjhylF9fMeJGmGxOAHRYwmz2wAAYpFCqFDc0KYaZ7ddNCTNdxQAgGc3BRcpPLWM/UvbCsUNbebAsSotYXYbACCob7cOunhouvIKAqplkUKb4Lcr2gyz2wAAJ5qbm619Ryv11hZmqrYFihvaDLPbAAAn+tQ5vdWzUxKXS9sIxQ1tgtltAICTSUqI040TMrV4w36VlFWd+gloEcUNbWJ+fsPstulj2ToWAPBxcydlq7be6ZnlgVM/GC2iuOGsNcxuK9LlI3ura8dE33EAAGFmcK8umtivu55aVijnnO84EY3ihrP2+sb9OlRRw2VSAECz5k7K1vYD5crfdch3lIhGccNZyysIqFeXZF00mNltAICTu2Z0hjolxevJD1mkcDYobjgrJWVVWrJpv2ZMYHYbAKB5nZITNH1cX720Zq+OVtb4jhOx+E2Ls7JwZZHq6p1mTeAyKQCgZXNys3W8pk7Pr9rjO0rEorjhjDnnGma3ZXfTEGa3AQBOYVx2Nw3r3UVPM9PtjFHccMbW7TmqjfvKWJQAAGgVM9OcSdlaFWiY/YnTR3HDGcsrCM5uG9PXdxQAQISYMT5TSfFx7KRwhihuOCPMbgMAnIkenZJ0+cjeenZFkSpr6nzHiTgUN5wRZrcBAM7UTZOydeR4jV5ZX+w7SsShuOGM5BUE1Ds1WRcPSfcdBQAQYS4clKbMbh301LLdvqNEHIobTttHs9vGZyk+znzHAQBEmLg405zcbL279aAKSyt8x4koFDecto9mt01kQ3kAwJmZlZslM+npfBYpnA6KG06Lc07z8wMal91Ng3sxuw0AcGYyu3XQxUPSNT8/oLp6Np5vLYobTsu6PUe1qZjZbQCAszd3Urb2Ha3UW5tLfEeJGBQ3nJa8goCSEuJ0HbPbAABn6bJzeqtHpyRmup0Gihtaraq2Ts+tLNIVI5jdBgA4e0kJcbpxfKZe21CskrIq33EiAsUNrbZk434dZnYbAKANzZ2Urdp6p2dXBHxHiQgUN7Ra4+y2i5jdBgBoI0N6d9GEnG56clmhnGORwqlQ3NAqDbPbSpjdBgBoczdNytH2knIV7DrkO0rYo7ihVf4xu43LpACAtnXNmAx1SorXkyxSOCWKG06pcXbb+JxuGtyrs+84AIAo0yk5QdeN7asXV+9VWWWN7zhhjeKGU2J2GwAg1OZMytbxmjo9v2qv7yhhjeKGU5qfX6ikhDhdy+w2AECIjM/upqG9O+sptsBqEcUNLaqqrdPCVXv06ZF91LUDs9sAAKFh1rDx/KrCw9q476jvOGGL4oYWvb6B2W0AgPZx44QsJcYbOym0gOKGFjXObpsyOM13FABAlOvRKUlXjOijZ1cUqaq2znecsERxQ7P2l1Xqjc0lunECs9sAAO1j7qRsHa6o0Svrin1HCUsUNzRr4Yo9qqt3mjmBy6QAgPYxZXCaMrt10NMsUjgpihtOyjmnvAJmtwEA2ldcnGl2bpbe3nJAhaUVvuOEHYobTmptEbPbAAB+zM7Nlpk0v4CN509EccNJ5RUwuw0A4Edmtw66aEi65ucXqq6ejeeborjhE5jdBgDwbW5utvYeqdTbW0p8RwkrFDd8QuPsttlcJgUAeHLZiF7q0SmJmW4noLjhE/IKAuqTmqILmd0GAPAkOSFeM8Zn6rUNxTpwrMp3nLBBccPH/GN2Wyaz2wAAXs2dlK2aOqdnlxf5jhI2KG74mOdWFDXMbuMyKQDAs6G9u2h8Tjc9lV8o51ikIFHc0ETT2W2D0pndBgDw76ZJ2dq6/5iW7z7kO0pYoLjhI6sDR7S5+JhmT8z2HQUAAEnSNWP6qmNSPIsUgihu+EheQUDJCXG6dmyG7ygAAEiSOicn6LoxffXC6r06VlXrO453FDdIkipr6rRwZZGuHNVHqSnMbgMAhI85k7JVUV2nF1bt8R3FO4obJEmvbSjW0cpaLpMCAMLOhJxuGtKrs57kcinFDQ3m5wfUt2uKzh/U03cUAAA+xsw0d1K2VhYe1qZ9Zb7jeEVxg/YFtxSZOTGL2W0AgLA0Y3ymEuMt5hcpUNygZ1YEVO+kWcxuAwCEqZ6dk3X5iN56dkVAVbV1vuN4Q3GLcc455eUHNLl/D/Xr2cl3HAAAmjV3Uo4OVdTo1fXFvqN4Q3GLcct3H9b2A+WalcvZNgBAeJsyOE19u6bE9OVSiluMyysoVIfEeF09mtltAIDwFh9nmp2brXe2HlDgUIXvOF5Q3GLY8eo6Pb9qr64enaHOyQm+4wAAcEqzg1eI5ucHPCfxg+IWw15et0/Hqmo/+iEAACDcZXXvqCmD0zQ/v1B19bG38TzFLYbNLyhUdo8Omty/h+8oAAC02txJ2dpzpFLvbD3gO0q7o7jFqMChCr237aBmTchWHLPbAAAR5PIRvdW9Y6KeWrbbd5R2R3GLUc8sL5Jz0o0TMn1HAQDgtCQnxGvG+Cy9ur5YB49V+Y7TrihuMai+3imvIKALBvVUdo+OvuMAAHDa5k7KVk2d07MrinxHaVcUtxi0bGepdpdWsCgBABCxhvXponHZ3fTUskI5FzuLFChuMWh+QUCdkxN05UhmtwEAItdNk7K1Zf8xLd992HeUdkNxizHlVbV6ac1eXTsmQx2S4n3HAQDgjF07tq86JsXr6RjaSSGkxc3MrjSzTWa21czuOcn9F5vZcjOrNbNZTY5PNbOVTW6VZnZD8L4/mtmOJveNC+XXEG1eWrNXFdV1bCgPAIh4nZMTdO2YDD2/eo+OVdX6jtMuQlbczCxe0u8lXSVphKR5ZjbihIftlvQFSY83PeicW+KcG+ecGydpmqQKSa80ecg3G+93zq0MzVcQneYXBDQwrZMm9uvuOwoAAGdt7qRsVVTX6cXVe3xHaRehPOM2WdJW59x251y1pCclXd/0Ac65nc651ZLqW3idWZL+5pyLzU3J2tCug+X6cEepZk7Mkhmz2wAAkW9CTncN7tVZT8bI5dJQFrdMSU3/VwwEj52umyQ9ccKxn5nZajO7z8ySzzRgrFlQEFCcMbsNABA9zExzc7O1YvdhbS4u8x0n5MJ6cYKZZUgaLenlJoe/LWm4pEmSekj6VjPP/bKZ5ZtZfklJScizhrv6eqcFy4s0ZUi6Mrp28B0HAIA2M2NCphLjTU/FwFm3UBa3IknZTT7PCh47HXMkPeucq2k84Jzb6xpUSXpUDZdkP8E595BzLtc5l5uenn6af230Wbr9oIoOH2dRAgAg6qR1TtZl5/TWsyuKVFVb5ztOSIWyuC2TNMTMBphZkhoueS46zdeYpxMukwbPwska3qR1g6S1Zx81+s3PL1RqSoKuGNHbdxQAANrc3EnZKi2v1mvr9/uOElIhK27OuVpJd6rhMucGSU8759aZ2Y/NbLokmdkkMwtImi3pQTNb1/h8M+uvhjN2b57w0o+Z2RpJaySlSfppqL6GaHG0skZ/X7dP08f1VUois9sAANHnoiHp6ts1RU/lR/fl0oRQvrhz7iVJL51w7PtNPl6mhkuoJ3vuTp1kMYNzblrbpox+L67eq8qaes2amH3qBwMAEIHi40yzcrP129e3KHCoQlndo3Mv7rBenIC2kVcQ0JBenTU2q6vvKAAAhMzs4Pu48woCnpOEDsUtym0rOaaCXYc0i9ltAIAol92jo6YMTtP8/IDq6qNz43mKW5TLKwgoPs40g9ltAIAYMCc3W0WHj+vdrQd8RwkJilsUq6t3emZ5QJcOTVevLim+4wAAEHJXjOytbh0To3amG8Utir29pUTFR6uY3QYAiBnJCfGaMT5Tr6zfp9Lyat9x2hzFLYrlFQTUvWOiPnUOs9sAALFj7qRs1dQ1XHWKNhS3KHWkokavrC/W9eMylZTAtxkAEDuG90nV2Oxuejq/UM5F1yIFfqNHqUWrilRdW89lUgBATLppUrY2Fx/TisLDvqO0KYpblMorCOicjFSNymR2GwAg9lw7JkMdEuP1dJQtUqC4RaHNxWVaFTjC2TYAQMzqkpKoa8dk6PlVe1ReVes7TpuhuEWhvIKAEuJMN4zr6zsKAADezJ2UrfLqOr24eq/vKG2G4hZlaurq9czyIk0b3ks9Oyf7jgMAgDcT+3XXoPROenLZbt9R2gzFLcq8ualEB45VaXYuG8oDAGKbmWnupGwt331YW4rLfMdpExS3KJNXEFBa5yRdOizddxQAALy7cUKWEuIsanZSoLhFkdLyai3eWKwbxmUqMZ5vLQAAaZ2Tddk5vfXMioYxWZGO3+5RZOHKItXUOc3KZTUpAACN5k7OVml5tV7bUOw7ylmjuEWR+fkBjc7squF9Un1HAQAgbFw8JF0ZXVOi4nIpxS1KrNtzROv3HtVszrYBAPAx8XGm2ROz9NaWEhUdPu47zlmhuEWJvIKAkuLjNH0ss9sAADjR7NxsOSfl5Uf2xvMUtyhQXVuvhSv36PIRvdWtY5LvOAAAhJ3sHh01ZXCans4vVH195G48T3GLAq9v3K/S8moWJQAA0II5k7JVdPi43t12wHeUM0ZxiwJ5BYXq1SVZFw1O8x0FAICwdcWI3urWMVFPRvAiBYpbhNtfVqklm0oaBgwyuw0AgGalJMbrhnGZenVdsQ6VV/uOc0b4TR/hFq7Yo7p6p1kTuUwKAMCpzJ2Ureq6ej27osh3lDNCcYtgzjnNLyjU+JxuGtyrs+84AACEvXMyUjU2q6ueWlYo5yJvkQLFLYKtKTqizcXHNHsiG8oDANBacyflaFNxmVYFjviOctoobhFsfn5AyQlxunZshu8oAABEjOvGZqhDYryeWrbbd5TTRnGLUJU1dVq4skhXjuqj1JRE33EAAIgYXVISdc2YDC1auUflVbW+45wWiluEem1DsY5W1nKZFACAMzB3UrbKq+v04pq9vqOcFopbhJqfH1Dfrik6f1BP31EAAIg4uf26a2B6Jz0dYTPdKG4RaN+RSr29pUQzJ2YpPs58xwEAIOKYmebmZit/1yFt3V/mO06rUdwi0DMrAqp30swJzG4DAOBM3TghSwlxpqcjaON5iluEcc4pLz+gyf17qH9aJ99xAACIWOldkvWpc3ppQUFA1bX1vuO0CsUtwizffVjbD5SzoTwAAG3gpkk5Olherdc3FvuO0ioUtwiTV1CoDonxuno0s9sAADhbFw9NV5/UlIjZeJ7iFkGOV9fphVV7dfXoDHVOTvAdBwCAiBcfZ5qdm6W3Npdoz+HjvuOcEsUtgry8bp/KqmrZUB4AgDY0e2K26p2UVxD+ixQobhFkfkGhsnt00LkDeviOAgBA1Mjp2VEXDu6pp/MLVV8f3hvPU9wiROBQhd7bdlCzJmQrjtltAAC0qTm52QocOq73th30HaVFFLcI8czyIjkn3Tgh03cUAACizqdH9lHXDol6Kj+8FylQ3CKAc055BQFdMKinsnt09B0HAICok5IYrxnjM/Xy2n06VF7tO06zKG4R4MMdpdpdWsGiBAAAQmjupGxV19XruZVFvqM0i+IWAeYXBNQ5OUFXjWJ2GwAAoXJORqrGZHXVU8sK5Vx4LlKguIW58qpavbRmr64dk6EOSfG+4wAAENXmTsrWxn1lWh044jvKSVHcwtxLa/aqorqOy6QAALSD68b2VUpiXNjupEBxC3N5BQENSOukif26+44CAEDUS01J1DWj++r5VXtUUV3rO84nUNzC2O6DFfpgR6lmTcySGbPbAABoD3MnZetYVa1eXL3Xd5RPoLiFsbyCQsUZs9sAAGhPk/p318C0Tno6DGe6UdzCVH2904LlRZoyJF0ZXTv4jgMAQMwwM82ZlK1lOw9p6/5jvuN8DMUtTC3dflBFh4+zKAEAAA9unJCphDjT/DA760ZxC1N5BQF1SUnQFSN6+44CAEDM6dUlRdOG99KC5QHV1NX7jvMRilsYOlpZo7+t3avpY/sqJZHZbQAA+HDT5GwdOFatxRv2+47yEYpbGHpx9V5V1tRrdm627ygAAMSsi4ekq3dqsp5attt3lI9Q3MJQXkFAQ3p11tisrr6jAAAQsxLi4zR7Yrbe3FyivUeO+44jieIWdraVHFPBrkPMbgMAIAzMyc1WvZPy8gO+o0iiuIWdBQUBxceZZoxndhsAAL7l9OyoCwb11NMFhaqv97/xPMUtjNTVOz2zvEiXDE1Xr9QU33EAAIAadlIoLD2updsP+o5CcQsnb28p0b6jlZrN7DYAAMLGp0f2UdcOiXoqDDaep7iFkbyCgLp3TNSnzmF2GwAA4SIlMV43jOurv6/bp8MV1V6zUNzCxJGKGr2yvljXj8tUUgLfFgAAwsncSTnq1SVZOw9WeM2R4PVvx0cWrSpSdW09W1wBABCGRvRN1VvfnKq4OL8THzi1EybyCgIa3qeLRvZN9R0FAACchO/SJlHcwsLm4jKtChzR7NxsZrcBAIBmUdzCQF5BQAlxphvG9fUdBQAAhLGQFjczu9LMNpnZVjO75yT3X2xmy82s1sxmnXBfnZmtDN4WNTk+wMw+CL7mU2aWFMqvIdRq6ur1zPIiTRveSz07J/uOAwAAwljIipuZxUv6vaSrJI2QNM/MRpzwsN2SviDp8ZO8xHHn3LjgbXqT4/8h6T7n3GBJhyTd1ubh29Gbm0p04FgVG8oDAIBTCuUZt8mStjrntjvnqiU9Ken6pg9wzu10zq2WVN+aF7SGN4BNk5QXPPQnSTe0WWIP8goCSuucpEuHpfuOAgAAwlwoi1umpKYjhgPBY62VYmb5Zva+md0QPNZT0mHnXO0ZvmZYKS2v1uKNxbphXKYS43m7IQAAaFk4z3Hr55wrMrOBkl43szWSjrT2yWb2ZUlflqScnJwQRTw7C1cWqabOaVYus9sAAMCphfI0T5Gkpm/cygoeaxXnXFHwz+2S3pA0XtJBSd3MrLFwNvuazrmHnHO5zrnc9PTwvAw5Pz+g0ZldNbwPs9sAAMCphbK4LZM0JLgKNEnSTZIWneI5kiQz625mycGP0yRdKGm9c85JWiKpcQXq5yUtbPPk7WDdniNav/eoZnO2DQAAtFLIilvwfWh3SnpZ0gZJTzvn1pnZj81suiSZ2SQzC0iaLelBM1sXfPo5kvLNbJUaitq9zrn1wfu+JeluM9uqhve8PRyqryGU8goCSoqP0/SxzG4DAACtE9L3uDnnXpL00gnHvt/k42VquNx54vPekzS6mdfcroYVqxGrurZeC1fu0eUjeqtbx4geQwcAANoRSxk9eH3jfpWWV7OhPAAAOC0UNw/yCgrVq0uyLhqS5jsKAACIIBS3dra/rFJLNpXoxglZSmB2GwAAOA00h3a2cMUe1dU7LpMCAIDTRnFrR845zS8o1Picbhrcq7PvOAAAIMJQ3NrRmqIj2lx8jLNtAADgjFDc2tH8/ICSE+J0HbPbAADAGaC4tZPKmjotXFmkK0f1UWpKou84AAAgAlHc2slrG4p1tLKWy6QAAOCMUdzayfz8gPp2TdEFg5jdBgAAzgzFrR3sO1Kpt7c0zG6LjzPfcQAAQISiuLWDZ1YEVO/EZVIAAHBWKG4h5pxTXn5Ak/v3UP+0Tr7jAACACEZxC7Hluw9r+4FyzrYBAICzRnELsbyCQnVIjNfVYzJ8RwEAABGO4hZCx6vr9MKqvbpqdB91Tk7wHQcAAEQ4ilsIvbxun8qqajV7YrbvKAAAIApQ3EJofkGhsnt00LkDeviOAgAAogDFLUQChyr03raDmjkhS3HMbgMAAG2A4hYizywvknPSzAmsJgUAAG2D4hYCzjnlFQR0/sCeyu7R0XccAAAQJShuIfDhjlLtLq3Q7FzOtgEAgLZDcQuB+QUBdU5O0JWj+viOAgAAogjFrY2VV9XqpTV7dc3oDHVMYnYbAABoOxS3NvbSmr2qqK7jMikAAGhzFLc2llcQ0IC0TprYr7vvKAAAIMpQ3NrQ7oMV+mBHqWZNzJIZs9sAAEDbori1obyCQplJN07I9B0FAABEIYpbG6mvd1qwvEhTBqcpo2sH33EAAEAUori1kaXbD6ro8HHNzmVDeQAAEBoUtzaSVxBQl5QEXTGit+8oAAAgSlHc2sDRyhr9be1eTR/bVymJ8b7jAACAKEVxawMvrt6rypp6LpMCAICQori1gdLyao3J6qqxWV19RwEAAFGMPZnawB1TB+trlwxidhsAAAgpzri1kbg4ShsAAAgtihsAAECEoLgBAABECIobAABAhKC4AQAARAiKGwAAQISguAEAAEQIihsAAECEoLgBAABECIobAABAhKC4AQAARAiKGwAAQISguAEAAEQIihsAAECEoLgBAABECIobAABAhKC4AQAARAiKGwAAQISguAEAAEQIihsAAECEoLgBAABECIobAABAhKC4AQAARAhzzvnOEHJmViJpV4j/mjRJB0L8d+D08X0JP3xPwhPfl/DD9yT8tNf3pJ9zLv1kd8REcWsPZpbvnMv1nQMfx/cl/PA9CU98X8IP35PwEw7fEy6VAgAARAiKGwAAQISguLWdh3wHwEnxfQk/fE/CE9+X8MP3JPx4/57wHjcAAIAIwRk3AACACEFxa0Nm9kMzKzKzlcHb1b4zxSozu9LMNpnZVjO7x3ceNDCznWa2Jvjzke87Tywys0fMbL+ZrW1yrIeZvWpmW4J/dveZMRY1833hd4pHZpZtZkvMbL2ZrTOz/xc87vXnheLW9u5zzo0L3l7yHSYWmVm8pN9LukrSCEnzzGyE31RoYmrw54MxB378UdKVJxy7R9Ji59wQSYuDn6N9/VGf/L5I/E7xqVbSN5xzIySdJ+mO4O8Srz8vFDdEo8mStjrntjvnqiU9Kel6z5mAsOCce0tS6QmHr5f0p+DHf5J0Q3tmQrPfF3jknNvrnFse/LhM0gZJmfL880Jxa3t3mtnq4GlvLjf4kSmpsMnngeAx+OckvWJmBWb2Zd9h8JHezrm9wY/3SertMww+ht8pYcDM+ksaL+kDef55obidJjN7zczWnuR2vaQHJA2SNE7SXkn/5TMrEIamOOcmqOEy9h1mdrHvQPg41zBqgHED4YHfKWHAzDpLWiDpX5xzR5ve5+PnJaE9/7Jo4Jy7rDWPM7M/SHohxHFwckWSspt8nhU8Bs+cc0XBP/eb2bNquKz9lt9UkFRsZhnOub1mliFpv+9AkJxzxY0f8zvFDzNLVENpe8w590zwsNefF864taHgN7DRDElrm3ssQmqZpCFmNsDMkiTdJGmR50wxz8w6mVmXxo8lXSF+RsLFIkmfD378eUkLPWZBEL9T/DIzk/SwpA3OuV83ucvrzwsDeNuQmf1FDae0naSdkr7S5Do42lFw2fx/S4qX9Ihz7md+E8HMBkp6NvhpgqTH+b60PzN7QtKlktIkFUv6gaTnJD0tKUfSLklznHO8Ub4dNfN9uVT8TvHGzKZIelvSGkn1wcP/rob3uXn7eaG4AQAARAgulQIAAEQIihsAAECEoLgBAABECIobAABAhKC4AQAARAiKG4CoYGZ1ZrayyS0sNkpvkqtvC4/5gZn94oRj48xsQ/DjJWZ2zMxyQ50XQHhjHAiAqGBmx5xzndv4NROcc7Vn+RqnzGVmQyX93Tk3sMmxeyVVOOd+HPz8DUn/6pzLP5s8ACIbZ9wARDUz22lmPzKz5Wa2xsyGB493Cm7c/aGZrQjuNywz+4KZLTKz1yUtNrOOZva0ma03s2fN7AMzyzWzL5rZfzf5e/7JzO5rRZ4rzGxpMM98M+vsnNss6ZCZndvkoXMkPdGm/2MAiHgUNwDRosMJl0rnNrnvQHBz+wck/Wvw2Hckve6cmyxpqqRfBbfikqQJkmY55y6RdLukQ865EZK+J2li8DFPS7ouuJehJN0q6ZGWAppZmqTvSrosmCdf0t3Bu59Qw/ZsMrPzJJU657ac/v8MAKIZm8wDiBbHnXPjmrmvcXPoAkk3Bj++QtJ0M2sscilq2MJGkl5tsoXNFEn/I0nOubVmtjr48bHgWblrg+9FS3TOrTlFxvMkjZD0bsM2iEqStDR431OS3jOzb6ihwHG2DcAnUNwAxIKq4J91+se/eyZppnNuU9MHBi9Xlrfydf9PDXsXbpT0aCseb2oohfNOvMM5V2hmOyRdImmmpPNbmQFADOFSKYBY9bKkuyx46svMxjfzuHfV8H4zmdkISaMb73DOfSApW9LNat0ZsvclXWhmg4Ov1ym4MKHRE5Luk7TdORc4vS8HQCyguAGIFie+x+3eUzz+J5ISJa02s3XBz0/mfknpZrZe0k8lrZN0pMn9T0t61zl36FQBnXMlkr4g6YngJdelkoY3ech8SSPFZVIAzWAcCAC0wMzi1fD+tUozGyTpNUnDnHPVwftfkHSfc25xM89vkzEljAMBIHHGDQBOpaOkd8xslaRnJd3unKs2s25mtlkNiyJOWtqCjp5qAO+pmNkSSQMl1ZzpawCIDpxxAwAAiBCccQMAAIgQFDcAAIAIQXEDAACIEBQ3AACACEFxAwAAiBAUNwAAgAjx/wEG2XWUxkdQawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(10,10))\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "ax.plot(ldos_calculator.energy_grid, ldos_calculator.density_of_states)\n",
    "\n",
    "ax.set_xlabel(\"Energy [eV]\")\n",
    "ax.set_ylabel(\"Density of States [1/eV]\")\n",
    "\n",
    "print(ldos_calculator.band_energy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Training a model (hands-on)\n",
    "\n",
    "Finally, we can train a neural network based model for the electronic structure. First, let us review which parameters we need for this. In the following we will slowly adapt the parameters until we get a good model out of it.\n",
    "\n",
    "Since we want to learn about the inner workings of MALA, we want full output. We will also fix the manual seed, so that all the models are comparable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 2\n",
    "parameters.manual_seed = 2023"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Since MALA provides quite a few reasonable default values, in the simplest case the only thing we have to decide upon is the data we want to learn on and the architecture of the neural network.\n",
    "\n",
    "For each training we have to specify training (`\"tr\"`) and validation (`\"va\"`) snapshots. The former are used to actually tune the network weights, the latter monitor model performance during training. They will become very relevant as we optimize the training process.\n",
    "\n",
    "Deciding on the layer sizes is usually done AFTER the data is loaded, since the first and last layer need to match up with the data provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No data rescaling will be performed.\n",
      "No data rescaling will be performed.\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n"
     ]
    }
   ],
   "source": [
    "data_handler = mala.DataHandler(parameters)\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "parameters.network.layer_sizes = [data_handler.input_dimension,\n",
    "                                  100,\n",
    "                                  data_handler.output_dimension]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we can actually train a network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Guess - validation data loss:  0.23660719517299106\n",
      "Epoch 0: validation data loss: 2.0642450877598352e-05, training data loss: 0.0010316168240138463\n",
      "Time for epoch[s]: 1.466733694076538\n",
      "Epoch 1: validation data loss: 8.426698190825327e-06, training data loss: 9.5022799713271e-06\n",
      "Time for epoch[s]: 1.4909157752990723\n",
      "Epoch 2: validation data loss: 6.573324224778584e-06, training data loss: 4.276818728872708e-06\n",
      "Time for epoch[s]: 1.4534635543823242\n",
      "Epoch 3: validation data loss: 6.066970527172088e-06, training data loss: 3.2637411994593483e-06\n",
      "Time for epoch[s]: 1.464097023010254\n",
      "Epoch 4: validation data loss: 5.8819215212549484e-06, training data loss: 2.9526403439896447e-06\n",
      "Time for epoch[s]: 1.4006333351135254\n",
      "Epoch 5: validation data loss: 5.756421280758722e-06, training data loss: 2.780921491129058e-06\n",
      "Time for epoch[s]: 1.5593664646148682\n",
      "Epoch 6: validation data loss: 5.729225065026964e-06, training data loss: 2.724048016326768e-06\n",
      "Time for epoch[s]: 1.4552748203277588\n",
      "Epoch 7: validation data loss: 5.697706980364663e-06, training data loss: 2.702274758900915e-06\n",
      "Time for epoch[s]: 1.2446489334106445\n",
      "Epoch 8: validation data loss: 5.712806646312986e-06, training data loss: 2.691476472786495e-06\n",
      "Time for epoch[s]: 1.4105827808380127\n",
      "Epoch 9: validation data loss: 5.707444357020515e-06, training data loss: 2.685247255223138e-06\n",
      "Time for epoch[s]: 1.34226655960083\n",
      "Final validation data loss:  5.707444357020515e-06\n"
     ]
    }
   ],
   "source": [
    "parameters.running.max_number_epochs = 10\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "trainer.train_network()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Well, how was that? Do we have a good model now, can we predict the LDOS with this? That is not an easy question to answer from this output alone. First of all, we see loss values being printed, and those look all nice, but they are not trivially related to physical/chemical accuracy, which we are actually looking for.\n",
    "\n",
    "We can test this by using the `Tester` class. The class works similar to the Trainer class. We add data, push them through the model, and then use the results to perform calculations.\n",
    "\n",
    "We just have to make sure that the LDOS is correctly integrated by setting the appropriate parameters. Then we can add data to test. We should always test on data different from the one we trained on. Also, we now have to specify the corresponding calculation output, since we may need this for integration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot2.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot2.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot3.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot3.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "DataHandler prepared for inference. No training possible with this setup. If this is not what you wanted, please revise the input script. Validation snapshots you may have entered willbe ignored.\n",
      "Consistency check successful.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n"
     ]
    }
   ],
   "source": [
    "parameters.targets.ldos_gridsize = 11\n",
    "parameters.targets.ldos_gridspacing_ev = 2.5\n",
    "parameters.targets.ldos_gridoffset_ev = -5\n",
    "\n",
    "data_handler.clear_data()\n",
    "data_handler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                          \"Be_snapshot2.out.npy\", data_path, \"te\",\n",
    "                          calculation_output_file=pj(data_path, \"Be_snapshot2.out\"))\n",
    "data_handler.add_snapshot(\"Be_snapshot3.in.npy\", data_path,\n",
    "                          \"Be_snapshot3.out.npy\", data_path, \"te\",\n",
    "                          calculation_output_file=pj(data_path, \"Be_snapshot3.out\"))\n",
    "data_handler.prepare_data(reparametrize_scaler=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now comes the actual object with which to test. We simply tell it which observables to test for and off we go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Had to readjust batch size from 10 to 12\n"
     ]
    }
   ],
   "source": [
    "tester = mala.Tester(parameters, network, data_handler, observables_to_test=[\"band_energy\"])\n",
    "results = tester.test_all_snapshots()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The results are given as a dictionary and in the units of meV/atom."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'band_energy': [0.07822372337799166, 0.08578635234309928]}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "That already looks pretty solid. On the machine this notebook was tested on, errors below 0.1 meV/atom were reported, i.e., the network reproduces the band energy excellently. We may also visualize the DOS, and will do so later, but for now let's focus on the band energy. Can we get even better results? Of course, for simple data like this, we already seem to be doing a good job. But larger and more diverse data sets can be tricky.\n",
    "\n",
    "So let us see if we can improve the training routine even further. To do so, first let's rewrite the code above into functions so that we can toy around with them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def training(parameters):\n",
    "    data_handler = mala.DataHandler(parameters)\n",
    "    data_handler.clear_data()\n",
    "\n",
    "    data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                              \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "    data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                              \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "    # This already loads data into RAM!\n",
    "    data_handler.prepare_data()\n",
    "    parameters.network.layer_sizes.insert(0, data_handler.input_dimension)\n",
    "    parameters.network.layer_sizes.append(data_handler.output_dimension)\n",
    "    network = mala.Network(parameters)\n",
    "    trainer = mala.Trainer(parameters, network, data_handler)\n",
    "    trainer.train_network()\n",
    "\n",
    "\n",
    "    return parameters, data_handler, network\n",
    "\n",
    "def testing(parameters, data_handler, network):\n",
    "\n",
    "    parameters.targets.ldos_gridsize = 11\n",
    "    parameters.targets.ldos_gridspacing_ev = 2.5\n",
    "    parameters.targets.ldos_gridoffset_ev = -5\n",
    "\n",
    "    data_handler.clear_data()\n",
    "    data_handler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                              \"Be_snapshot2.out.npy\", data_path, \"te\",\n",
    "                              calculation_output_file=pj(data_path, \"Be_snapshot2.out\"))\n",
    "    data_handler.add_snapshot(\"Be_snapshot3.in.npy\", data_path,\n",
    "                              \"Be_snapshot3.out.npy\", data_path, \"te\",\n",
    "                              calculation_output_file=pj(data_path, \"Be_snapshot3.out\"))\n",
    "    data_handler.prepare_data(reparametrize_scaler=False)\n",
    "\n",
    "    tester = mala.Tester(parameters, network, data_handler, observables_to_test=[\"band_energy\"])\n",
    "    results = tester.test_all_snapshots()\n",
    "    return results\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now let's see what happens if we start changing up the parameters. Let's start with those that influence the network architecture first. If we define the functions in an appropriate way, we can easily manipulate layer sizes of the neural networks for instance. What happens if we choose a ridiculously small number of neurons for the hidden layer? What happens if we choose a large number? Try e.g. 2 or 1000 as number for the hidden layer (the one in between)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No data rescaling will be performed.\n",
      "No data rescaling will be performed.\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Initial Guess - validation data loss:  0.244789306640625\n",
      "Epoch 0: validation data loss: 5.834028124809265e-06, training data loss: 0.00029535348074776786\n",
      "Time for epoch[s]: 0.9539053440093994\n",
      "Epoch 1: validation data loss: 5.244482308626175e-06, training data loss: 2.741139116031783e-06\n",
      "Time for epoch[s]: 0.9168529510498047\n",
      "Epoch 2: validation data loss: 4.929769784212113e-06, training data loss: 2.128387801349163e-06\n",
      "Time for epoch[s]: 0.8374111652374268\n",
      "Epoch 3: validation data loss: 4.762781517846244e-06, training data loss: 1.8161613760249955e-06\n",
      "Time for epoch[s]: 0.8597333431243896\n",
      "Epoch 4: validation data loss: 4.624924489430019e-06, training data loss: 1.6222886208977017e-06\n",
      "Time for epoch[s]: 0.7933599948883057\n",
      "Epoch 5: validation data loss: 4.525044134684971e-06, training data loss: 1.4753945704017366e-06\n",
      "Time for epoch[s]: 0.8480877876281738\n",
      "Epoch 6: validation data loss: 4.425929061004093e-06, training data loss: 1.3596758778606142e-06\n",
      "Time for epoch[s]: 0.8882327079772949\n",
      "Epoch 7: validation data loss: 4.361816282783236e-06, training data loss: 1.2635821476578712e-06\n",
      "Time for epoch[s]: 0.8873751163482666\n",
      "Epoch 8: validation data loss: 4.3037962168455125e-06, training data loss: 1.1865913069673946e-06\n",
      "Time for epoch[s]: 0.9497289657592773\n",
      "Epoch 9: validation data loss: 4.2643618902989796e-06, training data loss: 1.125034343983446e-06\n",
      "Time for epoch[s]: 1.1165359020233154\n",
      "Epoch 10: validation data loss: 4.2058969182627546e-06, training data loss: 1.071995190743889e-06\n",
      "Time for epoch[s]: 0.959904670715332\n",
      "Epoch 11: validation data loss: 4.193282819219998e-06, training data loss: 1.0275359132460184e-06\n",
      "Time for epoch[s]: 0.9726076126098633\n",
      "Epoch 12: validation data loss: 4.143711179494858e-06, training data loss: 9.87964349665812e-07\n",
      "Time for epoch[s]: 0.9131906032562256\n",
      "Epoch 13: validation data loss: 4.124534715499197e-06, training data loss: 9.520175600690501e-07\n",
      "Time for epoch[s]: 0.9408445358276367\n",
      "Epoch 14: validation data loss: 4.111648138080325e-06, training data loss: 9.204593620129994e-07\n",
      "Time for epoch[s]: 0.8366727828979492\n",
      "Epoch 15: validation data loss: 4.078522590654237e-06, training data loss: 8.922993604625974e-07\n",
      "Time for epoch[s]: 0.8850586414337158\n",
      "Epoch 16: validation data loss: 4.048402022038188e-06, training data loss: 8.674472836511475e-07\n",
      "Time for epoch[s]: 1.1145763397216797\n",
      "Epoch 17: validation data loss: 4.04401496052742e-06, training data loss: 8.44614247658423e-07\n",
      "Time for epoch[s]: 0.8859632015228271\n",
      "Epoch 18: validation data loss: 4.025137051939964e-06, training data loss: 8.236555648701531e-07\n",
      "Time for epoch[s]: 0.8323335647583008\n",
      "Epoch 19: validation data loss: 4.013977146574429e-06, training data loss: 8.03465023636818e-07\n",
      "Time for epoch[s]: 0.8863818645477295\n",
      "Epoch 20: validation data loss: 4.013364602412496e-06, training data loss: 7.841698825359344e-07\n",
      "Time for epoch[s]: 0.873464822769165\n",
      "Epoch 21: validation data loss: 3.995969625455992e-06, training data loss: 7.659875388656344e-07\n",
      "Time for epoch[s]: 1.0247023105621338\n",
      "Epoch 22: validation data loss: 3.981759239520346e-06, training data loss: 7.481735332735947e-07\n",
      "Time for epoch[s]: 0.9872980117797852\n",
      "Epoch 23: validation data loss: 3.974564905677523e-06, training data loss: 7.307136963520731e-07\n",
      "Time for epoch[s]: 0.9826028347015381\n",
      "Epoch 24: validation data loss: 3.948054143360683e-06, training data loss: 7.137264391141279e-07\n",
      "Time for epoch[s]: 0.846106767654419\n",
      "Epoch 25: validation data loss: 3.9370589490447725e-06, training data loss: 6.984172921095576e-07\n",
      "Time for epoch[s]: 0.8015139102935791\n",
      "Epoch 26: validation data loss: 3.934834950736591e-06, training data loss: 6.839311016457421e-07\n",
      "Time for epoch[s]: 1.043769121170044\n",
      "Epoch 27: validation data loss: 3.909551139388766e-06, training data loss: 6.700302474200725e-07\n",
      "Time for epoch[s]: 1.3587698936462402\n",
      "Epoch 28: validation data loss: 3.895905667117664e-06, training data loss: 6.546833818512303e-07\n",
      "Time for epoch[s]: 1.4599790573120117\n",
      "Epoch 29: validation data loss: 3.869765571185521e-06, training data loss: 6.408440614385265e-07\n",
      "Time for epoch[s]: 1.2522456645965576\n",
      "Epoch 30: validation data loss: 3.865611074226243e-06, training data loss: 6.294063558535917e-07\n",
      "Time for epoch[s]: 1.0408508777618408\n",
      "Epoch 31: validation data loss: 3.879226744174957e-06, training data loss: 6.193071603775024e-07\n",
      "Time for epoch[s]: 0.9917383193969727\n",
      "Epoch 32: validation data loss: 3.871617306556021e-06, training data loss: 6.091702463371413e-07\n",
      "Time for epoch[s]: 1.1100733280181885\n",
      "Epoch 33: validation data loss: 3.859372011252812e-06, training data loss: 5.991860026759761e-07\n",
      "Time for epoch[s]: 1.3517563343048096\n",
      "Epoch 34: validation data loss: 3.842448017426899e-06, training data loss: 5.898332622434412e-07\n",
      "Time for epoch[s]: 1.1708111763000488\n",
      "Epoch 35: validation data loss: 3.837319357054574e-06, training data loss: 5.810892741594996e-07\n",
      "Time for epoch[s]: 0.9588320255279541\n",
      "Epoch 36: validation data loss: 3.8042677832501274e-06, training data loss: 5.726385861635208e-07\n",
      "Time for epoch[s]: 0.8209900856018066\n",
      "Epoch 37: validation data loss: 3.808795873607908e-06, training data loss: 5.6473258882761e-07\n",
      "Time for epoch[s]: 0.7978405952453613\n",
      "Epoch 38: validation data loss: 3.8112801100526536e-06, training data loss: 5.570818404001849e-07\n",
      "Time for epoch[s]: 0.877997636795044\n",
      "Epoch 39: validation data loss: 3.809166806084769e-06, training data loss: 5.496346857398748e-07\n",
      "Time for epoch[s]: 1.1807048320770264\n",
      "Epoch 40: validation data loss: 3.7976261228322983e-06, training data loss: 5.42526033573917e-07\n",
      "Time for epoch[s]: 1.044703722000122\n",
      "Epoch 41: validation data loss: 3.804136599813189e-06, training data loss: 5.357798321970871e-07\n",
      "Time for epoch[s]: 0.8686115741729736\n",
      "Epoch 42: validation data loss: 3.7748076553855624e-06, training data loss: 5.289212401424136e-07\n",
      "Time for epoch[s]: 1.1975018978118896\n",
      "Epoch 43: validation data loss: 3.7694341902221954e-06, training data loss: 5.226787512323687e-07\n",
      "Time for epoch[s]: 1.253291368484497\n",
      "Epoch 44: validation data loss: 3.7639361939259938e-06, training data loss: 5.164961330592632e-07\n",
      "Time for epoch[s]: 1.4114727973937988\n",
      "Epoch 45: validation data loss: 3.757831241403307e-06, training data loss: 5.103229611579861e-07\n",
      "Time for epoch[s]: 1.345231533050537\n",
      "Epoch 46: validation data loss: 3.726411876933915e-06, training data loss: 5.04297370623265e-07\n",
      "Time for epoch[s]: 1.166452407836914\n",
      "Epoch 47: validation data loss: 3.7583075463771822e-06, training data loss: 4.987207773540701e-07\n",
      "Time for epoch[s]: 0.9407825469970703\n",
      "Epoch 48: validation data loss: 3.730817564896175e-06, training data loss: 4.931946085499866e-07\n",
      "Time for epoch[s]: 0.9946691989898682\n",
      "Epoch 49: validation data loss: 3.7243666925600596e-06, training data loss: 4.879633696483715e-07\n",
      "Time for epoch[s]: 1.1030848026275635\n",
      "Epoch 50: validation data loss: 3.732029614703996e-06, training data loss: 4.828534687736204e-07\n",
      "Time for epoch[s]: 1.0318608283996582\n",
      "Epoch 51: validation data loss: 3.7347754197461264e-06, training data loss: 4.779226811868804e-07\n",
      "Time for epoch[s]: 1.0676965713500977\n",
      "Epoch 52: validation data loss: 3.7011091730424335e-06, training data loss: 4.7281194877411636e-07\n",
      "Time for epoch[s]: 0.8976039886474609\n",
      "Epoch 53: validation data loss: 3.738165700009891e-06, training data loss: 4.6758393623999185e-07\n",
      "Time for epoch[s]: 0.9173331260681152\n",
      "Epoch 54: validation data loss: 3.7457112755094254e-06, training data loss: 4.6261163827564035e-07\n",
      "Time for epoch[s]: 0.963165283203125\n",
      "Epoch 55: validation data loss: 3.7369890404599055e-06, training data loss: 4.580716070319925e-07\n",
      "Time for epoch[s]: 0.8963465690612793\n",
      "Epoch 56: validation data loss: 3.7175725613321577e-06, training data loss: 4.536323581955263e-07\n",
      "Time for epoch[s]: 1.3438055515289307\n",
      "Epoch 57: validation data loss: 3.730743591274534e-06, training data loss: 4.494400096258947e-07\n",
      "Time for epoch[s]: 0.949730634689331\n",
      "Epoch 58: validation data loss: 3.7324904863323485e-06, training data loss: 4.453628124403102e-07\n",
      "Time for epoch[s]: 0.9233493804931641\n",
      "Epoch 59: validation data loss: 3.707300073334149e-06, training data loss: 4.413437896541187e-07\n",
      "Time for epoch[s]: 0.9144222736358643\n",
      "Epoch 60: validation data loss: 3.6912150681018828e-06, training data loss: 4.3746100605598517e-07\n",
      "Time for epoch[s]: 0.883652925491333\n",
      "Epoch 61: validation data loss: 3.6833501820053372e-06, training data loss: 4.3377300192202843e-07\n",
      "Time for epoch[s]: 0.837639331817627\n",
      "Epoch 62: validation data loss: 3.7265667425734657e-06, training data loss: 4.3018960527011326e-07\n",
      "Time for epoch[s]: 0.9514174461364746\n",
      "Epoch 63: validation data loss: 3.7289283105305264e-06, training data loss: 4.2662713011460644e-07\n",
      "Time for epoch[s]: 0.9508044719696045\n",
      "Epoch 64: validation data loss: 3.705254090683801e-06, training data loss: 4.230653201895101e-07\n",
      "Time for epoch[s]: 0.9344041347503662\n",
      "Epoch 65: validation data loss: 3.700103344661849e-06, training data loss: 4.195959772914648e-07\n",
      "Time for epoch[s]: 0.8698806762695312\n",
      "Epoch 66: validation data loss: 3.7127118557691575e-06, training data loss: 4.161677123712642e-07\n",
      "Time for epoch[s]: 0.9121205806732178\n",
      "Epoch 67: validation data loss: 3.6803696836744036e-06, training data loss: 4.1277314137135234e-07\n",
      "Time for epoch[s]: 0.8251993656158447\n",
      "Epoch 68: validation data loss: 3.6609566637447904e-06, training data loss: 4.0926993824541567e-07\n",
      "Time for epoch[s]: 0.8596961498260498\n",
      "Epoch 69: validation data loss: 3.667564530457769e-06, training data loss: 4.056044854223728e-07\n",
      "Time for epoch[s]: 0.912132740020752\n",
      "Epoch 70: validation data loss: 3.6425372319562095e-06, training data loss: 4.0205451659858225e-07\n",
      "Time for epoch[s]: 0.8348939418792725\n",
      "Epoch 71: validation data loss: 3.641250410250255e-06, training data loss: 3.9856372001980033e-07\n",
      "Time for epoch[s]: 1.0149204730987549\n",
      "Epoch 72: validation data loss: 3.65725186254297e-06, training data loss: 3.950945101678371e-07\n",
      "Time for epoch[s]: 0.8171305656433105\n",
      "Epoch 73: validation data loss: 3.6613097680466515e-06, training data loss: 3.9173057302832604e-07\n",
      "Time for epoch[s]: 0.7949614524841309\n",
      "Epoch 74: validation data loss: 3.6289628062929424e-06, training data loss: 3.883328089224441e-07\n",
      "Time for epoch[s]: 0.8401296138763428\n",
      "Epoch 75: validation data loss: 3.612676901476724e-06, training data loss: 3.853552376053163e-07\n",
      "Time for epoch[s]: 0.9985275268554688\n",
      "Epoch 76: validation data loss: 3.6322192421981268e-06, training data loss: 3.822708302842719e-07\n",
      "Time for epoch[s]: 1.2164177894592285\n",
      "Epoch 77: validation data loss: 3.584957814642361e-06, training data loss: 3.7902274302073887e-07\n",
      "Time for epoch[s]: 1.1049456596374512\n",
      "Epoch 78: validation data loss: 3.603496721812657e-06, training data loss: 3.762632674936737e-07\n",
      "Time for epoch[s]: 0.9448554515838623\n",
      "Epoch 79: validation data loss: 3.597346799714225e-06, training data loss: 3.7338238741670337e-07\n",
      "Time for epoch[s]: 0.8571634292602539\n",
      "Epoch 80: validation data loss: 3.5805281783853257e-06, training data loss: 3.707667347043753e-07\n",
      "Time for epoch[s]: 0.9321854114532471\n",
      "Epoch 81: validation data loss: 3.5859812051057817e-06, training data loss: 3.6806992388197356e-07\n",
      "Time for epoch[s]: 0.8846604824066162\n",
      "Epoch 82: validation data loss: 3.5816130361386707e-06, training data loss: 3.653071222028562e-07\n",
      "Time for epoch[s]: 0.9381594657897949\n",
      "Epoch 83: validation data loss: 3.5773145833185743e-06, training data loss: 3.6277941295078824e-07\n",
      "Time for epoch[s]: 0.8648409843444824\n",
      "Epoch 84: validation data loss: 3.58041056564876e-06, training data loss: 3.6026989775044576e-07\n",
      "Time for epoch[s]: 0.9122714996337891\n",
      "Epoch 85: validation data loss: 3.5864806600979395e-06, training data loss: 3.577428204672677e-07\n",
      "Time for epoch[s]: 0.8204946517944336\n",
      "Epoch 86: validation data loss: 3.5725754818746024e-06, training data loss: 3.5550747998058797e-07\n",
      "Time for epoch[s]: 0.9503178596496582\n",
      "Epoch 87: validation data loss: 3.569589661700385e-06, training data loss: 3.53209042389478e-07\n",
      "Time for epoch[s]: 0.8654520511627197\n",
      "Epoch 88: validation data loss: 3.5829943205629076e-06, training data loss: 3.5080167331865853e-07\n",
      "Time for epoch[s]: 0.9128880500793457\n",
      "Epoch 89: validation data loss: 3.539341368845531e-06, training data loss: 3.4875825180539065e-07\n",
      "Time for epoch[s]: 0.9130682945251465\n",
      "Epoch 90: validation data loss: 3.5582448222807476e-06, training data loss: 3.466461119907243e-07\n",
      "Time for epoch[s]: 0.816162109375\n",
      "Epoch 91: validation data loss: 3.549937158823013e-06, training data loss: 3.4448657450931413e-07\n",
      "Time for epoch[s]: 0.8356456756591797\n",
      "Epoch 92: validation data loss: 3.543146486793246e-06, training data loss: 3.425062833619969e-07\n",
      "Time for epoch[s]: 0.803959846496582\n",
      "Epoch 93: validation data loss: 3.5263411700725556e-06, training data loss: 3.403990329908473e-07\n",
      "Time for epoch[s]: 0.858290433883667\n",
      "Epoch 94: validation data loss: 3.517695835658482e-06, training data loss: 3.384517372718879e-07\n",
      "Time for epoch[s]: 0.8542344570159912\n",
      "Epoch 95: validation data loss: 3.5281255841255186e-06, training data loss: 3.366077518356698e-07\n",
      "Time for epoch[s]: 1.0944738388061523\n",
      "Epoch 96: validation data loss: 3.5274970744337354e-06, training data loss: 3.3476266876927443e-07\n",
      "Time for epoch[s]: 0.9098708629608154\n",
      "Epoch 97: validation data loss: 3.5471320152282717e-06, training data loss: 3.3283662716192856e-07\n",
      "Time for epoch[s]: 0.8842885494232178\n",
      "Epoch 98: validation data loss: 3.515759749071939e-06, training data loss: 3.311389258929661e-07\n",
      "Time for epoch[s]: 0.8268096446990967\n",
      "Epoch 99: validation data loss: 3.4991691687277384e-06, training data loss: 3.29408362241728e-07\n",
      "Time for epoch[s]: 0.8589010238647461\n",
      "Final validation data loss:  3.4991691687277384e-06\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot2.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot2.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot3.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot3.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "DataHandler prepared for inference. No training possible with this setup. If this is not what you wanted, please revise the input script. Validation snapshots you may have entered willbe ignored.\n",
      "Consistency check successful.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Had to readjust batch size from 10 to 12\n",
      "{'band_energy': [0.10552580387314592, 0.040525814617691225]}\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 2\n",
    "parameters.manual_seed = 2023\n",
    "parameters.network.layer_sizes = [1000]\n",
    "\n",
    "parameters, data_handler, network = training(parameters)\n",
    "print(testing(parameters, data_handler, network))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Ok, so this clearly has an effect. We can build better and worse models by simply adjusting this number. Models that are too small don't have enough information capacity - they simply cannot capture the information present in the data. This is called **underfitting**.\n",
    "\n",
    "But also big models (if you haven't tried - try e.g. 1000 neurons!) do not yield immediate improvement. In my case, 1000 neurons in the middle gave noticeably worse performance.\n",
    "\n",
    "Why is that? Large models possess a large capacity to store information. If they are not carefully fitted, they may very soon begin to **overfit**, i.e. learn to predict the training data - and only the training data. This is obviously not what is supposed to happen.\n",
    "\n",
    "This is what the aforementioned validation data is for. Validation data is data unseen by the model during the weight optimization, but checked after each epoch. The idea is simple: Since the model is not directly fitted on the validation data, we can use it to monitor fitting progress. If validation accuracy is drastically different from training accuracy, something is off.\n",
    "\n",
    "If you check your outputs, you will realize that both losses behave differently for different models. In the 1000 neuron case at the end you will encounter a factor of 10 between both of them! For a smaller model, it was only 2.\n",
    "\n",
    "There is one way to alleviate this: Early stopping. The idea is to end training prematurely if we see validation accuracy stagnating for a certain amount of time, i.e., the model starting to overfit. MALA supports this with a simple command. Let's try it out with 1000 neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No data rescaling will be performed.\n",
      "No data rescaling will be performed.\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Initial Guess - validation data loss:  0.244789306640625\n",
      "Epoch 0: validation data loss: 5.834028124809265e-06, training data loss: 0.00029535348074776786\n",
      "Time for epoch[s]: 1.2693874835968018\n",
      "Epoch 1: validation data loss: 5.244482308626175e-06, training data loss: 2.741139116031783e-06\n",
      "Time for epoch[s]: 1.1986780166625977\n",
      "Epoch 2: validation data loss: 4.929769784212113e-06, training data loss: 2.128387801349163e-06\n",
      "Time for epoch[s]: 1.1669394969940186\n",
      "Epoch 3: validation data loss: 4.762781517846244e-06, training data loss: 1.8161613760249955e-06\n",
      "Time for epoch[s]: 0.9472312927246094\n",
      "Epoch 4: validation data loss: 4.624924489430019e-06, training data loss: 1.6222886208977017e-06\n",
      "Time for epoch[s]: 1.0300989151000977\n",
      "Epoch 5: validation data loss: 4.525044134684971e-06, training data loss: 1.4753945704017366e-06\n",
      "Time for epoch[s]: 1.252479076385498\n",
      "Epoch 6: validation data loss: 4.425929061004093e-06, training data loss: 1.3596758778606142e-06\n",
      "Time for epoch[s]: 0.9820547103881836\n",
      "Epoch 7: validation data loss: 4.361816282783236e-06, training data loss: 1.2635821476578712e-06\n",
      "Time for epoch[s]: 1.1120128631591797\n",
      "Epoch 8: validation data loss: 4.3037962168455125e-06, training data loss: 1.1865913069673946e-06\n",
      "Time for epoch[s]: 0.9094545841217041\n",
      "Epoch 9: validation data loss: 4.2643618902989796e-06, training data loss: 1.125034343983446e-06\n",
      "Time for epoch[s]: 0.8620121479034424\n",
      "Epoch 10: validation data loss: 4.2058969182627546e-06, training data loss: 1.071995190743889e-06\n",
      "Time for epoch[s]: 0.8079037666320801\n",
      "Epoch 11: validation data loss: 4.193282819219998e-06, training data loss: 1.0275359132460184e-06\n",
      "Time for epoch[s]: 0.844515323638916\n",
      "Epoch 12: validation data loss: 4.143711179494858e-06, training data loss: 9.87964349665812e-07\n",
      "Time for epoch[s]: 0.7933821678161621\n",
      "Epoch 13: validation data loss: 4.124534715499197e-06, training data loss: 9.520175600690501e-07\n",
      "Time for epoch[s]: 0.8413608074188232\n",
      "Epoch 14: validation data loss: 4.111648138080325e-06, training data loss: 9.204593620129994e-07\n",
      "Time for epoch[s]: 0.8194360733032227\n",
      "Epoch 15: validation data loss: 4.078522590654237e-06, training data loss: 8.922993604625974e-07\n",
      "Time for epoch[s]: 0.802377462387085\n",
      "Epoch 16: validation data loss: 4.048402022038188e-06, training data loss: 8.674472836511475e-07\n",
      "Time for epoch[s]: 0.854393482208252\n",
      "Epoch 17: validation data loss: 4.04401496052742e-06, training data loss: 8.44614247658423e-07\n",
      "Time for epoch[s]: 0.7976665496826172\n",
      "Epoch 18: validation data loss: 4.025137051939964e-06, training data loss: 8.236555648701531e-07\n",
      "Time for epoch[s]: 0.7978990077972412\n",
      "Epoch 19: validation data loss: 4.013977146574429e-06, training data loss: 8.03465023636818e-07\n",
      "Time for epoch[s]: 0.801769495010376\n",
      "Epoch 20: validation data loss: 4.013364602412496e-06, training data loss: 7.841698825359344e-07\n",
      "Time for epoch[s]: 0.9166347980499268\n",
      "Epoch 21: validation data loss: 3.995969625455992e-06, training data loss: 7.659875388656344e-07\n",
      "Time for epoch[s]: 0.8892323970794678\n",
      "Epoch 22: validation data loss: 3.981759239520346e-06, training data loss: 7.481735332735947e-07\n",
      "Time for epoch[s]: 0.8756539821624756\n",
      "Epoch 23: validation data loss: 3.974564905677523e-06, training data loss: 7.307136963520731e-07\n",
      "Time for epoch[s]: 0.8676538467407227\n",
      "Epoch 24: validation data loss: 3.948054143360683e-06, training data loss: 7.137264391141279e-07\n",
      "Time for epoch[s]: 0.8055057525634766\n",
      "Epoch 25: validation data loss: 3.9370589490447725e-06, training data loss: 6.984172921095576e-07\n",
      "Time for epoch[s]: 0.7961726188659668\n",
      "Epoch 26: validation data loss: 3.934834950736591e-06, training data loss: 6.839311016457421e-07\n",
      "Time for epoch[s]: 0.8278684616088867\n",
      "Epoch 27: validation data loss: 3.909551139388766e-06, training data loss: 6.700302474200725e-07\n",
      "Time for epoch[s]: 0.8211498260498047\n",
      "Epoch 28: validation data loss: 3.895905667117664e-06, training data loss: 6.546833818512303e-07\n",
      "Time for epoch[s]: 0.8764598369598389\n",
      "Epoch 29: validation data loss: 3.869765571185521e-06, training data loss: 6.408440614385265e-07\n",
      "Time for epoch[s]: 0.8487873077392578\n",
      "Epoch 30: validation data loss: 3.865611074226243e-06, training data loss: 6.294063558535917e-07\n",
      "Time for epoch[s]: 0.8082249164581299\n",
      "Epoch 31: validation data loss: 3.879226744174957e-06, training data loss: 6.193071603775024e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.7916734218597412\n",
      "Epoch 32: validation data loss: 3.871617306556021e-06, training data loss: 6.091702463371413e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.9934244155883789\n",
      "Epoch 33: validation data loss: 3.859372011252812e-06, training data loss: 5.991860026759761e-07\n",
      "Time for epoch[s]: 0.8533694744110107\n",
      "Epoch 34: validation data loss: 3.842448017426899e-06, training data loss: 5.898332622434412e-07\n",
      "Time for epoch[s]: 0.8837182521820068\n",
      "Epoch 35: validation data loss: 3.837319357054574e-06, training data loss: 5.810892741594996e-07\n",
      "Time for epoch[s]: 1.1184439659118652\n",
      "Epoch 36: validation data loss: 3.8042677832501274e-06, training data loss: 5.726385861635208e-07\n",
      "Time for epoch[s]: 0.8387551307678223\n",
      "Epoch 37: validation data loss: 3.808795873607908e-06, training data loss: 5.6473258882761e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.891425371170044\n",
      "Epoch 38: validation data loss: 3.8112801100526536e-06, training data loss: 5.570818404001849e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8733289241790771\n",
      "Epoch 39: validation data loss: 3.809166806084769e-06, training data loss: 5.496346857398748e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8736782073974609\n",
      "Epoch 40: validation data loss: 3.7976261228322983e-06, training data loss: 5.42526033573917e-07\n",
      "Time for epoch[s]: 0.8144981861114502\n",
      "Epoch 41: validation data loss: 3.804136599813189e-06, training data loss: 5.357798321970871e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8638012409210205\n",
      "Epoch 42: validation data loss: 3.7748076553855624e-06, training data loss: 5.289212401424136e-07\n",
      "Time for epoch[s]: 0.8417935371398926\n",
      "Epoch 43: validation data loss: 3.7694341902221954e-06, training data loss: 5.226787512323687e-07\n",
      "Time for epoch[s]: 0.863114595413208\n",
      "Epoch 44: validation data loss: 3.7639361939259938e-06, training data loss: 5.164961330592632e-07\n",
      "Time for epoch[s]: 0.8453967571258545\n",
      "Epoch 45: validation data loss: 3.757831241403307e-06, training data loss: 5.103229611579861e-07\n",
      "Time for epoch[s]: 0.8555707931518555\n",
      "Epoch 46: validation data loss: 3.726411876933915e-06, training data loss: 5.04297370623265e-07\n",
      "Time for epoch[s]: 0.8558299541473389\n",
      "Epoch 47: validation data loss: 3.7583075463771822e-06, training data loss: 4.987207773540701e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 1.0649237632751465\n",
      "Epoch 48: validation data loss: 3.730817564896175e-06, training data loss: 4.931946085499866e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8836357593536377\n",
      "Epoch 49: validation data loss: 3.7243666925600596e-06, training data loss: 4.879633696483715e-07\n",
      "Time for epoch[s]: 0.7898688316345215\n",
      "Epoch 50: validation data loss: 3.732029614703996e-06, training data loss: 4.828534687736204e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.918311595916748\n",
      "Epoch 51: validation data loss: 3.7347754197461264e-06, training data loss: 4.779226811868804e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8406631946563721\n",
      "Epoch 52: validation data loss: 3.7011091730424335e-06, training data loss: 4.7281194877411636e-07\n",
      "Time for epoch[s]: 0.8173329830169678\n",
      "Epoch 53: validation data loss: 3.738165700009891e-06, training data loss: 4.6758393623999185e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8396792411804199\n",
      "Epoch 54: validation data loss: 3.7457112755094254e-06, training data loss: 4.6261163827564035e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8496429920196533\n",
      "Epoch 55: validation data loss: 3.7369890404599055e-06, training data loss: 4.580716070319925e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8154935836791992\n",
      "Epoch 56: validation data loss: 3.7175725613321577e-06, training data loss: 4.536323581955263e-07\n",
      "Validation accuracy has not improved enough.\n",
      "Stopping the training, validation accuracy has not improved for 4 epochs.\n",
      "Final validation data loss:  3.7175725613321577e-06\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot2.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot2.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot3.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot3.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "DataHandler prepared for inference. No training possible with this setup. If this is not what you wanted, please revise the input script. Validation snapshots you may have entered willbe ignored.\n",
      "Consistency check successful.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Had to readjust batch size from 10 to 12\n",
      "{'band_energy': [0.10128238388313093, 0.04330199676283186]}\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 2\n",
    "parameters.manual_seed = 2023\n",
    "parameters.network.layer_sizes = [1000]\n",
    "parameters.running.early_stopping_epochs = 4\n",
    "\n",
    "parameters, data_handler, network = training(parameters)\n",
    "print(testing(parameters, data_handler, network))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In my case, that helped a tiny bit with the accuracy of one snapshot. So we're getting there. Before we overthinking this specific aspect, let's introduce some more hyperparameters and test them out. We can toy with the length of the training, the learning rate (i.e. the aggressivness of the gradient updates of the neural networks, large learning rates mean larger updates per epoch, which can lead to oscillations, small learning rates can lead to stagnation) and the optimizer being used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 2\n",
    "parameters.manual_seed = 2023\n",
    "parameters.network.layer_sizes = [100]\n",
    "parameters.running.early_stopping_epochs = 4\n",
    "# Default value is 100 - try out longer trainings as well. Or shorter!\n",
    "parameters.running.max_number_epochs = 100\n",
    "# Default value is 0.5 - try out a few!\n",
    "parameters.running.learning_rate = 0.5\n",
    "# Default value is SGD - try out Adam instead!\n",
    "parameters.running.trainingtype = \"SGD\"\n",
    "\n",
    "parameters, data_handler, network = training(parameters)\n",
    "print(testing(parameters, data_handler, network))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "You may beginning to feel a bit overwhelmed by all the choices one can make here. We'll address this very real struggle in a second, but we'll just add a bit more gasoline to the fire while were at it.\n",
    "\n",
    "There are two aspects of ML that can be very important (and that MALA supports) that we have not yet talked about.\n",
    "\n",
    "The first is data scaling. The idea behind data scaling is to make data more uniform - vectorial fields as in our case may contain larger and smaller components per data point and it can be hard for a network to correctly learn this. By normalizing or standardizing data, performance is often improved. MALA supports normalization feature-wise or per entire dataset. Empirically, we have made good experiences with the following. (Be aware: if you change the scaling, the absolute loss values will inevitably change - this is why we monitor the band energy.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Initial Guess - validation data loss:  0.054730381556919644\n",
      "Epoch 0: validation data loss: 0.002600432804652623, training data loss: 0.0017240849903651645\n",
      "Time for epoch[s]: 0.5689492225646973\n",
      "Epoch 1: validation data loss: 0.002115297998700823, training data loss: 0.0007406753131321499\n",
      "Time for epoch[s]: 0.5992138385772705\n",
      "Epoch 2: validation data loss: 0.001884244918823242, training data loss: 0.0005785423687526158\n",
      "Time for epoch[s]: 0.4789886474609375\n",
      "Epoch 3: validation data loss: 0.0016306495666503905, training data loss: 0.0004649646622794015\n",
      "Time for epoch[s]: 0.5357327461242676\n",
      "Epoch 4: validation data loss: 0.0014923850468226841, training data loss: 0.0003904345716748919\n",
      "Time for epoch[s]: 0.6085658073425293\n",
      "Epoch 5: validation data loss: 0.001329531124659947, training data loss: 0.0003293121542249407\n",
      "Time for epoch[s]: 0.5431547164916992\n",
      "Epoch 6: validation data loss: 0.001223013060433524, training data loss: 0.0002836345263889858\n",
      "Time for epoch[s]: 0.5654911994934082\n",
      "Epoch 7: validation data loss: 0.0010912742614746095, training data loss: 0.00025134381226130895\n",
      "Time for epoch[s]: 0.49450230598449707\n",
      "Epoch 8: validation data loss: 0.0010236147471836634, training data loss: 0.00022441310541970389\n",
      "Time for epoch[s]: 0.5902736186981201\n",
      "Epoch 9: validation data loss: 0.0009318426677158901, training data loss: 0.00020276956898825508\n",
      "Time for epoch[s]: 0.4696216583251953\n",
      "Epoch 10: validation data loss: 0.0008938068662370954, training data loss: 0.00018409161908285957\n",
      "Time for epoch[s]: 0.6094183921813965\n",
      "Epoch 11: validation data loss: 0.0008327815192086356, training data loss: 0.0001698838642665318\n",
      "Time for epoch[s]: 0.6027736663818359\n",
      "Epoch 12: validation data loss: 0.0008069548606872559, training data loss: 0.00015681793008531843\n",
      "Time for epoch[s]: 0.4275987148284912\n",
      "Epoch 13: validation data loss: 0.000765747002192906, training data loss: 0.00014542531967163085\n",
      "Time for epoch[s]: 0.43206071853637695\n",
      "Epoch 14: validation data loss: 0.0007488994598388672, training data loss: 0.0001363550169127328\n",
      "Time for epoch[s]: 0.5591120719909668\n",
      "Epoch 15: validation data loss: 0.0007141206605093819, training data loss: 0.00012856424706322807\n",
      "Time for epoch[s]: 0.4885239601135254\n",
      "Epoch 16: validation data loss: 0.0007131549971444266, training data loss: 0.00012078116621289934\n",
      "Time for epoch[s]: 0.49959468841552734\n",
      "Epoch 17: validation data loss: 0.0006994085311889649, training data loss: 0.00011448664324624197\n",
      "Time for epoch[s]: 0.46938014030456543\n",
      "Epoch 18: validation data loss: 0.0006789231981549944, training data loss: 0.00010829601969037737\n",
      "Time for epoch[s]: 0.48134922981262207\n",
      "Epoch 19: validation data loss: 0.0006676930018833705, training data loss: 0.00010353785753250122\n",
      "Time for epoch[s]: 0.5197646617889404\n",
      "Epoch 20: validation data loss: 0.000667377199445452, training data loss: 9.87018346786499e-05\n",
      "Time for epoch[s]: 0.45720815658569336\n",
      "Epoch 21: validation data loss: 0.0006493257113865444, training data loss: 9.499633312225341e-05\n",
      "Time for epoch[s]: 0.5711503028869629\n",
      "Epoch 22: validation data loss: 0.0006327464921133859, training data loss: 9.001866408756801e-05\n",
      "Time for epoch[s]: 0.5810775756835938\n",
      "Epoch 23: validation data loss: 0.0006302296774727958, training data loss: 8.685459409441267e-05\n",
      "Time for epoch[s]: 0.4989748001098633\n",
      "Epoch 24: validation data loss: 0.0006333806855337961, training data loss: 8.35321205002921e-05\n",
      "Time for epoch[s]: 0.5152850151062012\n",
      "Epoch 25: validation data loss: 0.0006246117864336286, training data loss: 8.108027492250715e-05\n",
      "Time for epoch[s]: 0.5820825099945068\n",
      "Epoch 26: validation data loss: 0.0006284912654331752, training data loss: 7.772974457059588e-05\n",
      "Time for epoch[s]: 0.5007874965667725\n",
      "Epoch 27: validation data loss: 0.0006212819644383022, training data loss: 7.482621499470302e-05\n",
      "Time for epoch[s]: 0.5595214366912842\n",
      "Epoch 28: validation data loss: 0.000615316527230399, training data loss: 7.262876204081944e-05\n",
      "Time for epoch[s]: 0.4309098720550537\n",
      "Epoch 29: validation data loss: 0.0006032741410391671, training data loss: 7.00118328843798e-05\n",
      "Time for epoch[s]: 0.41736865043640137\n",
      "Epoch 30: validation data loss: 0.0006080447605678013, training data loss: 6.755230256489345e-05\n",
      "Time for epoch[s]: 0.4025270938873291\n",
      "Epoch 31: validation data loss: 0.000602400575365339, training data loss: 6.572359800338746e-05\n",
      "Time for epoch[s]: 0.40505504608154297\n",
      "Epoch 32: validation data loss: 0.0005884628977094378, training data loss: 6.276902556419373e-05\n",
      "Time for epoch[s]: 0.39629340171813965\n",
      "Epoch 33: validation data loss: 0.000590014934539795, training data loss: 6.147450208663941e-05\n",
      "Time for epoch[s]: 0.39157605171203613\n",
      "Epoch 34: validation data loss: 0.0005837083544049944, training data loss: 5.9207792793001445e-05\n",
      "Time for epoch[s]: 0.40552544593811035\n",
      "Epoch 35: validation data loss: 0.0005874847003391811, training data loss: 5.7681236948285786e-05\n",
      "Time for epoch[s]: 0.398850679397583\n",
      "Epoch 36: validation data loss: 0.0005781801768711635, training data loss: 5.545513119016375e-05\n",
      "Time for epoch[s]: 0.4725186824798584\n",
      "Epoch 37: validation data loss: 0.0005786760193961007, training data loss: 5.4123695407594956e-05\n",
      "Time for epoch[s]: 0.4595649242401123\n",
      "Epoch 38: validation data loss: 0.0005807206290108817, training data loss: 5.256196430751256e-05\n",
      "Time for epoch[s]: 0.40080809593200684\n",
      "Epoch 39: validation data loss: 0.0005809912681579589, training data loss: 5.078781502587455e-05\n",
      "Time for epoch[s]: 0.3883938789367676\n",
      "Epoch 40: validation data loss: 0.0005674213341304234, training data loss: 4.953251566205706e-05\n",
      "Time for epoch[s]: 0.41380834579467773\n",
      "Epoch 41: validation data loss: 0.0005593626499176026, training data loss: 4.817848971911839e-05\n",
      "Time for epoch[s]: 0.41025829315185547\n",
      "Epoch 42: validation data loss: 0.0005668220520019531, training data loss: 4.673965062413897e-05\n",
      "Time for epoch[s]: 0.4067525863647461\n",
      "Epoch 43: validation data loss: 0.0005543017727988107, training data loss: 4.545625192778451e-05\n",
      "Time for epoch[s]: 0.4501631259918213\n",
      "Epoch 44: validation data loss: 0.0005595346518925258, training data loss: 4.4564170496804374e-05\n",
      "Time for epoch[s]: 0.433643102645874\n",
      "Epoch 45: validation data loss: 0.0005571234226226806, training data loss: 4.2930462530681064e-05\n",
      "Time for epoch[s]: 0.41202688217163086\n",
      "Epoch 46: validation data loss: 0.0005624114104679652, training data loss: 4.2695931025913784e-05\n",
      "Time for epoch[s]: 0.44544219970703125\n",
      "Epoch 47: validation data loss: 0.0005470786094665527, training data loss: 4.153754455702646e-05\n",
      "Time for epoch[s]: 0.44005656242370605\n",
      "Epoch 48: validation data loss: 0.0005466411794934954, training data loss: 4.049028669084821e-05\n",
      "Time for epoch[s]: 0.4093434810638428\n",
      "Epoch 49: validation data loss: 0.0005446892125265939, training data loss: 3.9641810314995905e-05\n",
      "Time for epoch[s]: 0.41369080543518066\n",
      "Epoch 50: validation data loss: 0.0005347145966121128, training data loss: 3.866049221583775e-05\n",
      "Time for epoch[s]: 0.42542481422424316\n",
      "Epoch 51: validation data loss: 0.0005413805076054164, training data loss: 3.771242925098964e-05\n",
      "Time for epoch[s]: 0.4799916744232178\n",
      "Epoch 52: validation data loss: 0.0005414377961839948, training data loss: 3.7013964993613107e-05\n",
      "Time for epoch[s]: 0.4681978225708008\n",
      "Epoch 53: validation data loss: 0.000545579058783395, training data loss: 3.612976840564183e-05\n",
      "Time for epoch[s]: 0.5491938591003418\n",
      "Epoch 54: validation data loss: 0.0005338455608912876, training data loss: 3.541549827371325e-05\n",
      "Time for epoch[s]: 0.4841008186340332\n",
      "Epoch 55: validation data loss: 0.0005342479092734201, training data loss: 3.44326879296984e-05\n",
      "Time for epoch[s]: 0.7124834060668945\n",
      "Epoch 56: validation data loss: 0.0005250646727425712, training data loss: 3.4115974392209735e-05\n",
      "Time for epoch[s]: 1.1700105667114258\n",
      "Epoch 57: validation data loss: 0.0005219217027936663, training data loss: 3.3425056508609226e-05\n",
      "Time for epoch[s]: 1.224865198135376\n",
      "Epoch 58: validation data loss: 0.0005215444564819336, training data loss: 3.302478151662009e-05\n",
      "Time for epoch[s]: 1.1293365955352783\n",
      "Epoch 59: validation data loss: 0.0005273356778281075, training data loss: 3.238218597003392e-05\n",
      "Time for epoch[s]: 0.6793153285980225\n",
      "Epoch 60: validation data loss: 0.0005186018603188651, training data loss: 3.178612674985613e-05\n",
      "Time for epoch[s]: 1.390303134918213\n",
      "Epoch 61: validation data loss: 0.0005202345166887555, training data loss: 3.1297458069665094e-05\n",
      "Time for epoch[s]: 1.2634379863739014\n",
      "Epoch 62: validation data loss: 0.0005166962146759033, training data loss: 3.070584791047232e-05\n",
      "Time for epoch[s]: 0.7762551307678223\n",
      "Epoch 63: validation data loss: 0.0005153846740722656, training data loss: 3.012762750898089e-05\n",
      "Time for epoch[s]: 1.3292157649993896\n",
      "Epoch 64: validation data loss: 0.0005131649289812361, training data loss: 2.9732423169272287e-05\n",
      "Time for epoch[s]: 0.6974673271179199\n",
      "Epoch 65: validation data loss: 0.000506195068359375, training data loss: 2.9069014957972936e-05\n",
      "Time for epoch[s]: 0.48081302642822266\n",
      "Epoch 66: validation data loss: 0.0005189828191484723, training data loss: 2.8815695217677526e-05\n",
      "Time for epoch[s]: 0.4520707130432129\n",
      "Epoch 67: validation data loss: 0.0005031188556126186, training data loss: 2.809599893433707e-05\n",
      "Time for epoch[s]: 0.4165327548980713\n",
      "Epoch 68: validation data loss: 0.0005108623504638672, training data loss: 2.7967993702207293e-05\n",
      "Time for epoch[s]: 0.4012413024902344\n",
      "Epoch 69: validation data loss: 0.0005055053915296282, training data loss: 2.7454748749732973e-05\n",
      "Time for epoch[s]: 0.4769456386566162\n",
      "Epoch 70: validation data loss: 0.0005115983486175537, training data loss: 2.711637318134308e-05\n",
      "Time for epoch[s]: 0.5851750373840332\n",
      "Epoch 71: validation data loss: 0.0005096389225551061, training data loss: 2.6754494224275862e-05\n",
      "Time for epoch[s]: 0.4380989074707031\n",
      "Epoch 72: validation data loss: 0.0005093855857849121, training data loss: 2.6223557335989815e-05\n",
      "Time for epoch[s]: 0.40836524963378906\n",
      "Epoch 73: validation data loss: 0.0005052408490862165, training data loss: 2.5989830493927e-05\n",
      "Time for epoch[s]: 0.4076876640319824\n",
      "Epoch 74: validation data loss: 0.0005037470204489572, training data loss: 2.562944165297917e-05\n",
      "Time for epoch[s]: 0.4022030830383301\n",
      "Epoch 75: validation data loss: 0.00050250244140625, training data loss: 2.519410422870091e-05\n",
      "Time for epoch[s]: 0.6789491176605225\n",
      "Epoch 76: validation data loss: 0.0005033470221928188, training data loss: 2.4882659316062927e-05\n",
      "Time for epoch[s]: 0.5645167827606201\n",
      "Epoch 77: validation data loss: 0.0005009912763323102, training data loss: 2.4513363838195802e-05\n",
      "Time for epoch[s]: 0.5867531299591064\n",
      "Epoch 78: validation data loss: 0.0004957279137202672, training data loss: 2.430455812386104e-05\n",
      "Time for epoch[s]: 0.5849399566650391\n",
      "Epoch 79: validation data loss: 0.0004987611089433943, training data loss: 2.3945942521095274e-05\n",
      "Time for epoch[s]: 0.479691743850708\n",
      "Epoch 80: validation data loss: 0.0004953630651746478, training data loss: 2.3637186203684125e-05\n",
      "Time for epoch[s]: 0.706742525100708\n",
      "Epoch 81: validation data loss: 0.0004950695378439767, training data loss: 2.3381158709526063e-05\n",
      "Time for epoch[s]: 0.5249426364898682\n",
      "Epoch 82: validation data loss: 0.0004957619394574846, training data loss: 2.3090317845344544e-05\n",
      "Time for epoch[s]: 0.5586967468261719\n",
      "Epoch 83: validation data loss: 0.0004911063058035714, training data loss: 2.2719227841922216e-05\n",
      "Time for epoch[s]: 0.45817136764526367\n",
      "Epoch 84: validation data loss: 0.0004861319065093994, training data loss: 2.248134996209826e-05\n",
      "Time for epoch[s]: 0.4238300323486328\n",
      "Epoch 85: validation data loss: 0.0004929366111755371, training data loss: 2.2246831229754858e-05\n",
      "Time for epoch[s]: 0.4160735607147217\n",
      "Epoch 86: validation data loss: 0.0004929019042423793, training data loss: 2.1937653422355652e-05\n",
      "Time for epoch[s]: 0.4149138927459717\n",
      "Epoch 87: validation data loss: 0.0004898176193237305, training data loss: 2.1616484437670027e-05\n",
      "Time for epoch[s]: 0.39681196212768555\n",
      "Epoch 88: validation data loss: 0.0004914012977055141, training data loss: 2.1547830530575342e-05\n",
      "Time for epoch[s]: 0.6498222351074219\n",
      "Epoch 89: validation data loss: 0.000490537234715053, training data loss: 2.13002690247127e-05\n",
      "Time for epoch[s]: 0.7063117027282715\n",
      "Epoch 90: validation data loss: 0.000491154534476144, training data loss: 2.102278811591012e-05\n",
      "Time for epoch[s]: 0.6055908203125\n",
      "Epoch 91: validation data loss: 0.00047773408889770506, training data loss: 2.0896658301353456e-05\n",
      "Time for epoch[s]: 0.742572546005249\n",
      "Epoch 92: validation data loss: 0.00048268815449305946, training data loss: 2.068880413259779e-05\n",
      "Time for epoch[s]: 0.6297104358673096\n",
      "Epoch 93: validation data loss: 0.0004881516184125628, training data loss: 2.0330284323011126e-05\n",
      "Time for epoch[s]: 0.5251009464263916\n",
      "Epoch 94: validation data loss: 0.0004814927237374442, training data loss: 2.0216831139155798e-05\n",
      "Time for epoch[s]: 0.6148157119750977\n",
      "Epoch 95: validation data loss: 0.00048323913982936315, training data loss: 1.993641257286072e-05\n",
      "Time for epoch[s]: 0.6748700141906738\n",
      "Epoch 96: validation data loss: 0.0004765101841517857, training data loss: 1.9764144505773273e-05\n",
      "Time for epoch[s]: 0.5970304012298584\n",
      "Epoch 97: validation data loss: 0.0004792407240186419, training data loss: 1.9568298544202532e-05\n",
      "Time for epoch[s]: 0.5381877422332764\n",
      "Epoch 98: validation data loss: 0.00048382721628461564, training data loss: 1.9390725663730076e-05\n",
      "Time for epoch[s]: 0.4985616207122803\n",
      "Epoch 99: validation data loss: 0.00048050570487976073, training data loss: 1.9126251339912414e-05\n",
      "Time for epoch[s]: 0.4946012496948242\n",
      "Final validation data loss:  0.00048050570487976073\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot2.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot2.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot3.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot3.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "DataHandler prepared for inference. No training possible with this setup. If this is not what you wanted, please revise the input script. Validation snapshots you may have entered willbe ignored.\n",
      "Consistency check successful.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Had to readjust batch size from 10 to 12\n",
      "{'band_energy': [0.007741880422457825, 0.019942810278099188]}\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 2\n",
    "parameters.manual_seed = 2023\n",
    "parameters.network.layer_sizes = [100]\n",
    "parameters.data.input_rescaling_type = \"feature-wise-standard\"\n",
    "parameters.data.output_rescaling_type = \"normal\"\n",
    "\n",
    "\n",
    "parameters, data_handler, network = training(parameters)\n",
    "print(testing(parameters, data_handler, network))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In my case, adding the data scaling improved performance drastically. Feel free to try out other forms of scaling (for no scaling, just specify `\"None\"`).\n",
    "\n",
    "Secondly, we can adapt the activation functions of the neural net, i.e., the linear transformation at the end of each layer. In MALA, this is realized by a list as well. We can specify individual layers, or simply give only one type to be used at each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Initial Guess - validation data loss:  0.1100400652204241\n",
      "Epoch 0: validation data loss: 0.005833011082240514, training data loss: 0.0018355793271745955\n",
      "Time for epoch[s]: 0.4657137393951416\n",
      "Epoch 1: validation data loss: 0.005376566750662668, training data loss: 0.00023572427885872976\n",
      "Time for epoch[s]: 0.4514944553375244\n",
      "Epoch 2: validation data loss: 0.004776518140520368, training data loss: 0.000136334342615945\n",
      "Time for epoch[s]: 0.47327756881713867\n",
      "Epoch 3: validation data loss: 0.004433749880109515, training data loss: 0.00010212361812591552\n",
      "Time for epoch[s]: 0.48726916313171387\n",
      "Epoch 4: validation data loss: 0.004345704487391881, training data loss: 8.234414884022303e-05\n",
      "Time for epoch[s]: 0.4009108543395996\n",
      "Epoch 5: validation data loss: 0.004199553625924246, training data loss: 7.057221446718488e-05\n",
      "Time for epoch[s]: 0.39235520362854004\n",
      "Epoch 6: validation data loss: 0.004137655530657087, training data loss: 6.050078783716474e-05\n",
      "Time for epoch[s]: 0.4072573184967041\n",
      "Epoch 7: validation data loss: 0.0038486172812325615, training data loss: 5.4640139852251326e-05\n",
      "Time for epoch[s]: 0.41512322425842285\n",
      "Epoch 8: validation data loss: 0.0038214473724365234, training data loss: 4.9103992325919014e-05\n",
      "Time for epoch[s]: 0.3918154239654541\n",
      "Epoch 9: validation data loss: 0.0037963466644287108, training data loss: 4.451135226658412e-05\n",
      "Time for epoch[s]: 0.3876621723175049\n",
      "Epoch 10: validation data loss: 0.003604072570800781, training data loss: 4.074361068861825e-05\n",
      "Time for epoch[s]: 0.4146387577056885\n",
      "Epoch 11: validation data loss: 0.0034924561636788505, training data loss: 3.7991144827434e-05\n",
      "Time for epoch[s]: 0.41200685501098633\n",
      "Epoch 12: validation data loss: 0.003527862548828125, training data loss: 3.530997676508767e-05\n",
      "Time for epoch[s]: 0.4142136573791504\n",
      "Epoch 13: validation data loss: 0.003384603772844587, training data loss: 3.338563442230225e-05\n",
      "Time for epoch[s]: 0.46420907974243164\n",
      "Epoch 14: validation data loss: 0.003344726017543248, training data loss: 3.1996726989746096e-05\n",
      "Time for epoch[s]: 0.4794769287109375\n",
      "Epoch 15: validation data loss: 0.003364438465663365, training data loss: 2.990806528500148e-05\n",
      "Time for epoch[s]: 0.4105398654937744\n",
      "Epoch 16: validation data loss: 0.0031825719560895646, training data loss: 2.8103406940187726e-05\n",
      "Time for epoch[s]: 0.39943408966064453\n",
      "Epoch 17: validation data loss: 0.0032328248705182755, training data loss: 2.6495288525308882e-05\n",
      "Time for epoch[s]: 0.3908841609954834\n",
      "Epoch 18: validation data loss: 0.0032210576193673272, training data loss: 2.5668150612286158e-05\n",
      "Time for epoch[s]: 0.39919137954711914\n",
      "Epoch 19: validation data loss: 0.003159924370901925, training data loss: 2.4086564779281617e-05\n",
      "Time for epoch[s]: 0.4818265438079834\n",
      "Epoch 20: validation data loss: 0.0030978036608014787, training data loss: 2.341599123818534e-05\n",
      "Time for epoch[s]: 0.4600670337677002\n",
      "Epoch 21: validation data loss: 0.003045166015625, training data loss: 2.2212750145367213e-05\n",
      "Time for epoch[s]: 0.44528627395629883\n",
      "Epoch 22: validation data loss: 0.0030066452026367187, training data loss: 2.128727946962629e-05\n",
      "Time for epoch[s]: 0.42533230781555176\n",
      "Epoch 23: validation data loss: 0.003029665538242885, training data loss: 1.990381521838052e-05\n",
      "Time for epoch[s]: 0.4607231616973877\n",
      "Epoch 24: validation data loss: 0.0029104159218924387, training data loss: 1.961992042405265e-05\n",
      "Time for epoch[s]: 0.4286520481109619\n",
      "Epoch 25: validation data loss: 0.0029627685546875, training data loss: 1.9040678228650773e-05\n",
      "Time for epoch[s]: 0.41148900985717773\n",
      "Epoch 26: validation data loss: 0.002813303538731166, training data loss: 1.8437136496816362e-05\n",
      "Time for epoch[s]: 0.43645358085632324\n",
      "Epoch 27: validation data loss: 0.002902818134852818, training data loss: 1.795161621911185e-05\n",
      "Time for epoch[s]: 0.4361543655395508\n",
      "Epoch 28: validation data loss: 0.0028042567116873603, training data loss: 1.7495824822357724e-05\n",
      "Time for epoch[s]: 0.4093017578125\n",
      "Epoch 29: validation data loss: 0.0027938690185546873, training data loss: 1.6563719936779568e-05\n",
      "Time for epoch[s]: 0.4037511348724365\n",
      "Epoch 30: validation data loss: 0.0027851905822753906, training data loss: 1.5679066734654563e-05\n",
      "Time for epoch[s]: 0.4170539379119873\n",
      "Epoch 31: validation data loss: 0.0028246021270751953, training data loss: 1.5323619757379804e-05\n",
      "Time for epoch[s]: 0.4115312099456787\n",
      "Epoch 32: validation data loss: 0.002690527779715402, training data loss: 1.4979449766022818e-05\n",
      "Time for epoch[s]: 0.41831088066101074\n",
      "Epoch 33: validation data loss: 0.002711484364100865, training data loss: 1.4888060944420951e-05\n",
      "Time for epoch[s]: 0.41959214210510254\n",
      "Epoch 34: validation data loss: 0.002625955309186663, training data loss: 1.4088737113135201e-05\n",
      "Time for epoch[s]: 0.45502638816833496\n",
      "Epoch 35: validation data loss: 0.0026993756975446427, training data loss: 1.3837364103112902e-05\n",
      "Time for epoch[s]: 0.42018580436706543\n",
      "Epoch 36: validation data loss: 0.0025987047467912945, training data loss: 1.3872135962758745e-05\n",
      "Time for epoch[s]: 0.40758180618286133\n",
      "Epoch 37: validation data loss: 0.0026126172201974053, training data loss: 1.282433101109096e-05\n",
      "Time for epoch[s]: 0.43296241760253906\n",
      "Epoch 38: validation data loss: 0.0026369737897600446, training data loss: 1.3291973088468824e-05\n",
      "Time for epoch[s]: 0.4138023853302002\n",
      "Epoch 39: validation data loss: 0.002623125076293945, training data loss: 1.2908597077642168e-05\n",
      "Time for epoch[s]: 0.40144801139831543\n",
      "Epoch 40: validation data loss: 0.0025862671988351005, training data loss: 1.2332271252359663e-05\n",
      "Time for epoch[s]: 0.41210412979125977\n",
      "Epoch 41: validation data loss: 0.0025728705269949777, training data loss: 1.2384135808263507e-05\n",
      "Time for epoch[s]: 0.41452622413635254\n",
      "Epoch 42: validation data loss: 0.0025507425580705914, training data loss: 1.1896021664142608e-05\n",
      "Time for epoch[s]: 0.4043445587158203\n",
      "Epoch 43: validation data loss: 0.0025385556902204243, training data loss: 1.1638239026069641e-05\n",
      "Time for epoch[s]: 0.44728994369506836\n",
      "Epoch 44: validation data loss: 0.002495064871651786, training data loss: 1.0847645146506174e-05\n",
      "Time for epoch[s]: 0.4001610279083252\n",
      "Epoch 45: validation data loss: 0.0023940176282610214, training data loss: 1.1053303522723062e-05\n",
      "Time for epoch[s]: 0.3976900577545166\n",
      "Epoch 46: validation data loss: 0.002516383307320731, training data loss: 1.0702901652881078e-05\n",
      "Time for epoch[s]: 0.3958883285522461\n",
      "Epoch 47: validation data loss: 0.002521176201956613, training data loss: 1.069664316517966e-05\n",
      "Time for epoch[s]: 0.3953554630279541\n",
      "Epoch 48: validation data loss: 0.0024692467280796595, training data loss: 1.029118150472641e-05\n",
      "Time for epoch[s]: 0.388538122177124\n",
      "Epoch 49: validation data loss: 0.0025110645294189452, training data loss: 1.0681752647672381e-05\n",
      "Time for epoch[s]: 0.3918166160583496\n",
      "Epoch 50: validation data loss: 0.0023877620697021484, training data loss: 1.0153899235384805e-05\n",
      "Time for epoch[s]: 0.39596056938171387\n",
      "Epoch 51: validation data loss: 0.0024466446467808316, training data loss: 9.886738445077623e-06\n",
      "Time for epoch[s]: 0.40648460388183594\n",
      "Epoch 52: validation data loss: 0.002418077196393694, training data loss: 9.93635186127254e-06\n",
      "Time for epoch[s]: 0.4027981758117676\n",
      "Epoch 53: validation data loss: 0.0024005385807582312, training data loss: 9.70515183040074e-06\n",
      "Time for epoch[s]: 0.44172000885009766\n",
      "Epoch 54: validation data loss: 0.002433061327253069, training data loss: 9.86452294247491e-06\n",
      "Time for epoch[s]: 0.43831801414489746\n",
      "Epoch 55: validation data loss: 0.002402321951729911, training data loss: 9.123830923012325e-06\n",
      "Time for epoch[s]: 0.4766659736633301\n",
      "Epoch 56: validation data loss: 0.0023772166115897044, training data loss: 8.857053305421556e-06\n",
      "Time for epoch[s]: 0.4653170108795166\n",
      "Epoch 57: validation data loss: 0.002364264896937779, training data loss: 8.876792022160121e-06\n",
      "Time for epoch[s]: 0.48350071907043457\n",
      "Epoch 58: validation data loss: 0.0023037569863455635, training data loss: 8.48681213600295e-06\n",
      "Time for epoch[s]: 0.5122535228729248\n",
      "Epoch 59: validation data loss: 0.002329183850969587, training data loss: 8.730231119053704e-06\n",
      "Time for epoch[s]: 0.40257883071899414\n",
      "Epoch 60: validation data loss: 0.002357107707432338, training data loss: 8.458825094359261e-06\n",
      "Time for epoch[s]: 0.46357274055480957\n",
      "Epoch 61: validation data loss: 0.0023541927337646486, training data loss: 8.280670004231588e-06\n",
      "Time for epoch[s]: 0.4697871208190918\n",
      "Epoch 62: validation data loss: 0.002281731060573033, training data loss: 8.20723335657801e-06\n",
      "Time for epoch[s]: 0.5195093154907227\n",
      "Epoch 63: validation data loss: 0.0023074493408203126, training data loss: 7.9225982938494e-06\n",
      "Time for epoch[s]: 0.5537447929382324\n",
      "Epoch 64: validation data loss: 0.002280211584908622, training data loss: 8.038830544267382e-06\n",
      "Time for epoch[s]: 0.5171189308166504\n",
      "Epoch 65: validation data loss: 0.0022938782828194754, training data loss: 7.717728614807129e-06\n",
      "Time for epoch[s]: 0.4346959590911865\n",
      "Epoch 66: validation data loss: 0.0022992125919887, training data loss: 7.697110729558128e-06\n",
      "Time for epoch[s]: 0.42299914360046387\n",
      "Epoch 67: validation data loss: 0.002294536590576172, training data loss: 7.629804313182831e-06\n",
      "Time for epoch[s]: 0.4227297306060791\n",
      "Epoch 68: validation data loss: 0.0023029417310442245, training data loss: 7.4769705533981325e-06\n",
      "Time for epoch[s]: 0.4237697124481201\n",
      "Epoch 69: validation data loss: 0.0022542433057512554, training data loss: 7.322309272629874e-06\n",
      "Time for epoch[s]: 0.4078378677368164\n",
      "Epoch 70: validation data loss: 0.002208630153111049, training data loss: 7.146569234984262e-06\n",
      "Time for epoch[s]: 0.3967761993408203\n",
      "Epoch 71: validation data loss: 0.0022178340639386856, training data loss: 7.40970777613776e-06\n",
      "Time for epoch[s]: 0.4253885746002197\n",
      "Epoch 72: validation data loss: 0.002247774396623884, training data loss: 6.948984095028468e-06\n",
      "Time for epoch[s]: 0.39454102516174316\n",
      "Epoch 73: validation data loss: 0.002244729859488351, training data loss: 6.7867173680237365e-06\n",
      "Time for epoch[s]: 0.40417909622192383\n",
      "Epoch 74: validation data loss: 0.002222381319318499, training data loss: 6.7781502647059306e-06\n",
      "Time for epoch[s]: 0.3934633731842041\n",
      "Epoch 75: validation data loss: 0.0021965227127075195, training data loss: 6.767006857054574e-06\n",
      "Time for epoch[s]: 0.40454936027526855\n",
      "Epoch 76: validation data loss: 0.00218835939679827, training data loss: 6.662487983703614e-06\n",
      "Time for epoch[s]: 0.4176449775695801\n",
      "Epoch 77: validation data loss: 0.002200015068054199, training data loss: 6.7827808005469185e-06\n",
      "Time for epoch[s]: 0.4229273796081543\n",
      "Epoch 78: validation data loss: 0.0021700425829206194, training data loss: 6.582636386156082e-06\n",
      "Time for epoch[s]: 0.4367680549621582\n",
      "Epoch 79: validation data loss: 0.0021405745915004186, training data loss: 6.204424692051752e-06\n",
      "Time for epoch[s]: 0.4229395389556885\n",
      "Epoch 80: validation data loss: 0.00219405460357666, training data loss: 6.389953728233065e-06\n",
      "Time for epoch[s]: 0.4695601463317871\n",
      "Epoch 81: validation data loss: 0.002186969212123326, training data loss: 6.633773446083069e-06\n",
      "Time for epoch[s]: 0.4239017963409424\n",
      "Epoch 82: validation data loss: 0.002148679869515555, training data loss: 6.268545453037534e-06\n",
      "Time for epoch[s]: 0.41341352462768555\n",
      "Epoch 83: validation data loss: 0.0021630584171840124, training data loss: 6.086444216115134e-06\n",
      "Time for epoch[s]: 0.39042019844055176\n",
      "Epoch 84: validation data loss: 0.0021416803087506977, training data loss: 5.86393049785069e-06\n",
      "Time for epoch[s]: 0.4125545024871826\n",
      "Epoch 85: validation data loss: 0.0021167898178100584, training data loss: 5.731753472770963e-06\n",
      "Time for epoch[s]: 0.39409899711608887\n",
      "Epoch 86: validation data loss: 0.002136255809238979, training data loss: 5.755219076360975e-06\n",
      "Time for epoch[s]: 0.4122598171234131\n",
      "Epoch 87: validation data loss: 0.0021182872227260043, training data loss: 5.7464167475700375e-06\n",
      "Time for epoch[s]: 0.3991880416870117\n",
      "Epoch 88: validation data loss: 0.0021277119772774833, training data loss: 5.716156214475632e-06\n",
      "Time for epoch[s]: 0.39869093894958496\n",
      "Epoch 89: validation data loss: 0.002137368202209473, training data loss: 5.601302321468081e-06\n",
      "Time for epoch[s]: 0.4546189308166504\n",
      "Epoch 90: validation data loss: 0.0021274285997663223, training data loss: 5.6093468197754455e-06\n",
      "Time for epoch[s]: 0.42525219917297363\n",
      "Epoch 91: validation data loss: 0.0021103692735944474, training data loss: 5.693997655596052e-06\n",
      "Time for epoch[s]: 0.45383715629577637\n",
      "Epoch 92: validation data loss: 0.002102403232029506, training data loss: 5.5932109909398215e-06\n",
      "Time for epoch[s]: 0.4276244640350342\n",
      "Epoch 93: validation data loss: 0.0021010849816458566, training data loss: 5.302683583327702e-06\n",
      "Time for epoch[s]: 0.40276503562927246\n",
      "Epoch 94: validation data loss: 0.002100345747811454, training data loss: 5.4391297910894665e-06\n",
      "Time for epoch[s]: 0.4040851593017578\n",
      "Epoch 95: validation data loss: 0.002098571913582938, training data loss: 5.386707506009511e-06\n",
      "Time for epoch[s]: 0.4004859924316406\n",
      "Epoch 96: validation data loss: 0.0021090251377650668, training data loss: 5.266247583287103e-06\n",
      "Time for epoch[s]: 0.39354395866394043\n",
      "Epoch 97: validation data loss: 0.0020634384155273438, training data loss: 5.17364644578525e-06\n",
      "Time for epoch[s]: 0.3999757766723633\n",
      "Epoch 98: validation data loss: 0.0020992469787597654, training data loss: 5.0521535532815115e-06\n",
      "Time for epoch[s]: 0.401090145111084\n",
      "Epoch 99: validation data loss: 0.002057680947440011, training data loss: 5.338533648422786e-06\n",
      "Time for epoch[s]: 0.40978550910949707\n",
      "Final validation data loss:  0.002057680947440011\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot2.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot2.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot3.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot3.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "DataHandler prepared for inference. No training possible with this setup. If this is not what you wanted, please revise the input script. Validation snapshots you may have entered willbe ignored.\n",
      "Consistency check successful.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Had to readjust batch size from 10 to 12\n",
      "{'band_energy': [0.21546747334905803, 0.09541568381050425]}\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 2\n",
    "parameters.manual_seed = 2023\n",
    "parameters.network.layer_sizes = [100]\n",
    "parameters.data.input_rescaling_type = \"feature-wise-standard\"\n",
    "parameters.data.output_rescaling_type = \"normal\"\n",
    "\n",
    "# Sigmoid is default, try out ReLU or LeakyReLU!\n",
    "parameters.network.layer_activations = [\"LeakyReLU\"]\n",
    "\n",
    "\n",
    "parameters, data_handler, network = training(parameters)\n",
    "print(testing(parameters, data_handler, network))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "As you will have noticed, the result seem to vary quite a bit.\n",
    "\n",
    "So what do we make of all of this?\n",
    "We can build different models, and they give different accuracy. But that's also dependent on the data scaling to be used. And that is also dependent on the way we train them. What now?\n",
    "\n",
    "Well, welcome to hell. This complexity is what makes (deep) ML hard, just as the obvious speed-up makes it worthwhile.\n",
    "\n",
    "There are two important concepts at play here:\n",
    "\n",
    "1. Intuition and experience - understanding your data. Your data has inherent structure that can be exploited in several ways. As a rule of thumb activation functions may be chosen in a way that reflects the data itself, i.e. if we have a lot of smooth curves, Sigmoid functions may work well (they do here). LeakyReLU on the other hand is a standard choice for more complicated data. Model size should reflect the complexity in your data set, and a simple data set does not warrant a huge model. We have seen this just now. Training routines should be optimized for the model.\n",
    "2. Good, old fashioned non-linear optimization. Essentially we have a N-dimensional hypersphere with some distinct or float valued choices and can simply test out all the combinations.\n",
    "\n",
    "The best possible route is a combination of the two. By adding as much prior experience from 1. into 2. we can perform a limited optimization and still end up with the best possible model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Tuning the Hyperparameters (hands-on)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "MALA provides a convenient way to tune the hyperparameters of models. There are multiple algorithms implemented, but we will focus on the `optuna` library, to which MALA provides an interface. The idea is easy: we give MALA a set of hyperparameters to optimize, fix the others, give it some data and let optuna do its work. Optuna internally uses elaborate algorithms to determine optimal hyperparameters from observed data.\n",
    "\n",
    "First, let us think about which hyperparameters to fix. We want to fix as many as possible and reasonable to get the best trade-off between optimal results and minimal computational effort.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 2\n",
    "parameters.manual_seed = 2023\n",
    "parameters.data.input_rescaling_type = \"feature-wise-standard\"\n",
    "parameters.data.output_rescaling_type = \"normal\"\n",
    "parameters.running.max_number_epochs = 100\n",
    "parameters.running.mini_batch_size = 40\n",
    "parameters.running.trainingtype = \"SGD\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we can specify data to be used.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n"
     ]
    }
   ],
   "source": [
    "data_handler = mala.DataHandler(parameters)\n",
    "data_handler.clear_data()\n",
    "\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Before we instantiate the hyperparameter optimizer, two important parameters to set are\n",
    "\n",
    "1. The number of trials (test networks) to train\n",
    "2. How often to train a test network - if we train each proposed network a number of times and evaluate the average performant, we can discard unrobust outliers. In the interest of time, let's keep it at 1 here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fiedlerl/.local/lib/python3.10/site-packages/optuna/samplers/_tpe/sampler.py:263: ExperimentalWarning: ``multivariate`` option is an experimental feature. The interface can change in the future.\n",
      "  warnings.warn(\n",
      "\u001b[32m[I 2023-11-09 18:12:31,840]\u001b[0m A new study created in memory with name: no-name-c16579fb-1e4c-49e8-84ae-da2f970bbe68\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "parameters.hyperparameters.n_trials = 5\n",
    "parameters.hyperparameters.number_training_per_trial = 1\n",
    "\n",
    "hyperoptimizer = mala.HyperOptOptuna(parameters, data_handler)\n",
    "\n",
    "hyperoptimizer.add_hyperparameter(\"categorical\", \"learning_rate\",\n",
    "                                     choices=[0.1, 0.2, 0.5])\n",
    "hyperoptimizer.add_hyperparameter(\"categorical\", \"ff_neurons_layer_00\", choices=[10, 100, 200])\n",
    "hyperoptimizer.add_hyperparameter(\"categorical\", \"ff_neurons_layer_01\", choices=[10, 100, 200])\n",
    "hyperoptimizer.add_hyperparameter(\"categorical\", \"early_stopping_epochs\", choices=[4, 8])\n",
    "\n",
    "# Choices for activation function at each layer will be optimized.\n",
    "hyperoptimizer.add_hyperparameter(\"categorical\", \"layer_activation_00\",\n",
    "                                  choices=[\"ReLU\", \"Sigmoid\", \"LeakyReLU\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The syntax is a bit clunky at times, since we are trying to cramp a complicated system into a few function calls. Now let's run. This may take a while. Grab a coffee and sit back :)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Guess - validation data loss:  0.13116421982577947\n",
      "Epoch 0: validation data loss: 0.0031034954606670223, training data loss: 0.008716743286341836\n",
      "Time for epoch[s]: 0.6511399745941162\n",
      "Epoch 1: validation data loss: 0.0025456060557604923, training data loss: 0.0005864466978534716\n",
      "Time for epoch[s]: 0.5954339504241943\n",
      "Epoch 2: validation data loss: 0.0023794136090910054, training data loss: 0.0003730091670332434\n",
      "Time for epoch[s]: 0.5179638862609863\n",
      "Epoch 3: validation data loss: 0.0021896199004290853, training data loss: 0.00028405842035328415\n",
      "Time for epoch[s]: 0.5422563552856445\n",
      "Epoch 4: validation data loss: 0.0021643593703230767, training data loss: 0.0002323468492183511\n",
      "Time for epoch[s]: 0.5443508625030518\n",
      "Epoch 5: validation data loss: 0.0020938209474903264, training data loss: 0.00019799883915409106\n",
      "Time for epoch[s]: 0.5421957969665527\n",
      "Epoch 6: validation data loss: 0.0021044136998860257, training data loss: 0.00017276930210252876\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5464789867401123\n",
      "Epoch 7: validation data loss: 0.0020575311085949204, training data loss: 0.0001533102907546579\n",
      "Time for epoch[s]: 0.5542418956756592\n",
      "Epoch 8: validation data loss: 0.002037269067546548, training data loss: 0.00013851007812371537\n",
      "Time for epoch[s]: 0.5792534351348877\n",
      "Epoch 9: validation data loss: 0.0020569415941630326, training data loss: 0.00012681835658474055\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5926518440246582\n",
      "Epoch 10: validation data loss: 0.002081947783901267, training data loss: 0.00011735446921222286\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6161863803863525\n",
      "Epoch 11: validation data loss: 0.0020356189170384516, training data loss: 0.00010851739170072286\n",
      "Time for epoch[s]: 0.5997252464294434\n",
      "Epoch 12: validation data loss: 0.0020662640055564984, training data loss: 0.00010159840492625214\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5543415546417236\n",
      "Epoch 13: validation data loss: 0.0020529750275285275, training data loss: 9.592618265924933e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5586919784545898\n",
      "Epoch 14: validation data loss: 0.0020433389432898395, training data loss: 9.064494576900517e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5470147132873535\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-11-09 18:12:44,748]\u001b[0m Trial 0 finished with value: 0.0020488141334220156 and parameters: {'learning_rate': 0.2, 'ff_neurons_layer_00': 200, 'ff_neurons_layer_01': 100, 'early_stopping_epochs': 4, 'layer_activation_00': 'ReLU'}. Best is trial 0 with value: 0.0020488141334220156.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15: validation data loss: 0.0020488141334220156, training data loss: 8.603363174553876e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Stopping the training, validation accuracy has not improved for 4 epochs.\n",
      "Final validation data loss:  0.0020488141334220156\n",
      "Initial Guess - validation data loss:  0.06764695851225831\n",
      "Epoch 0: validation data loss: 0.018154446937177823, training data loss: 0.03201883350877457\n",
      "Time for epoch[s]: 0.4913489818572998\n",
      "Epoch 1: validation data loss: 0.00959580674019034, training data loss: 0.009023511246459125\n",
      "Time for epoch[s]: 0.6443936824798584\n",
      "Epoch 2: validation data loss: 0.007900271241523359, training data loss: 0.004891595884000874\n",
      "Time for epoch[s]: 0.5373954772949219\n",
      "Epoch 3: validation data loss: 0.007445743639175206, training data loss: 0.004006068183951182\n",
      "Time for epoch[s]: 0.6244256496429443\n",
      "Epoch 4: validation data loss: 0.007277229605200084, training data loss: 0.0037422215557533856\n",
      "Time for epoch[s]: 0.5407280921936035\n",
      "Epoch 5: validation data loss: 0.007193303543683057, training data loss: 0.0036367082704692128\n",
      "Time for epoch[s]: 0.4772169589996338\n",
      "Epoch 6: validation data loss: 0.007140714283947531, training data loss: 0.0035832385494284436\n",
      "Time for epoch[s]: 0.48340868949890137\n",
      "Epoch 7: validation data loss: 0.0071003404382157, training data loss: 0.0035486014466307478\n",
      "Time for epoch[s]: 0.4551420211791992\n",
      "Epoch 8: validation data loss: 0.007065751781202343, training data loss: 0.0035230843988183426\n",
      "Time for epoch[s]: 0.4503598213195801\n",
      "Epoch 9: validation data loss: 0.007033767765515471, training data loss: 0.003496989539769142\n",
      "Time for epoch[s]: 0.5479869842529297\n",
      "Epoch 10: validation data loss: 0.007002812542327463, training data loss: 0.003477328988515079\n",
      "Time for epoch[s]: 0.5402224063873291\n",
      "Epoch 11: validation data loss: 0.0069716782330378, training data loss: 0.0034548398022237977\n",
      "Time for epoch[s]: 0.5238113403320312\n",
      "Epoch 12: validation data loss: 0.006939253306279988, training data loss: 0.003441198775757393\n",
      "Time for epoch[s]: 0.4687056541442871\n",
      "Epoch 13: validation data loss: 0.006906626975699647, training data loss: 0.00341400762671205\n",
      "Time for epoch[s]: 0.4422295093536377\n",
      "Epoch 14: validation data loss: 0.006872167870334294, training data loss: 0.0033925583917800693\n",
      "Time for epoch[s]: 0.4524726867675781\n",
      "Epoch 15: validation data loss: 0.006836669630111624, training data loss: 0.0033720086698662746\n",
      "Time for epoch[s]: 0.5852680206298828\n",
      "Epoch 16: validation data loss: 0.006800310796798636, training data loss: 0.0033438412566163225\n",
      "Time for epoch[s]: 0.5031101703643799\n",
      "Epoch 17: validation data loss: 0.006761115435595926, training data loss: 0.0033184434180934677\n",
      "Time for epoch[s]: 0.4404726028442383\n",
      "Epoch 18: validation data loss: 0.006721571155879051, training data loss: 0.0032943394630466968\n",
      "Time for epoch[s]: 0.4284367561340332\n",
      "Epoch 19: validation data loss: 0.006679680249462389, training data loss: 0.0032704135054322683\n",
      "Time for epoch[s]: 0.4558722972869873\n",
      "Epoch 20: validation data loss: 0.006636493282231022, training data loss: 0.0032411130595969284\n",
      "Time for epoch[s]: 0.4502232074737549\n",
      "Epoch 21: validation data loss: 0.006591149116759975, training data loss: 0.0032148810282145463\n",
      "Time for epoch[s]: 0.45380401611328125\n",
      "Epoch 22: validation data loss: 0.006543132812465163, training data loss: 0.003184035488459618\n",
      "Time for epoch[s]: 0.5014476776123047\n",
      "Epoch 23: validation data loss: 0.00649350527759012, training data loss: 0.0031540886452208917\n",
      "Time for epoch[s]: 0.5205132961273193\n",
      "Epoch 24: validation data loss: 0.0064419284803137935, training data loss: 0.0031220278783475972\n",
      "Time for epoch[s]: 0.5353336334228516\n",
      "Epoch 25: validation data loss: 0.0063883550635211546, training data loss: 0.00308879733629967\n",
      "Time for epoch[s]: 0.5128490924835205\n",
      "Epoch 26: validation data loss: 0.0063326658179226536, training data loss: 0.0030561456941578485\n",
      "Time for epoch[s]: 0.5185747146606445\n",
      "Epoch 27: validation data loss: 0.006274739901224772, training data loss: 0.0030222978766106033\n",
      "Time for epoch[s]: 0.5176792144775391\n",
      "Epoch 28: validation data loss: 0.0062149452836546175, training data loss: 0.002990730004767849\n",
      "Time for epoch[s]: 0.6107742786407471\n",
      "Epoch 29: validation data loss: 0.006152777366986557, training data loss: 0.00294991007678585\n",
      "Time for epoch[s]: 0.6304895877838135\n",
      "Epoch 30: validation data loss: 0.0060885262815919645, training data loss: 0.002912422021230062\n",
      "Time for epoch[s]: 0.5919804573059082\n",
      "Epoch 31: validation data loss: 0.006022442965747015, training data loss: 0.0028728763806765483\n",
      "Time for epoch[s]: 0.5503263473510742\n",
      "Epoch 32: validation data loss: 0.005955028751669409, training data loss: 0.002833974688020471\n",
      "Time for epoch[s]: 0.5151035785675049\n",
      "Epoch 33: validation data loss: 0.005885507962475084, training data loss: 0.0027936411230531457\n",
      "Time for epoch[s]: 0.6378355026245117\n",
      "Epoch 34: validation data loss: 0.005814205021618708, training data loss: 0.002752698175439007\n",
      "Time for epoch[s]: 0.5865340232849121\n",
      "Epoch 35: validation data loss: 0.005741676783452839, training data loss: 0.0027121678334937247\n",
      "Time for epoch[s]: 0.5802388191223145\n",
      "Epoch 36: validation data loss: 0.0056677125904658066, training data loss: 0.002669048635926965\n",
      "Time for epoch[s]: 0.5008304119110107\n",
      "Epoch 37: validation data loss: 0.005593612858149559, training data loss: 0.0026267209009492778\n",
      "Time for epoch[s]: 0.5209047794342041\n",
      "Epoch 38: validation data loss: 0.005518313956587282, training data loss: 0.002586382981304709\n",
      "Time for epoch[s]: 0.5187773704528809\n",
      "Epoch 39: validation data loss: 0.005442576321292685, training data loss: 0.0025460755444008465\n",
      "Time for epoch[s]: 0.5171799659729004\n",
      "Epoch 40: validation data loss: 0.005366088592842834, training data loss: 0.002498993046207515\n",
      "Time for epoch[s]: 0.5307419300079346\n",
      "Epoch 41: validation data loss: 0.0052891811823736045, training data loss: 0.0024568329663037167\n",
      "Time for epoch[s]: 0.5261807441711426\n",
      "Epoch 42: validation data loss: 0.005213459877118672, training data loss: 0.002414265179742961\n",
      "Time for epoch[s]: 0.5170669555664062\n",
      "Epoch 43: validation data loss: 0.005137458783850822, training data loss: 0.0023715991407768913\n",
      "Time for epoch[s]: 0.5309159755706787\n",
      "Epoch 44: validation data loss: 0.005061110405072774, training data loss: 0.002331655047255564\n",
      "Time for epoch[s]: 0.5582108497619629\n",
      "Epoch 45: validation data loss: 0.004986657399565117, training data loss: 0.0022902423388337437\n",
      "Time for epoch[s]: 0.6087543964385986\n",
      "Epoch 46: validation data loss: 0.004912516297815053, training data loss: 0.0022471835624137426\n",
      "Time for epoch[s]: 0.5431778430938721\n",
      "Epoch 47: validation data loss: 0.004838410577817595, training data loss: 0.002208124965293222\n",
      "Time for epoch[s]: 0.5326943397521973\n",
      "Epoch 48: validation data loss: 0.00476644953636274, training data loss: 0.0021682035977437617\n",
      "Time for epoch[s]: 0.5152254104614258\n",
      "Epoch 49: validation data loss: 0.004695481905654141, training data loss: 0.002130613343356407\n",
      "Time for epoch[s]: 0.6402721405029297\n",
      "Epoch 50: validation data loss: 0.0046251010677041526, training data loss: 0.0020915234469931964\n",
      "Time for epoch[s]: 0.5410888195037842\n",
      "Epoch 51: validation data loss: 0.004557477829118842, training data loss: 0.0020539914364139785\n",
      "Time for epoch[s]: 0.4322991371154785\n",
      "Epoch 52: validation data loss: 0.004491719208895888, training data loss: 0.0020189271669953926\n",
      "Time for epoch[s]: 0.4508047103881836\n",
      "Epoch 53: validation data loss: 0.004426627942960556, training data loss: 0.001981837700491082\n",
      "Time for epoch[s]: 0.43065595626831055\n",
      "Epoch 54: validation data loss: 0.0043640980437465995, training data loss: 0.0019468720917288026\n",
      "Time for epoch[s]: 0.44399356842041016\n",
      "Epoch 55: validation data loss: 0.004302953476230848, training data loss: 0.001913702787329617\n",
      "Time for epoch[s]: 0.45174622535705566\n",
      "Epoch 56: validation data loss: 0.004242750335501754, training data loss: 0.0018826731263774715\n",
      "Time for epoch[s]: 0.4490933418273926\n",
      "Epoch 57: validation data loss: 0.00418451632538887, training data loss: 0.0018485290546939798\n",
      "Time for epoch[s]: 0.41474080085754395\n",
      "Epoch 58: validation data loss: 0.004130212955823227, training data loss: 0.0018176154731071158\n",
      "Time for epoch[s]: 0.5782134532928467\n",
      "Epoch 59: validation data loss: 0.00407496434912834, training data loss: 0.0017876710793743396\n",
      "Time for epoch[s]: 0.5958378314971924\n",
      "Epoch 60: validation data loss: 0.004022502463702197, training data loss: 0.0017580622679566685\n",
      "Time for epoch[s]: 0.6732819080352783\n",
      "Epoch 61: validation data loss: 0.003971949015578178, training data loss: 0.0017293914812340583\n",
      "Time for epoch[s]: 0.5429055690765381\n",
      "Epoch 62: validation data loss: 0.003924073421791808, training data loss: 0.0017058130812971559\n",
      "Time for epoch[s]: 0.5209534168243408\n",
      "Epoch 63: validation data loss: 0.003876628396717925, training data loss: 0.0016747915853648424\n",
      "Time for epoch[s]: 0.5225543975830078\n",
      "Epoch 64: validation data loss: 0.003832553619663465, training data loss: 0.0016485858181295875\n",
      "Time for epoch[s]: 0.554157018661499\n",
      "Epoch 65: validation data loss: 0.003787875447643402, training data loss: 0.0016243367978971298\n",
      "Time for epoch[s]: 0.5916085243225098\n",
      "Epoch 66: validation data loss: 0.0037480432148937763, training data loss: 0.0016000266760995943\n",
      "Time for epoch[s]: 0.545297384262085\n",
      "Epoch 67: validation data loss: 0.003705742968816191, training data loss: 0.0015773403045793648\n",
      "Time for epoch[s]: 0.7266905307769775\n",
      "Epoch 68: validation data loss: 0.0036683430954745915, training data loss: 0.0015545396227814835\n",
      "Time for epoch[s]: 0.6723289489746094\n",
      "Epoch 69: validation data loss: 0.0036297549395800724, training data loss: 0.0015315111369302828\n",
      "Time for epoch[s]: 0.5640959739685059\n",
      "Epoch 70: validation data loss: 0.0035927902617955317, training data loss: 0.0015091879727089242\n",
      "Time for epoch[s]: 0.570183277130127\n",
      "Epoch 71: validation data loss: 0.003558790302712079, training data loss: 0.0014888242771636405\n",
      "Time for epoch[s]: 0.5883481502532959\n",
      "Epoch 72: validation data loss: 0.0035257905585580765, training data loss: 0.0014697121430749763\n",
      "Time for epoch[s]: 0.5523066520690918\n",
      "Epoch 73: validation data loss: 0.0034932605752117556, training data loss: 0.0014486348247963543\n",
      "Time for epoch[s]: 0.5567233562469482\n",
      "Epoch 74: validation data loss: 0.0034638055383342586, training data loss: 0.001430273464281265\n",
      "Time for epoch[s]: 0.6528110504150391\n",
      "Epoch 75: validation data loss: 0.0034361397294693343, training data loss: 0.0014116274167413582\n",
      "Time for epoch[s]: 0.6734697818756104\n",
      "Epoch 76: validation data loss: 0.0034054871563497743, training data loss: 0.0013940437743652902\n",
      "Time for epoch[s]: 0.7480058670043945\n",
      "Epoch 77: validation data loss: 0.00337857089630545, training data loss: 0.0013766471109433805\n",
      "Time for epoch[s]: 0.7112722396850586\n",
      "Epoch 78: validation data loss: 0.0033516559970977644, training data loss: 0.001360128595404429\n",
      "Time for epoch[s]: 0.7169003486633301\n",
      "Epoch 79: validation data loss: 0.003327084458581933, training data loss: 0.0013440188751917452\n",
      "Time for epoch[s]: 0.6988859176635742\n",
      "Epoch 80: validation data loss: 0.0033007615233120852, training data loss: 0.0013285175850402274\n",
      "Time for epoch[s]: 0.7612326145172119\n",
      "Epoch 81: validation data loss: 0.0032773053265053387, training data loss: 0.0013132157935399442\n",
      "Time for epoch[s]: 0.842118501663208\n",
      "Epoch 82: validation data loss: 0.003255711026387672, training data loss: 0.0012976436037995499\n",
      "Time for epoch[s]: 0.5931429862976074\n",
      "Epoch 83: validation data loss: 0.0032335010837746536, training data loss: 0.001288002076214307\n",
      "Time for epoch[s]: 0.6165637969970703\n",
      "Epoch 84: validation data loss: 0.003210508660094379, training data loss: 0.0012696641220894035\n",
      "Time for epoch[s]: 0.5982356071472168\n",
      "Epoch 85: validation data loss: 0.0031911028574590815, training data loss: 0.0012619500835192258\n",
      "Time for epoch[s]: 0.5193099975585938\n",
      "Epoch 86: validation data loss: 0.0031697883453543326, training data loss: 0.0012434224287668865\n",
      "Time for epoch[s]: 0.5109403133392334\n",
      "Epoch 87: validation data loss: 0.0031492971394160023, training data loss: 0.001230488901268946\n",
      "Time for epoch[s]: 0.6999602317810059\n",
      "Epoch 88: validation data loss: 0.003130416619723246, training data loss: 0.0012176051803919822\n",
      "Time for epoch[s]: 0.7372949123382568\n",
      "Epoch 89: validation data loss: 0.003112889588151348, training data loss: 0.0012055962053063797\n",
      "Time for epoch[s]: 0.7755959033966064\n",
      "Epoch 90: validation data loss: 0.0030939266017582863, training data loss: 0.001194071960231485\n",
      "Time for epoch[s]: 0.637197732925415\n",
      "Epoch 91: validation data loss: 0.003077777281199416, training data loss: 0.0011828032802773394\n",
      "Time for epoch[s]: 0.5688583850860596\n",
      "Epoch 92: validation data loss: 0.0030606206693605744, training data loss: 0.0011714887401284692\n",
      "Time for epoch[s]: 0.5559542179107666\n",
      "Epoch 93: validation data loss: 0.0030433021179617267, training data loss: 0.0011605561868240845\n",
      "Time for epoch[s]: 0.5320560932159424\n",
      "Epoch 94: validation data loss: 0.003029237871300684, training data loss: 0.0011502554699710514\n",
      "Time for epoch[s]: 0.6578543186187744\n",
      "Epoch 95: validation data loss: 0.003012523259202095, training data loss: 0.0011396719712644951\n",
      "Time for epoch[s]: 0.6498537063598633\n",
      "Epoch 96: validation data loss: 0.0029985820321731917, training data loss: 0.0011292952516851905\n",
      "Time for epoch[s]: 0.6704690456390381\n",
      "Epoch 97: validation data loss: 0.0029816344448420555, training data loss: 0.0011274549242568342\n",
      "Time for epoch[s]: 0.5300207138061523\n",
      "Epoch 98: validation data loss: 0.002968095753290882, training data loss: 0.00111012841196365\n",
      "Time for epoch[s]: 0.5124051570892334\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-11-09 18:13:40,927]\u001b[0m Trial 1 finished with value: 0.002954056001689336 and parameters: {'learning_rate': 0.1, 'ff_neurons_layer_00': 100, 'ff_neurons_layer_01': 10, 'early_stopping_epochs': 4, 'layer_activation_00': 'Sigmoid'}. Best is trial 0 with value: 0.0020488141334220156.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 99: validation data loss: 0.002954056001689336, training data loss: 0.001100732243224366\n",
      "Time for epoch[s]: 0.47902488708496094\n",
      "Final validation data loss:  0.002954056001689336\n",
      "Initial Guess - validation data loss:  0.13168105904914473\n",
      "Epoch 0: validation data loss: 0.003099715600819348, training data loss: 0.00836452555983034\n",
      "Time for epoch[s]: 0.523815393447876\n",
      "Epoch 1: validation data loss: 0.002546436982612087, training data loss: 0.0005816882603789029\n",
      "Time for epoch[s]: 0.5319154262542725\n",
      "Epoch 2: validation data loss: 0.0023768721105845553, training data loss: 0.0003706208306904797\n",
      "Time for epoch[s]: 0.5085389614105225\n",
      "Epoch 3: validation data loss: 0.0021895511781788307, training data loss: 0.00028284489428071673\n",
      "Time for epoch[s]: 0.5293693542480469\n",
      "Epoch 4: validation data loss: 0.002161558360269625, training data loss: 0.00023146063837830878\n",
      "Time for epoch[s]: 0.5106923580169678\n",
      "Epoch 5: validation data loss: 0.0020894713053420254, training data loss: 0.000197194482637867\n",
      "Time for epoch[s]: 0.516183614730835\n",
      "Epoch 6: validation data loss: 0.0021021479068825777, training data loss: 0.0001720103124777476\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5034875869750977\n",
      "Epoch 7: validation data loss: 0.002053712873154035, training data loss: 0.00015265523502815804\n",
      "Time for epoch[s]: 0.4984917640686035\n",
      "Epoch 8: validation data loss: 0.002034600466898043, training data loss: 0.00013792213715919076\n",
      "Time for epoch[s]: 0.5128946304321289\n",
      "Epoch 9: validation data loss: 0.0020566205727999614, training data loss: 0.00012626834043629094\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5059020519256592\n",
      "Epoch 10: validation data loss: 0.0020808887808290246, training data loss: 0.00011681674311966657\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.499908447265625\n",
      "Epoch 11: validation data loss: 0.002037320779338819, training data loss: 0.00010792664401063091\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5109331607818604\n",
      "Epoch 12: validation data loss: 0.002067057373316865, training data loss: 0.00010102339190979526\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5201117992401123\n",
      "Epoch 13: validation data loss: 0.0020553451967021646, training data loss: 9.537249225187519e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5274057388305664\n",
      "Epoch 14: validation data loss: 0.002047660688287047, training data loss: 9.010546759927654e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5216174125671387\n",
      "Epoch 15: validation data loss: 0.0020494456977060396, training data loss: 8.555869125340083e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5525126457214355\n",
      "Epoch 16: validation data loss: 0.0020338189384164333, training data loss: 8.13069613012549e-05\n",
      "Time for epoch[s]: 0.6041028499603271\n",
      "Epoch 17: validation data loss: 0.00203055592432414, training data loss: 7.762742062954054e-05\n",
      "Time for epoch[s]: 0.584958553314209\n",
      "Epoch 18: validation data loss: 0.0020588469015408867, training data loss: 7.408596235051002e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6082956790924072\n",
      "Epoch 19: validation data loss: 0.0020583429837335737, training data loss: 7.126737624134648e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.642977237701416\n",
      "Epoch 20: validation data loss: 0.0020343702133387735, training data loss: 6.848772707050794e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6185600757598877\n",
      "Epoch 21: validation data loss: 0.0020403886494571215, training data loss: 6.600615215492031e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6081562042236328\n",
      "Epoch 22: validation data loss: 0.0020057606642649053, training data loss: 6.395955591305206e-05\n",
      "Time for epoch[s]: 0.5933175086975098\n",
      "Epoch 23: validation data loss: 0.0019917331601931078, training data loss: 6.152293113268674e-05\n",
      "Time for epoch[s]: 0.6001570224761963\n",
      "Epoch 24: validation data loss: 0.002001547241864139, training data loss: 6.000463293703724e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6214501857757568\n",
      "Epoch 25: validation data loss: 0.002035518623378179, training data loss: 5.809403105413533e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5986809730529785\n",
      "Epoch 26: validation data loss: 0.00200639821623014, training data loss: 5.639767163692544e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6014642715454102\n",
      "Epoch 27: validation data loss: 0.001999645880912537, training data loss: 5.493012839528524e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6191306114196777\n",
      "Epoch 28: validation data loss: 0.001987141425206781, training data loss: 5.3165608072934084e-05\n",
      "Time for epoch[s]: 0.595369815826416\n",
      "Epoch 29: validation data loss: 0.0020238171973729242, training data loss: 5.171982545966971e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6180059909820557\n",
      "Epoch 30: validation data loss: 0.0019935858031930446, training data loss: 5.066095421847688e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6009011268615723\n",
      "Epoch 31: validation data loss: 0.0019864502562779813, training data loss: 4.9526871237308466e-05\n",
      "Time for epoch[s]: 0.5919437408447266\n",
      "Epoch 32: validation data loss: 0.0019998011523730133, training data loss: 4.811766536132386e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5954115390777588\n",
      "Epoch 33: validation data loss: 0.0020057878809976795, training data loss: 4.720754206997074e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6059648990631104\n",
      "Epoch 34: validation data loss: 0.00199430881569919, training data loss: 4.629274940790107e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.7087883949279785\n",
      "Epoch 35: validation data loss: 0.0019898191434607657, training data loss: 4.50898718820315e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.7620790004730225\n",
      "Epoch 36: validation data loss: 0.00197956755281039, training data loss: 4.432958520982908e-05\n",
      "Time for epoch[s]: 0.6302220821380615\n",
      "Epoch 37: validation data loss: 0.001963493078266649, training data loss: 4.337363149205299e-05\n",
      "Time for epoch[s]: 0.5920031070709229\n",
      "Epoch 38: validation data loss: 0.001963542884887626, training data loss: 4.23052343943892e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6069679260253906\n",
      "Epoch 39: validation data loss: 0.001943906692609395, training data loss: 4.152698042594135e-05\n",
      "Time for epoch[s]: 0.6137845516204834\n",
      "Epoch 40: validation data loss: 0.002007439256258751, training data loss: 4.0777980192611206e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6061155796051025\n",
      "Epoch 41: validation data loss: 0.0019603306300019566, training data loss: 3.9902753855811954e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5973978042602539\n",
      "Epoch 42: validation data loss: 0.001943547703904104, training data loss: 3.9341834003794684e-05\n",
      "Time for epoch[s]: 0.5971686840057373\n",
      "Epoch 43: validation data loss: 0.0019493255441047285, training data loss: 3.85015939264537e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5357890129089355\n",
      "Epoch 44: validation data loss: 0.001969440750879784, training data loss: 3.7989196405835346e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5219237804412842\n",
      "Epoch 45: validation data loss: 0.0019823322285255885, training data loss: 3.7369131191408256e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5300281047821045\n",
      "Epoch 46: validation data loss: 0.001955951185531268, training data loss: 3.6781891908275486e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5248682498931885\n",
      "Epoch 47: validation data loss: 0.00195774109396216, training data loss: 3.616522453283066e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5039510726928711\n",
      "Epoch 48: validation data loss: 0.001936141486581602, training data loss: 3.557973520056298e-05\n",
      "Time for epoch[s]: 0.5197350978851318\n",
      "Epoch 49: validation data loss: 0.0019141812574917868, training data loss: 3.509965967642118e-05\n",
      "Time for epoch[s]: 0.5082592964172363\n",
      "Epoch 50: validation data loss: 0.0019263554381453284, training data loss: 3.45354334166333e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5039222240447998\n",
      "Epoch 51: validation data loss: 0.0019278342593206119, training data loss: 3.4090522761758605e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5875208377838135\n",
      "Epoch 52: validation data loss: 0.0019166220540869725, training data loss: 3.3451465370992544e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5991160869598389\n",
      "Epoch 53: validation data loss: 0.0019201689387020999, training data loss: 3.314234069697389e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5905048847198486\n",
      "Epoch 54: validation data loss: 0.0019588343901176977, training data loss: 3.263853133133013e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6061022281646729\n",
      "Epoch 55: validation data loss: 0.0019394869673742007, training data loss: 3.226836888145094e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5817379951477051\n",
      "Epoch 56: validation data loss: 0.001911586550272763, training data loss: 3.19191122844339e-05\n",
      "Time for epoch[s]: 0.5777873992919922\n",
      "Epoch 57: validation data loss: 0.0019029397942704153, training data loss: 3.137295538363936e-05\n",
      "Time for epoch[s]: 0.6001830101013184\n",
      "Epoch 58: validation data loss: 0.0019185058602459355, training data loss: 3.101973747621932e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6189913749694824\n",
      "Epoch 59: validation data loss: 0.001922521144832106, training data loss: 3.0580586987679405e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6155481338500977\n",
      "Epoch 60: validation data loss: 0.0019124069986822397, training data loss: 3.033101380075494e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6221959590911865\n",
      "Epoch 61: validation data loss: 0.001901301619124739, training data loss: 2.988405338569319e-05\n",
      "Time for epoch[s]: 0.6030392646789551\n",
      "Epoch 62: validation data loss: 0.0019035834700005239, training data loss: 2.9602328306872006e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6814501285552979\n",
      "Epoch 63: validation data loss: 0.0019043166887814597, training data loss: 2.922953986754156e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.8391814231872559\n",
      "Epoch 64: validation data loss: 0.001903819166906348, training data loss: 2.892061719271146e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6587412357330322\n",
      "Epoch 65: validation data loss: 0.0019006971354898253, training data loss: 2.8629727729651482e-05\n",
      "Time for epoch[s]: 0.713263750076294\n",
      "Epoch 66: validation data loss: 0.001908606862368649, training data loss: 2.8362682676070358e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6221950054168701\n",
      "Epoch 67: validation data loss: 0.0018927562454519751, training data loss: 2.7971625004865262e-05\n",
      "Time for epoch[s]: 0.5360894203186035\n",
      "Epoch 68: validation data loss: 0.0019071257277710797, training data loss: 2.758197920167283e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5917088985443115\n",
      "Epoch 69: validation data loss: 0.0018926153988598688, training data loss: 2.7470975331792004e-05\n",
      "Time for epoch[s]: 0.7733423709869385\n",
      "Epoch 70: validation data loss: 0.0018950522490288025, training data loss: 2.70897730946813e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.7112016677856445\n",
      "Epoch 71: validation data loss: 0.0019152175345921625, training data loss: 2.6770408124956366e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.7226719856262207\n",
      "Epoch 72: validation data loss: 0.0018922998208433526, training data loss: 2.6521185778727816e-05\n",
      "Time for epoch[s]: 0.6274263858795166\n",
      "Epoch 73: validation data loss: 0.0018973336916535957, training data loss: 2.632174879040348e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6514360904693604\n",
      "Epoch 74: validation data loss: 0.0018847233628573484, training data loss: 2.6030338382067744e-05\n",
      "Time for epoch[s]: 0.5230488777160645\n",
      "Epoch 75: validation data loss: 0.0018947261925701682, training data loss: 2.5810110361472658e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6360354423522949\n",
      "Epoch 76: validation data loss: 0.0018893578281141308, training data loss: 2.5551389800767375e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6918730735778809\n",
      "Epoch 77: validation data loss: 0.0018939420784989448, training data loss: 2.531895465026163e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6554253101348877\n",
      "Epoch 78: validation data loss: 0.0018921913621632476, training data loss: 2.5072943027977528e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5301539897918701\n",
      "Epoch 79: validation data loss: 0.0018906474929966338, training data loss: 2.482810937990881e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.624453067779541\n",
      "Epoch 80: validation data loss: 0.001887822532218341, training data loss: 2.464895523642296e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5219039916992188\n",
      "Epoch 81: validation data loss: 0.0018902573411323164, training data loss: 2.442450649594063e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5820083618164062\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-11-09 18:14:30,262]\u001b[0m Trial 2 finished with value: 0.001890825218261649 and parameters: {'learning_rate': 0.2, 'ff_neurons_layer_00': 200, 'ff_neurons_layer_01': 100, 'early_stopping_epochs': 8, 'layer_activation_00': 'LeakyReLU'}. Best is trial 2 with value: 0.001890825218261649.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 82: validation data loss: 0.001890825218261649, training data loss: 2.424158666231861e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Stopping the training, validation accuracy has not improved for 8 epochs.\n",
      "Final validation data loss:  0.001890825218261649\n",
      "Initial Guess - validation data loss:  0.1280695684424274\n",
      "Epoch 0: validation data loss: 0.026301059548713302, training data loss: 0.058791334770585846\n",
      "Time for epoch[s]: 0.5681591033935547\n",
      "Epoch 1: validation data loss: 0.019513974995373592, training data loss: 0.017112139697488586\n",
      "Time for epoch[s]: 0.655630350112915\n",
      "Epoch 2: validation data loss: 0.007752214936905256, training data loss: 0.013614544585415217\n",
      "Time for epoch[s]: 0.7709591388702393\n",
      "Epoch 3: validation data loss: 0.004707333159773317, training data loss: 0.0022906606600164824\n",
      "Time for epoch[s]: 0.6728386878967285\n",
      "Epoch 4: validation data loss: 0.00394625043215817, training data loss: 0.0012456377347310383\n",
      "Time for epoch[s]: 0.5950474739074707\n",
      "Epoch 5: validation data loss: 0.0034004180398705887, training data loss: 0.0009438537027193531\n",
      "Time for epoch[s]: 0.501873254776001\n",
      "Epoch 6: validation data loss: 0.0030304087351446284, training data loss: 0.0007554049769492999\n",
      "Time for epoch[s]: 0.4905097484588623\n",
      "Epoch 7: validation data loss: 0.002806793609166254, training data loss: 0.0006076572420390229\n",
      "Time for epoch[s]: 0.5924479961395264\n",
      "Epoch 8: validation data loss: 0.002616472440223171, training data loss: 0.0004984356769143719\n",
      "Time for epoch[s]: 0.4427628517150879\n",
      "Epoch 9: validation data loss: 0.00243860564819754, training data loss: 0.0004168272902976432\n",
      "Time for epoch[s]: 0.41466712951660156\n",
      "Epoch 10: validation data loss: 0.002274291700424125, training data loss: 0.0003560352815340643\n",
      "Time for epoch[s]: 0.4185507297515869\n",
      "Epoch 11: validation data loss: 0.0021630273834211096, training data loss: 0.00031237096683075436\n",
      "Time for epoch[s]: 0.3819143772125244\n",
      "Epoch 12: validation data loss: 0.0020947024974648813, training data loss: 0.0002799715047285437\n",
      "Time for epoch[s]: 0.4123833179473877\n",
      "Epoch 13: validation data loss: 0.0019746194147083856, training data loss: 0.0002558973349937021\n",
      "Time for epoch[s]: 0.4361429214477539\n",
      "Epoch 14: validation data loss: 0.0019323457049452552, training data loss: 0.0002367326556003257\n",
      "Time for epoch[s]: 0.43689465522766113\n",
      "Epoch 15: validation data loss: 0.001855532736538752, training data loss: 0.00022200899853553946\n",
      "Time for epoch[s]: 0.5433006286621094\n",
      "Epoch 16: validation data loss: 0.0018002513336808715, training data loss: 0.00020871467922376172\n",
      "Time for epoch[s]: 0.6700704097747803\n",
      "Epoch 17: validation data loss: 0.0017652896713448443, training data loss: 0.0001981688416711816\n",
      "Time for epoch[s]: 0.6165821552276611\n",
      "Epoch 18: validation data loss: 0.0017439067091571687, training data loss: 0.00018930190230069096\n",
      "Time for epoch[s]: 0.5322561264038086\n",
      "Epoch 19: validation data loss: 0.0016775442857176201, training data loss: 0.00018111786477641972\n",
      "Time for epoch[s]: 0.5232944488525391\n",
      "Epoch 20: validation data loss: 0.0016865451314133596, training data loss: 0.00017293494994237543\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5238265991210938\n",
      "Epoch 21: validation data loss: 0.0016400042462022338, training data loss: 0.00016582337076261164\n",
      "Time for epoch[s]: 0.48128747940063477\n",
      "Epoch 22: validation data loss: 0.0016224881013234456, training data loss: 0.00015982837505536536\n",
      "Time for epoch[s]: 0.6119458675384521\n",
      "Epoch 23: validation data loss: 0.0015908496020591422, training data loss: 0.00015340110958983366\n",
      "Time for epoch[s]: 0.5216264724731445\n",
      "Epoch 24: validation data loss: 0.001549896992504869, training data loss: 0.00014762064977867962\n",
      "Time for epoch[s]: 0.5410563945770264\n",
      "Epoch 25: validation data loss: 0.0015567444503035174, training data loss: 0.00014181826348718443\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6329512596130371\n",
      "Epoch 26: validation data loss: 0.0015347797304527944, training data loss: 0.00013700760390660533\n",
      "Time for epoch[s]: 0.5661830902099609\n",
      "Epoch 27: validation data loss: 0.00150409218383162, training data loss: 0.00013237366597402043\n",
      "Time for epoch[s]: 0.48776960372924805\n",
      "Epoch 28: validation data loss: 0.00148442999957359, training data loss: 0.00012909455148324574\n",
      "Time for epoch[s]: 0.5041067600250244\n",
      "Epoch 29: validation data loss: 0.0014765020374838075, training data loss: 0.000125125850036264\n",
      "Time for epoch[s]: 0.508075475692749\n",
      "Epoch 30: validation data loss: 0.0014543289735436983, training data loss: 0.00012116432700255145\n",
      "Time for epoch[s]: 0.5015180110931396\n",
      "Epoch 31: validation data loss: 0.0014294851316164617, training data loss: 0.00011827920873959859\n",
      "Time for epoch[s]: 0.4860665798187256\n",
      "Epoch 32: validation data loss: 0.0014148302818542203, training data loss: 0.00011524721367718423\n",
      "Time for epoch[s]: 0.5027284622192383\n",
      "Epoch 33: validation data loss: 0.0014109038599005573, training data loss: 0.00011250494073515069\n",
      "Time for epoch[s]: 0.4758028984069824\n",
      "Epoch 34: validation data loss: 0.0014060210419572106, training data loss: 0.00010961470232434469\n",
      "Time for epoch[s]: 0.5029253959655762\n",
      "Epoch 35: validation data loss: 0.001377213082901419, training data loss: 0.0001073996855379784\n",
      "Time for epoch[s]: 0.47477030754089355\n",
      "Epoch 36: validation data loss: 0.0013803121161787477, training data loss: 0.00010499803000661336\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.488785982131958\n",
      "Epoch 37: validation data loss: 0.0013538348620340704, training data loss: 0.00010281568180480503\n",
      "Time for epoch[s]: 0.4678921699523926\n",
      "Epoch 38: validation data loss: 0.0013488751568206368, training data loss: 0.00010051696505992924\n",
      "Time for epoch[s]: 0.4871640205383301\n",
      "Epoch 39: validation data loss: 0.0013412524303889166, training data loss: 9.855326077981627e-05\n",
      "Time for epoch[s]: 0.4768078327178955\n",
      "Epoch 40: validation data loss: 0.0013389090696970622, training data loss: 9.664106416647838e-05\n",
      "Time for epoch[s]: 0.4914572238922119\n",
      "Epoch 41: validation data loss: 0.001319207012925518, training data loss: 9.477300118637956e-05\n",
      "Time for epoch[s]: 0.4922294616699219\n",
      "Epoch 42: validation data loss: 0.0013137463837453764, training data loss: 9.27602132297542e-05\n",
      "Time for epoch[s]: 0.4701247215270996\n",
      "Epoch 43: validation data loss: 0.0013075974433933762, training data loss: 9.08185374927303e-05\n",
      "Time for epoch[s]: 0.46984386444091797\n",
      "Epoch 44: validation data loss: 0.0013059449794629936, training data loss: 8.931258463696258e-05\n",
      "Time for epoch[s]: 0.479386568069458\n",
      "Epoch 45: validation data loss: 0.0013068367357123387, training data loss: 8.745189569039977e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.4589855670928955\n",
      "Epoch 46: validation data loss: 0.001296615899969998, training data loss: 8.627738310321826e-05\n",
      "Time for epoch[s]: 0.4536874294281006\n",
      "Epoch 47: validation data loss: 0.0012933304320731664, training data loss: 8.530389160326082e-05\n",
      "Time for epoch[s]: 0.48327159881591797\n",
      "Epoch 48: validation data loss: 0.0012809134781632794, training data loss: 8.287894024968692e-05\n",
      "Time for epoch[s]: 0.4754762649536133\n",
      "Epoch 49: validation data loss: 0.0012664579909686085, training data loss: 8.228843921123575e-05\n",
      "Time for epoch[s]: 0.5156850814819336\n",
      "Epoch 50: validation data loss: 0.001281207554960904, training data loss: 8.063727658088894e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.484525203704834\n",
      "Epoch 51: validation data loss: 0.0012793742358412372, training data loss: 7.933049067242504e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.47775959968566895\n",
      "Epoch 52: validation data loss: 0.0012620397626537166, training data loss: 7.754215570889651e-05\n",
      "Time for epoch[s]: 0.44751453399658203\n",
      "Epoch 53: validation data loss: 0.0012557299714110214, training data loss: 7.647049685591433e-05\n",
      "Time for epoch[s]: 0.4656510353088379\n",
      "Epoch 54: validation data loss: 0.0012533495959625941, training data loss: 7.552807495746439e-05\n",
      "Time for epoch[s]: 0.4719696044921875\n",
      "Epoch 55: validation data loss: 0.0012469810165771065, training data loss: 7.356968644547136e-05\n",
      "Time for epoch[s]: 0.6623072624206543\n",
      "Epoch 56: validation data loss: 0.0012425123012229189, training data loss: 7.289779887079648e-05\n",
      "Time for epoch[s]: 0.766538143157959\n",
      "Epoch 57: validation data loss: 0.0012386706593918474, training data loss: 7.163356462297919e-05\n",
      "Time for epoch[s]: 0.8063182830810547\n",
      "Epoch 58: validation data loss: 0.0012330303997753961, training data loss: 7.039397852878048e-05\n",
      "Time for epoch[s]: 0.7154138088226318\n",
      "Epoch 59: validation data loss: 0.0012297552742370186, training data loss: 6.976119799701047e-05\n",
      "Time for epoch[s]: 0.68231201171875\n",
      "Epoch 60: validation data loss: 0.0012332218694904623, training data loss: 6.865748293595771e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6924641132354736\n",
      "Epoch 61: validation data loss: 0.0012360153949424013, training data loss: 6.754254097263562e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.7475404739379883\n",
      "Epoch 62: validation data loss: 0.0012217621280722422, training data loss: 6.665972371896108e-05\n",
      "Time for epoch[s]: 0.6380493640899658\n",
      "Epoch 63: validation data loss: 0.0012299538203026062, training data loss: 6.599610747948085e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6568937301635742\n",
      "Epoch 64: validation data loss: 0.0012254069929253566, training data loss: 6.488074450732366e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6397964954376221\n",
      "Epoch 65: validation data loss: 0.0012102699987420209, training data loss: 6.394818016927536e-05\n",
      "Time for epoch[s]: 0.5808191299438477\n",
      "Epoch 66: validation data loss: 0.0012136458261916626, training data loss: 6.353235952386028e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5299150943756104\n",
      "Epoch 67: validation data loss: 0.0012115551728636162, training data loss: 6.249942072586381e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5461301803588867\n",
      "Epoch 68: validation data loss: 0.0012136825687809078, training data loss: 6.194550918253589e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6098036766052246\n",
      "Epoch 69: validation data loss: 0.0011957841648903068, training data loss: 6.106658307112516e-05\n",
      "Time for epoch[s]: 0.49443697929382324\n",
      "Epoch 70: validation data loss: 0.0012090134021898384, training data loss: 6.0644723713125815e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5771152973175049\n",
      "Epoch 71: validation data loss: 0.0012049423233014807, training data loss: 5.963574414519959e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.592634916305542\n",
      "Epoch 72: validation data loss: 0.0011860333621229755, training data loss: 5.8877315865531903e-05\n",
      "Time for epoch[s]: 0.6245675086975098\n",
      "Epoch 73: validation data loss: 0.0012034822816718114, training data loss: 5.8022268184515984e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5600874423980713\n",
      "Epoch 74: validation data loss: 0.0011911071054467328, training data loss: 5.743738059719948e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5978567600250244\n",
      "Epoch 75: validation data loss: 0.0011791185429107108, training data loss: 5.695558914310856e-05\n",
      "Time for epoch[s]: 0.6221659183502197\n",
      "Epoch 76: validation data loss: 0.0011804415482908623, training data loss: 5.622351006285785e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.6658632755279541\n",
      "Epoch 77: validation data loss: 0.0011842248102301332, training data loss: 5.5597346599243545e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.638683557510376\n",
      "Epoch 78: validation data loss: 0.0011865778328621224, training data loss: 5.503138314643407e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5043275356292725\n",
      "Epoch 79: validation data loss: 0.0011764564742780712, training data loss: 5.434154103335725e-05\n",
      "Time for epoch[s]: 0.5594394207000732\n",
      "Epoch 80: validation data loss: 0.0011713802814483643, training data loss: 5.39378020657252e-05\n",
      "Time for epoch[s]: 0.48505353927612305\n",
      "Epoch 81: validation data loss: 0.0011635419984930727, training data loss: 5.338780718034805e-05\n",
      "Time for epoch[s]: 0.4807109832763672\n",
      "Epoch 82: validation data loss: 0.001191474531339184, training data loss: 5.256402046849194e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5977137088775635\n",
      "Epoch 83: validation data loss: 0.00116573076814277, training data loss: 5.2282152927085144e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.48230671882629395\n",
      "Epoch 84: validation data loss: 0.0011610961668023236, training data loss: 5.163982952838619e-05\n",
      "Time for epoch[s]: 0.4807744026184082\n",
      "Epoch 85: validation data loss: 0.001160159775111229, training data loss: 5.095884986391895e-05\n",
      "Time for epoch[s]: 0.4683964252471924\n",
      "Epoch 86: validation data loss: 0.0011565541023533094, training data loss: 5.0639244621474994e-05\n",
      "Time for epoch[s]: 0.48079657554626465\n",
      "Epoch 87: validation data loss: 0.0011650200031663729, training data loss: 4.9896189543210206e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.4867522716522217\n",
      "Epoch 88: validation data loss: 0.0011660648535375725, training data loss: 4.9374487302074694e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.4777686595916748\n",
      "Epoch 89: validation data loss: 0.001172857741787009, training data loss: 4.876885120863239e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.4037749767303467\n",
      "Epoch 90: validation data loss: 0.0011569728317870397, training data loss: 4.818514584814577e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.41504907608032227\n",
      "Epoch 91: validation data loss: 0.0011612450423305982, training data loss: 4.771490024241138e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.4324343204498291\n",
      "Epoch 92: validation data loss: 0.0011530321210486705, training data loss: 4.714261740446091e-05\n",
      "Time for epoch[s]: 0.39008545875549316\n",
      "Epoch 93: validation data loss: 0.0011521266203492744, training data loss: 4.676182554464906e-05\n",
      "Time for epoch[s]: 0.4151620864868164\n",
      "Epoch 94: validation data loss: 0.0011457540945375346, training data loss: 4.604277647521398e-05\n",
      "Time for epoch[s]: 0.4062478542327881\n",
      "Epoch 95: validation data loss: 0.0011454255885729506, training data loss: 4.584577410868858e-05\n",
      "Time for epoch[s]: 0.41641664505004883\n",
      "Epoch 96: validation data loss: 0.0011522983579330792, training data loss: 4.53158260480454e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.5172176361083984\n",
      "Epoch 97: validation data loss: 0.0011439871842458369, training data loss: 4.500885957588344e-05\n",
      "Time for epoch[s]: 0.5196144580841064\n",
      "Epoch 98: validation data loss: 0.0011327926698885007, training data loss: 4.4318868621299256e-05\n",
      "Time for epoch[s]: 0.5013623237609863\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-11-09 18:15:23,522]\u001b[0m Trial 3 finished with value: 0.0011358471357659117 and parameters: {'learning_rate': 0.2, 'ff_neurons_layer_00': 10, 'ff_neurons_layer_01': 10, 'early_stopping_epochs': 8, 'layer_activation_00': 'LeakyReLU'}. Best is trial 3 with value: 0.0011358471357659117.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 99: validation data loss: 0.0011358471357659117, training data loss: 4.404742423802206e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.47202181816101074\n",
      "Final validation data loss:  0.0011358471357659117\n",
      "Initial Guess - validation data loss:  0.056024612357082976\n",
      "Epoch 0: validation data loss: 0.007264077935588959, training data loss: 0.00581682218264227\n",
      "Time for epoch[s]: 0.5433928966522217\n",
      "Epoch 1: validation data loss: 0.00719345378004797, training data loss: 0.003626645155693298\n",
      "Time for epoch[s]: 0.5596456527709961\n",
      "Epoch 2: validation data loss: 0.0071079567687152186, training data loss: 0.003570293454819074\n",
      "Time for epoch[s]: 0.5619573593139648\n",
      "Epoch 3: validation data loss: 0.0070283407490003055, training data loss: 0.003512501172279114\n",
      "Time for epoch[s]: 0.5754339694976807\n",
      "Epoch 4: validation data loss: 0.006942951515929339, training data loss: 0.003453353768614329\n",
      "Time for epoch[s]: 0.5445222854614258\n",
      "Epoch 5: validation data loss: 0.006831633990213751, training data loss: 0.0033907416748673947\n",
      "Time for epoch[s]: 0.5995156764984131\n",
      "Epoch 6: validation data loss: 0.006719848881029103, training data loss: 0.0033147302392410906\n",
      "Time for epoch[s]: 0.6050221920013428\n",
      "Epoch 7: validation data loss: 0.006601111529624625, training data loss: 0.0032400546008593415\n",
      "Time for epoch[s]: 0.4743688106536865\n",
      "Epoch 8: validation data loss: 0.00646820569147258, training data loss: 0.0031544419184123002\n",
      "Time for epoch[s]: 0.4800593852996826\n",
      "Epoch 9: validation data loss: 0.00631572503477471, training data loss: 0.0030603827951161283\n",
      "Time for epoch[s]: 0.46599316596984863\n",
      "Epoch 10: validation data loss: 0.006158027474738692, training data loss: 0.0029604647257556655\n",
      "Time for epoch[s]: 0.4792804718017578\n",
      "Epoch 11: validation data loss: 0.005979061126708984, training data loss: 0.0028546860773269443\n",
      "Time for epoch[s]: 0.47464990615844727\n",
      "Epoch 12: validation data loss: 0.005791754483087967, training data loss: 0.0027413253914820006\n",
      "Time for epoch[s]: 0.4686093330383301\n",
      "Epoch 13: validation data loss: 0.0055925133021454835, training data loss: 0.0026260745035458916\n",
      "Time for epoch[s]: 0.4866006374359131\n",
      "Epoch 14: validation data loss: 0.005383527986535199, training data loss: 0.0025065881476554696\n",
      "Time for epoch[s]: 0.47725486755371094\n",
      "Epoch 15: validation data loss: 0.005174739720070199, training data loss: 0.0023864605655408884\n",
      "Time for epoch[s]: 0.47658777236938477\n",
      "Epoch 16: validation data loss: 0.004975319453026062, training data loss: 0.002271517907103447\n",
      "Time for epoch[s]: 0.48330187797546387\n",
      "Epoch 17: validation data loss: 0.004771573358474801, training data loss: 0.002161020149379016\n",
      "Time for epoch[s]: 0.49689197540283203\n",
      "Epoch 18: validation data loss: 0.004581809043884277, training data loss: 0.0020588537057240805\n",
      "Time for epoch[s]: 0.4678981304168701\n",
      "Epoch 19: validation data loss: 0.004407836421983971, training data loss: 0.001964637541879802\n",
      "Time for epoch[s]: 0.478191614151001\n",
      "Epoch 20: validation data loss: 0.0042372269717525675, training data loss: 0.0018790961672726287\n",
      "Time for epoch[s]: 0.5281932353973389\n",
      "Epoch 21: validation data loss: 0.004092312566765912, training data loss: 0.001802488276947579\n",
      "Time for epoch[s]: 0.5495848655700684\n",
      "Epoch 22: validation data loss: 0.003962661033351671, training data loss: 0.0017334239123618766\n",
      "Time for epoch[s]: 0.571040153503418\n",
      "Epoch 23: validation data loss: 0.003837252588576922, training data loss: 0.001673930992274524\n",
      "Time for epoch[s]: 0.560309886932373\n",
      "Epoch 24: validation data loss: 0.0037308218272309327, training data loss: 0.0016176368547901171\n",
      "Time for epoch[s]: 0.5466861724853516\n",
      "Epoch 25: validation data loss: 0.0036313541947978817, training data loss: 0.0015692640112959632\n",
      "Time for epoch[s]: 0.5491445064544678\n",
      "Epoch 26: validation data loss: 0.003535649820005513, training data loss: 0.0015190030069656025\n",
      "Time for epoch[s]: 0.5827293395996094\n",
      "Epoch 27: validation data loss: 0.003455485927459856, training data loss: 0.0014753341674804688\n",
      "Time for epoch[s]: 0.5520334243774414\n",
      "Epoch 28: validation data loss: 0.003385765367446969, training data loss: 0.0014343635948825646\n",
      "Time for epoch[s]: 0.6420376300811768\n",
      "Epoch 29: validation data loss: 0.003317074688602256, training data loss: 0.0013961072109605623\n",
      "Time for epoch[s]: 0.8544392585754395\n",
      "Epoch 30: validation data loss: 0.003253321669417429, training data loss: 0.0013597523240738262\n",
      "Time for epoch[s]: 0.8020448684692383\n",
      "Epoch 31: validation data loss: 0.0032112133013059013, training data loss: 0.0013254904039374225\n",
      "Time for epoch[s]: 0.7581214904785156\n",
      "Epoch 32: validation data loss: 0.003158817824707728, training data loss: 0.0012947903376191719\n",
      "Time for epoch[s]: 0.789799690246582\n",
      "Epoch 33: validation data loss: 0.0030962318590242567, training data loss: 0.001265731984621858\n",
      "Time for epoch[s]: 0.7596797943115234\n",
      "Epoch 34: validation data loss: 0.003057546539393734, training data loss: 0.001235458676673506\n",
      "Time for epoch[s]: 0.7084236145019531\n",
      "Epoch 35: validation data loss: 0.0030152811307340996, training data loss: 0.0012126699702380455\n",
      "Time for epoch[s]: 0.8374521732330322\n",
      "Epoch 36: validation data loss: 0.0029708928169180815, training data loss: 0.0011810920282041646\n",
      "Time for epoch[s]: 0.6097991466522217\n",
      "Epoch 37: validation data loss: 0.002934419401160114, training data loss: 0.001156097405577359\n",
      "Time for epoch[s]: 0.5641207695007324\n",
      "Epoch 38: validation data loss: 0.0029071504122590366, training data loss: 0.0011319284705810894\n",
      "Time for epoch[s]: 0.6739723682403564\n",
      "Epoch 39: validation data loss: 0.002859026601869766, training data loss: 0.001110921439514857\n",
      "Time for epoch[s]: 0.6481649875640869\n",
      "Epoch 40: validation data loss: 0.002840678441470072, training data loss: 0.0010898636492420006\n",
      "Time for epoch[s]: 0.6650772094726562\n",
      "Epoch 41: validation data loss: 0.0027993618081149445, training data loss: 0.0010693993878691163\n",
      "Time for epoch[s]: 0.5996792316436768\n",
      "Epoch 42: validation data loss: 0.002770141107306633, training data loss: 0.0010498940400336975\n",
      "Time for epoch[s]: 0.6528778076171875\n",
      "Epoch 43: validation data loss: 0.002739777303721807, training data loss: 0.001031153958682056\n",
      "Time for epoch[s]: 0.7953033447265625\n",
      "Epoch 44: validation data loss: 0.0027129397000351997, training data loss: 0.0010142861163779482\n",
      "Time for epoch[s]: 0.8619744777679443\n",
      "Epoch 45: validation data loss: 0.0026837209043981823, training data loss: 0.0009969670886862767\n",
      "Time for epoch[s]: 0.8630115985870361\n",
      "Epoch 46: validation data loss: 0.0026570207452120847, training data loss: 0.0009813583604821332\n",
      "Time for epoch[s]: 0.7982122898101807\n",
      "Epoch 47: validation data loss: 0.0026336579017987535, training data loss: 0.0009658191710302274\n",
      "Time for epoch[s]: 0.8569724559783936\n",
      "Epoch 48: validation data loss: 0.002610873141789545, training data loss: 0.0009506606075861683\n",
      "Time for epoch[s]: 0.6405019760131836\n",
      "Epoch 49: validation data loss: 0.0025874046974530504, training data loss: 0.0009366827740516836\n",
      "Time for epoch[s]: 0.6617248058319092\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Input \u001b[0;32mIn [27]\u001b[0m, in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mhyperoptimizer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mperform_study\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m# Will save the results directly to the parameters.\u001b[39;00m\n\u001b[1;32m      4\u001b[0m hyperoptimizer\u001b[38;5;241m.\u001b[39mset_optimal_parameters()\n",
      "File \u001b[0;32m~/codes/mala/mala/network/hyper_opt_optuna.py:104\u001b[0m, in \u001b[0;36mHyperOptOptuna.perform_study\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparams\u001b[38;5;241m.\u001b[39mhyperparameters\u001b[38;5;241m.\u001b[39mcheckpoints_each_trial \u001b[38;5;241m!=\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[1;32m    102\u001b[0m     callback_list\u001b[38;5;241m.\u001b[39mappend(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__create_checkpointing)\n\u001b[0;32m--> 104\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstudy\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43moptimize\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mobjective\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    105\u001b[0m \u001b[43m                    \u001b[49m\u001b[43mn_trials\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    106\u001b[0m \u001b[43m                    \u001b[49m\u001b[43mcallbacks\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcallback_list\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    108\u001b[0m \u001b[38;5;66;03m# Return the best lost value we could achieve.\u001b[39;00m\n\u001b[1;32m    109\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstudy\u001b[38;5;241m.\u001b[39mbest_value\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/optuna/study/study.py:400\u001b[0m, in \u001b[0;36mStudy.optimize\u001b[0;34m(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)\u001b[0m\n\u001b[1;32m    392\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m n_jobs \u001b[38;5;241m!=\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[1;32m    393\u001b[0m     warnings\u001b[38;5;241m.\u001b[39mwarn(\n\u001b[1;32m    394\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m`n_jobs` argument has been deprecated in v2.7.0. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    395\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThis feature will be removed in v4.0.0. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    396\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSee https://github.com/optuna/optuna/releases/tag/v2.7.0.\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m    397\u001b[0m         \u001b[38;5;167;01mFutureWarning\u001b[39;00m,\n\u001b[1;32m    398\u001b[0m     )\n\u001b[0;32m--> 400\u001b[0m \u001b[43m_optimize\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    401\u001b[0m \u001b[43m    \u001b[49m\u001b[43mstudy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    402\u001b[0m \u001b[43m    \u001b[49m\u001b[43mfunc\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfunc\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    403\u001b[0m \u001b[43m    \u001b[49m\u001b[43mn_trials\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mn_trials\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    404\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    405\u001b[0m \u001b[43m    \u001b[49m\u001b[43mn_jobs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mn_jobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    406\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcatch\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcatch\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    407\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcallbacks\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcallbacks\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    408\u001b[0m \u001b[43m    \u001b[49m\u001b[43mgc_after_trial\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgc_after_trial\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    409\u001b[0m \u001b[43m    \u001b[49m\u001b[43mshow_progress_bar\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mshow_progress_bar\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    410\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/optuna/study/_optimize.py:66\u001b[0m, in \u001b[0;36m_optimize\u001b[0;34m(study, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)\u001b[0m\n\u001b[1;32m     64\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m     65\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m n_jobs \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[0;32m---> 66\u001b[0m         \u001b[43m_optimize_sequential\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m     67\u001b[0m \u001b[43m            \u001b[49m\u001b[43mstudy\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     68\u001b[0m \u001b[43m            \u001b[49m\u001b[43mfunc\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     69\u001b[0m \u001b[43m            \u001b[49m\u001b[43mn_trials\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     70\u001b[0m \u001b[43m            \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     71\u001b[0m \u001b[43m            \u001b[49m\u001b[43mcatch\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     72\u001b[0m \u001b[43m            \u001b[49m\u001b[43mcallbacks\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     73\u001b[0m \u001b[43m            \u001b[49m\u001b[43mgc_after_trial\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     74\u001b[0m \u001b[43m            \u001b[49m\u001b[43mreseed_sampler_rng\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m     75\u001b[0m \u001b[43m            \u001b[49m\u001b[43mtime_start\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m     76\u001b[0m \u001b[43m            \u001b[49m\u001b[43mprogress_bar\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mprogress_bar\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     77\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     78\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m     79\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m show_progress_bar:\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/optuna/study/_optimize.py:163\u001b[0m, in \u001b[0;36m_optimize_sequential\u001b[0;34m(study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng, time_start, progress_bar)\u001b[0m\n\u001b[1;32m    160\u001b[0m         \u001b[38;5;28;01mbreak\u001b[39;00m\n\u001b[1;32m    162\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 163\u001b[0m     trial \u001b[38;5;241m=\u001b[39m \u001b[43m_run_trial\u001b[49m\u001b[43m(\u001b[49m\u001b[43mstudy\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunc\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcatch\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    164\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m:\n\u001b[1;32m    165\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/optuna/study/_optimize.py:213\u001b[0m, in \u001b[0;36m_run_trial\u001b[0;34m(study, func, catch)\u001b[0m\n\u001b[1;32m    210\u001b[0m     thread\u001b[38;5;241m.\u001b[39mstart()\n\u001b[1;32m    212\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 213\u001b[0m     value_or_values \u001b[38;5;241m=\u001b[39m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrial\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    214\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m exceptions\u001b[38;5;241m.\u001b[39mTrialPruned \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    215\u001b[0m     \u001b[38;5;66;03m# TODO(mamu): Handle multi-objective cases.\u001b[39;00m\n\u001b[1;32m    216\u001b[0m     state \u001b[38;5;241m=\u001b[39m TrialState\u001b[38;5;241m.\u001b[39mPRUNED\n",
      "File \u001b[0;32m~/codes/mala/mala/network/objective_base.py:86\u001b[0m, in \u001b[0;36mObjectiveBase.__call__\u001b[0;34m(self, trial)\u001b[0m\n\u001b[1;32m     83\u001b[0m test_network \u001b[38;5;241m=\u001b[39m Network(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparams)\n\u001b[1;32m     84\u001b[0m test_trainer \u001b[38;5;241m=\u001b[39m Trainer(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparams, test_network,\n\u001b[1;32m     85\u001b[0m                        \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdata_handler)\n\u001b[0;32m---> 86\u001b[0m \u001b[43mtest_trainer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_network\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     87\u001b[0m final_validation_loss\u001b[38;5;241m.\u001b[39mappend(test_trainer\u001b[38;5;241m.\u001b[39mfinal_validation_loss)\n\u001b[1;32m     88\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtrial_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124moptuna\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m \\\n\u001b[1;32m     89\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparams\u001b[38;5;241m.\u001b[39mhyperparameters\u001b[38;5;241m.\u001b[39mpruner \\\n\u001b[1;32m     90\u001b[0m         \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmulti_training\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     94\u001b[0m     \u001b[38;5;66;03m# use it for one of the losses during multiple trainings.\u001b[39;00m\n\u001b[1;32m     95\u001b[0m     \u001b[38;5;66;03m# It should not pose a problem though.\u001b[39;00m\n",
      "File \u001b[0;32m~/codes/mala/mala/network/trainer.py:342\u001b[0m, in \u001b[0;36mTrainer.train_network\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    339\u001b[0m             batchid \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[1;32m    340\u001b[0m     training_loss \u001b[38;5;241m=\u001b[39m training_loss_sum\u001b[38;5;241m.\u001b[39mitem() \u001b[38;5;241m/\u001b[39m batchid\n\u001b[0;32m--> 342\u001b[0m vloss \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__validate_network\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnetwork\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    343\u001b[0m \u001b[43m                                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mvalidation\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    344\u001b[0m \u001b[43m                                \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparameters\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\n\u001b[1;32m    345\u001b[0m \u001b[43m                                \u001b[49m\u001b[43mduring_training_metric\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    347\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparameters_full\u001b[38;5;241m.\u001b[39muse_horovod:\n\u001b[1;32m    348\u001b[0m     vloss \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__average_validation(vloss, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124maverage_loss\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "File \u001b[0;32m~/codes/mala/mala/network/trainer.py:808\u001b[0m, in \u001b[0;36mTrainer.__validate_network\u001b[0;34m(self, network, data_set_type, validation_type)\u001b[0m\n\u001b[1;32m    805\u001b[0m                 y \u001b[38;5;241m=\u001b[39m y\u001b[38;5;241m.\u001b[39mto(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparameters\u001b[38;5;241m.\u001b[39m_configuration[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdevice\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[1;32m    806\u001b[0m                 prediction \u001b[38;5;241m=\u001b[39m network(x)\n\u001b[1;32m    807\u001b[0m                 validation_loss_sum \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \\\n\u001b[0;32m--> 808\u001b[0m                     \u001b[43mnetwork\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcalculate_loss\u001b[49m\u001b[43m(\u001b[49m\u001b[43mprediction\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mitem()\n\u001b[1;32m    809\u001b[0m                 batchid \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[1;32m    811\u001b[0m validation_loss \u001b[38;5;241m=\u001b[39m validation_loss_sum\u001b[38;5;241m.\u001b[39mitem() \u001b[38;5;241m/\u001b[39m batchid\n",
      "File \u001b[0;32m~/codes/mala/mala/network/network.py:148\u001b[0m, in \u001b[0;36mNetwork.calculate_loss\u001b[0;34m(self, output, target)\u001b[0m\n\u001b[1;32m    130\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcalculate_loss\u001b[39m(\u001b[38;5;28mself\u001b[39m, output, target):\n\u001b[1;32m    131\u001b[0m     \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    132\u001b[0m \u001b[38;5;124;03m    Calculate the loss for a predicted output and target.\u001b[39;00m\n\u001b[1;32m    133\u001b[0m \n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    146\u001b[0m \n\u001b[1;32m    147\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 148\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mloss_func\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutput\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtarget\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/torch/nn/functional.py:3292\u001b[0m, in \u001b[0;36mmse_loss\u001b[0;34m(input, target, size_average, reduce, reduction)\u001b[0m\n\u001b[1;32m   3289\u001b[0m     reduction \u001b[38;5;241m=\u001b[39m _Reduction\u001b[38;5;241m.\u001b[39mlegacy_get_string(size_average, reduce)\n\u001b[1;32m   3291\u001b[0m expanded_input, expanded_target \u001b[38;5;241m=\u001b[39m torch\u001b[38;5;241m.\u001b[39mbroadcast_tensors(\u001b[38;5;28minput\u001b[39m, target)\n\u001b[0;32m-> 3292\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_C\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_nn\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmse_loss\u001b[49m\u001b[43m(\u001b[49m\u001b[43mexpanded_input\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mexpanded_target\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m_Reduction\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_enum\u001b[49m\u001b[43m(\u001b[49m\u001b[43mreduction\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "hyperoptimizer.perform_study()\n",
    "\n",
    "# Will save the results directly to the parameters.\n",
    "hyperoptimizer.set_optimal_parameters()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's train the best network one final time and see where we stand. In my case, the network that Optuna suggest is bigger and deeper then the ones we have used before, but that may differ. Keep in mind we are performing a very\n",
    " limited search here in the interest of time.\n",
    "\n",
    "Also, please keep in mind that the identified network already includes a first and last layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Initial Guess - validation data loss:  0.12444681664035745\n",
      "Epoch 0: validation data loss: 0.0025094491706046883, training data loss: 0.010554606511712618\n",
      "Time for epoch[s]: 0.17014431953430176\n",
      "Epoch 1: validation data loss: 0.0017253474829948112, training data loss: 0.00034198600407604757\n",
      "Time for epoch[s]: 0.15175318717956543\n",
      "Epoch 2: validation data loss: 0.0016061313892608365, training data loss: 0.00022392296423650768\n",
      "Time for epoch[s]: 0.15952706336975098\n",
      "Epoch 3: validation data loss: 0.001399337837140854, training data loss: 0.00017810407090404807\n",
      "Time for epoch[s]: 0.15450429916381836\n",
      "Epoch 4: validation data loss: 0.0012269305856260535, training data loss: 0.0001491936408493617\n",
      "Time for epoch[s]: 0.17688608169555664\n",
      "Epoch 5: validation data loss: 0.0011764581072820377, training data loss: 0.00012724550619517286\n",
      "Time for epoch[s]: 0.16974329948425293\n",
      "Epoch 6: validation data loss: 0.0011647255181177565, training data loss: 0.00011044397065628609\n",
      "Time for epoch[s]: 0.17380189895629883\n",
      "Epoch 7: validation data loss: 0.001157008485706974, training data loss: 9.814191388485094e-05\n",
      "Time for epoch[s]: 0.16643357276916504\n",
      "Epoch 8: validation data loss: 0.0010705691630437495, training data loss: 9.098011983311884e-05\n",
      "Time for epoch[s]: 0.1758885383605957\n",
      "Epoch 9: validation data loss: 0.001052861616491727, training data loss: 8.51106102744194e-05\n",
      "Time for epoch[s]: 0.15315604209899902\n",
      "Epoch 10: validation data loss: 0.0010055777144758668, training data loss: 7.657784135102137e-05\n",
      "Time for epoch[s]: 0.16622447967529297\n",
      "Epoch 11: validation data loss: 0.0009664407331649571, training data loss: 7.214897299466067e-05\n",
      "Time for epoch[s]: 0.17079854011535645\n",
      "Epoch 12: validation data loss: 0.0009705225626627604, training data loss: 6.89233606468597e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1626453399658203\n",
      "Epoch 13: validation data loss: 0.0009376623723060573, training data loss: 6.219576278777971e-05\n",
      "Time for epoch[s]: 0.18732118606567383\n",
      "Epoch 14: validation data loss: 0.0009357706732945899, training data loss: 6.024650038649502e-05\n",
      "Time for epoch[s]: 0.21503472328186035\n",
      "Epoch 15: validation data loss: 0.0009114424523697596, training data loss: 5.814910241185802e-05\n",
      "Time for epoch[s]: 0.19207382202148438\n",
      "Epoch 16: validation data loss: 0.0009086976448694865, training data loss: 5.593231653786141e-05\n",
      "Time for epoch[s]: 0.1710498332977295\n",
      "Epoch 17: validation data loss: 0.0008852447140706729, training data loss: 5.209504639449185e-05\n",
      "Time for epoch[s]: 0.15834760665893555\n",
      "Epoch 18: validation data loss: 0.0008852823412037332, training data loss: 5.2140579137900104e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15780353546142578\n",
      "Epoch 19: validation data loss: 0.0008763251103222642, training data loss: 4.758639473757243e-05\n",
      "Time for epoch[s]: 0.16171550750732422\n",
      "Epoch 20: validation data loss: 0.0008599337649671998, training data loss: 4.624139908786234e-05\n",
      "Time for epoch[s]: 0.171356201171875\n",
      "Epoch 21: validation data loss: 0.0008524152105801726, training data loss: 4.434494565338849e-05\n",
      "Time for epoch[s]: 0.20040440559387207\n",
      "Epoch 22: validation data loss: 0.0008470118726225204, training data loss: 4.2876781530031874e-05\n",
      "Time for epoch[s]: 0.17025256156921387\n",
      "Epoch 23: validation data loss: 0.0008314881967082959, training data loss: 4.085114217239972e-05\n",
      "Time for epoch[s]: 0.20725488662719727\n",
      "Epoch 24: validation data loss: 0.0008405388490250121, training data loss: 3.9751900861796726e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1723320484161377\n",
      "Epoch 25: validation data loss: 0.0008271073233591367, training data loss: 3.91015400217004e-05\n",
      "Time for epoch[s]: 0.20135879516601562\n",
      "Epoch 26: validation data loss: 0.0008053013451023189, training data loss: 3.770194930709116e-05\n",
      "Time for epoch[s]: 0.1560053825378418\n",
      "Epoch 27: validation data loss: 0.0008036482687954489, training data loss: 3.564914424805881e-05\n",
      "Time for epoch[s]: 0.16039729118347168\n",
      "Epoch 28: validation data loss: 0.0008001558856877018, training data loss: 3.486478139888751e-05\n",
      "Time for epoch[s]: 0.16507911682128906\n",
      "Epoch 29: validation data loss: 0.0007951622276001325, training data loss: 3.43077420575978e-05\n",
      "Time for epoch[s]: 0.18441271781921387\n",
      "Epoch 30: validation data loss: 0.0007902358628843473, training data loss: 3.337464826904476e-05\n",
      "Time for epoch[s]: 0.19577264785766602\n",
      "Epoch 31: validation data loss: 0.0007891650205333483, training data loss: 3.294608679059978e-05\n",
      "Time for epoch[s]: 0.1870403289794922\n",
      "Epoch 32: validation data loss: 0.0007760328108861566, training data loss: 3.2671693219169635e-05\n",
      "Time for epoch[s]: 0.15325713157653809\n",
      "Epoch 33: validation data loss: 0.0007667263212813634, training data loss: 3.0746171038308645e-05\n",
      "Time for epoch[s]: 0.16130614280700684\n",
      "Epoch 34: validation data loss: 0.0007747844473956383, training data loss: 2.9470561046714652e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16405773162841797\n",
      "Epoch 35: validation data loss: 0.0007614100768685886, training data loss: 2.9407499026354044e-05\n",
      "Time for epoch[s]: 0.1932666301727295\n",
      "Epoch 36: validation data loss: 0.0007509915796044754, training data loss: 2.9757961239580693e-05\n",
      "Time for epoch[s]: 0.1939401626586914\n",
      "Epoch 37: validation data loss: 0.0007532060010248124, training data loss: 2.8075973533059907e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1838977336883545\n",
      "Epoch 38: validation data loss: 0.0007585627303275888, training data loss: 2.8096517913689898e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15201449394226074\n",
      "Epoch 39: validation data loss: 0.0007395644302237524, training data loss: 2.665141359243763e-05\n",
      "Time for epoch[s]: 0.1666545867919922\n",
      "Epoch 40: validation data loss: 0.0007428793602338121, training data loss: 2.654076481586722e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15734386444091797\n",
      "Epoch 41: validation data loss: 0.0007342596848805746, training data loss: 2.6656144626064387e-05\n",
      "Time for epoch[s]: 0.15958523750305176\n",
      "Epoch 42: validation data loss: 0.0007405273582293018, training data loss: 2.6304030272106058e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1600503921508789\n",
      "Epoch 43: validation data loss: 0.0007289476590613797, training data loss: 2.5128329079172926e-05\n",
      "Time for epoch[s]: 0.19213271141052246\n",
      "Epoch 44: validation data loss: 0.000723479477237893, training data loss: 2.4533509510701107e-05\n",
      "Time for epoch[s]: 0.15506911277770996\n",
      "Epoch 45: validation data loss: 0.0007140222749753629, training data loss: 2.572952544308144e-05\n",
      "Time for epoch[s]: 0.18371224403381348\n",
      "Epoch 46: validation data loss: 0.0007203201588974696, training data loss: 2.406497770860859e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17177486419677734\n",
      "Epoch 47: validation data loss: 0.0007190507704808831, training data loss: 2.451033914061986e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.18706226348876953\n",
      "Epoch 48: validation data loss: 0.0007071290930656537, training data loss: 2.2985068783504233e-05\n",
      "Time for epoch[s]: 0.16471219062805176\n",
      "Epoch 49: validation data loss: 0.0007155913876616247, training data loss: 2.2865257749002275e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17649507522583008\n",
      "Epoch 50: validation data loss: 0.0007106427732667967, training data loss: 2.2901309288391783e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16104412078857422\n",
      "Epoch 51: validation data loss: 0.0007096109188854966, training data loss: 2.2667847131485265e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17308402061462402\n",
      "Epoch 52: validation data loss: 0.0007032541787787659, training data loss: 2.1996950415987946e-05\n",
      "Time for epoch[s]: 0.16472482681274414\n",
      "Epoch 53: validation data loss: 0.0007088757268914349, training data loss: 2.175740914665945e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1689453125\n",
      "Epoch 54: validation data loss: 0.0007013776530958201, training data loss: 2.130770154263331e-05\n",
      "Time for epoch[s]: 0.1568772792816162\n",
      "Epoch 55: validation data loss: 0.0006949306214781112, training data loss: 2.084132581609025e-05\n",
      "Time for epoch[s]: 0.1887671947479248\n",
      "Epoch 56: validation data loss: 0.0006843434485126304, training data loss: 2.1353306580487997e-05\n",
      "Time for epoch[s]: 0.18845057487487793\n",
      "Epoch 57: validation data loss: 0.0006867415828791928, training data loss: 2.0562951797491883e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.2067873477935791\n",
      "Epoch 58: validation data loss: 0.0006869493145920915, training data loss: 2.045508635710908e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17269039154052734\n",
      "Epoch 59: validation data loss: 0.0006853467933663495, training data loss: 1.992727098263562e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1629328727722168\n",
      "Epoch 60: validation data loss: 0.0006810079018274943, training data loss: 1.9992608151751568e-05\n",
      "Time for epoch[s]: 0.1613926887512207\n",
      "Epoch 61: validation data loss: 0.000679116134774195, training data loss: 1.9689560463847635e-05\n",
      "Time for epoch[s]: 0.17031431198120117\n",
      "Epoch 62: validation data loss: 0.0006758851683847436, training data loss: 1.941294490134335e-05\n",
      "Time for epoch[s]: 0.18761134147644043\n",
      "Epoch 63: validation data loss: 0.0006722103651255778, training data loss: 1.9375054106184336e-05\n",
      "Time for epoch[s]: 0.18986153602600098\n",
      "Epoch 64: validation data loss: 0.0006766331522432092, training data loss: 1.9015350961657963e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15864157676696777\n",
      "Epoch 65: validation data loss: 0.0006737308279020057, training data loss: 1.8788459094147705e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15986061096191406\n",
      "Epoch 66: validation data loss: 0.0006717803407477462, training data loss: 1.827460080044999e-05\n",
      "Time for epoch[s]: 0.15409636497497559\n",
      "Epoch 67: validation data loss: 0.0006629363314746178, training data loss: 1.8372285481729463e-05\n",
      "Time for epoch[s]: 0.17244887351989746\n",
      "Epoch 68: validation data loss: 0.0006605109123334492, training data loss: 1.857674693408078e-05\n",
      "Time for epoch[s]: 0.17470312118530273\n",
      "Epoch 69: validation data loss: 0.0006669856369767559, training data loss: 1.7954319392301176e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1890578269958496\n",
      "Epoch 70: validation data loss: 0.0006588647762934367, training data loss: 1.7974738320803534e-05\n",
      "Time for epoch[s]: 0.1599733829498291\n",
      "Epoch 71: validation data loss: 0.0006622470677171123, training data loss: 1.749089179075744e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16024327278137207\n",
      "Epoch 72: validation data loss: 0.0006614937085539238, training data loss: 1.769904451249125e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15470314025878906\n",
      "Epoch 73: validation data loss: 0.000652344803831893, training data loss: 1.7771965156407117e-05\n",
      "Time for epoch[s]: 0.1587998867034912\n",
      "Epoch 74: validation data loss: 0.0006601827465780249, training data loss: 1.734852701527615e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15402865409851074\n",
      "Epoch 75: validation data loss: 0.0006584254982264618, training data loss: 1.698989765591001e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17872095108032227\n",
      "Epoch 76: validation data loss: 0.000653778921523595, training data loss: 1.7577974704098484e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.19594740867614746\n",
      "Epoch 77: validation data loss: 0.0006557165487716187, training data loss: 1.6848870327687698e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16784358024597168\n",
      "Epoch 78: validation data loss: 0.0006549216840909496, training data loss: 1.6408756621964446e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15732336044311523\n",
      "Epoch 79: validation data loss: 0.0006568613525939314, training data loss: 1.7167950372241404e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17362570762634277\n",
      "Epoch 80: validation data loss: 0.0006504925690829482, training data loss: 1.7063645433345342e-05\n",
      "Time for epoch[s]: 0.17179298400878906\n",
      "Epoch 81: validation data loss: 0.0006529187366842679, training data loss: 1.665125452152126e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.21519112586975098\n",
      "Epoch 82: validation data loss: 0.0006527249535469159, training data loss: 1.618377630467012e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17599010467529297\n",
      "Epoch 83: validation data loss: 0.0006482474607964084, training data loss: 1.6866487846390842e-05\n",
      "Time for epoch[s]: 0.16019701957702637\n",
      "Epoch 84: validation data loss: 0.0006401540209713592, training data loss: 1.5945256725974278e-05\n",
      "Time for epoch[s]: 0.19672584533691406\n",
      "Epoch 85: validation data loss: 0.0006467081504325344, training data loss: 1.619686053632057e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1672985553741455\n",
      "Epoch 86: validation data loss: 0.0006414965543572761, training data loss: 1.575175000856456e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16373801231384277\n",
      "Epoch 87: validation data loss: 0.0006458564708221993, training data loss: 1.6080882170633094e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.17925548553466797\n",
      "Epoch 88: validation data loss: 0.0006457352202776904, training data loss: 1.497358747132837e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1890261173248291\n",
      "Epoch 89: validation data loss: 0.0006431688184607519, training data loss: 1.5128600585474272e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16095662117004395\n",
      "Epoch 90: validation data loss: 0.0006407561231421554, training data loss: 1.5279521621321434e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.15194296836853027\n",
      "Epoch 91: validation data loss: 0.0006378161716678915, training data loss: 1.5568077462176753e-05\n",
      "Time for epoch[s]: 0.1581864356994629\n",
      "Epoch 92: validation data loss: 0.0006291192007935755, training data loss: 1.5319572744644396e-05\n",
      "Time for epoch[s]: 0.16324520111083984\n",
      "Epoch 93: validation data loss: 0.0006327810080628416, training data loss: 1.5191983678229323e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1646277904510498\n",
      "Epoch 94: validation data loss: 0.0006387722954902475, training data loss: 1.4849652463918952e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.18356990814208984\n",
      "Epoch 95: validation data loss: 0.0006314690254594637, training data loss: 1.500310061593034e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16827130317687988\n",
      "Epoch 96: validation data loss: 0.000633737812303517, training data loss: 1.4825834633280698e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.1516587734222412\n",
      "Epoch 97: validation data loss: 0.0006336841953399519, training data loss: 1.4668743053799896e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16455507278442383\n",
      "Epoch 98: validation data loss: 0.0006299982332203486, training data loss: 1.4793369111064906e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.19806551933288574\n",
      "Epoch 99: validation data loss: 0.0006297527382907258, training data loss: 1.460756813112187e-05\n",
      "Validation accuracy has not improved enough.\n",
      "Time for epoch[s]: 0.16311264038085938\n",
      "Final validation data loss:  0.0006297527382907258\n",
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot2.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot2.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot3.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot3.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "DataHandler prepared for inference. No training possible with this setup. If this is not what you wanted, please revise the input script. Validation snapshots you may have entered willbe ignored.\n",
      "Consistency check successful.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Had to readjust batch size from 40 to 54\n",
      "{'band_energy': [-0.016781997686205585, -0.026142064746764504]}\n"
     ]
    }
   ],
   "source": [
    "data_handler.clear_data()\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "trainer.train_network()\n",
    "\n",
    "\n",
    "print(testing(parameters, data_handler, network))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The results may not necessarily be better with this limited search then what we had above. In my case they are very good, but again, this is a limited search compared to reasonable defaults. For actual production runs, one would set up a thorough, longer search before training production level models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Excurse: How to tune bispectrum hyperparameters (presentation only)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "With all that talk about hyperparameters, you may have found yourself wondering how we determine the bispectrum component hyperparameters that we had used far above to calculate atomic density representations on the grid. That is a very valid question.\n",
    "\n",
    "Technically, in the same way you have just done - by testing different values and training models.\n",
    "However, physical intuition can be exploited quite a fair bit here. Namely, it is possible to correlate LDOS and bispectrum values at different points in space. And by doing so, it is enough to simply calculate descriptors and then calculate a simple metric. No training needed, yielding a speed-up of many orders of magnitude.\n",
    "\n",
    "Presume you have two separate points in space. If the LDOS of these points are very similar, so should the bispectrum descriptors. If not, then the model will never have a chance of learning from the data provided.\n",
    "\n",
    "MALA implements an analysis based on this concept called ACSD. Since you need the LAMMPS library to perform it, we will only look at it and not try it out ourselves.\n",
    "\n",
    "To show you - in a very very simple way - that it works, we will give it two values to choose from for the two hyperparameters.\n",
    "\n",
    "The first hyperparameter for bispectrum descriptors is the cutoff, i.e., how much information from surrounding grid points are incorporate into the components. The second one is called twojmax and dictates the dimensionality of the descriptor vectors. If it is too small, the information is underrepresented. If it is too large, we learn a lot of noise.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting ACSD analysis of snapshot 0\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot1.out\n",
      "Trial 0 finished with ACSD=0.008543260362761715 and parameters: [cutoff: 0.5, twojmax: 2]. Best trial is 0 with 0.008543260362761715\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot1.out\n",
      "Trial 1 finished with ACSD=0.004255519941706275 and parameters: [cutoff: 0.5, twojmax: 10]. Best trial is 1 with 0.004255519941706275\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot1.out\n",
      "Trial 2 finished with ACSD=0.05425582155921187 and parameters: [cutoff: 2.0, twojmax: 2]. Best trial is 1 with 0.004255519941706275\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot1.out\n",
      "Trial 3 finished with ACSD=0.02995452470356675 and parameters: [cutoff: 2.0, twojmax: 10]. Best trial is 1 with 0.004255519941706275\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot1.out\n",
      "Trial 4 finished with ACSD=0.0032500308232971706 and parameters: [cutoff: 4.67637, twojmax: 2]. Best trial is 4 with 0.0032500308232971706\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot1.out\n",
      "Trial 5 finished with ACSD=0.009901549287490076 and parameters: [cutoff: 4.67637, twojmax: 10]. Best trial is 4 with 0.0032500308232971706\n",
      "Starting ACSD analysis of snapshot 1\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot2.out\n",
      "Trial 0 finished with ACSD=0.005488460318059493 and parameters: [cutoff: 0.5, twojmax: 2]. Best trial is 4 with 0.0032500308232971706\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot2.out\n",
      "Trial 1 finished with ACSD=0.0033989430133834005 and parameters: [cutoff: 0.5, twojmax: 10]. Best trial is 4 with 0.0032500308232971706\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot2.out\n",
      "Trial 2 finished with ACSD=0.060003975351102544 and parameters: [cutoff: 2.0, twojmax: 2]. Best trial is 4 with 0.0032500308232971706\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot2.out\n",
      "Trial 3 finished with ACSD=0.030400712762102357 and parameters: [cutoff: 2.0, twojmax: 10]. Best trial is 4 with 0.0032500308232971706\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot2.out\n",
      "Trial 4 finished with ACSD=0.0029143522698116073 and parameters: [cutoff: 4.67637, twojmax: 2]. Best trial is 4 with 0.0029143522698116073\n",
      "Calculating descriptors from /home/fiedlerl/data/mala_data_repo/Be2/Be_snapshot2.out\n",
      "Trial 5 finished with ACSD=0.013444155133685163 and parameters: [cutoff: 4.67637, twojmax: 10]. Best trial is 4 with 0.0029143522698116073\n",
      "ACSD analysis finished, optimal parameters: \n",
      "Bispectrum twojmax:  2\n",
      "Bispectrum cutoff:  4.67637\n"
     ]
    }
   ],
   "source": [
    "acsd_parameters = mala.Parameters()\n",
    "acsd_parameters.descriptors.descriptor_type = \"Bispectrum\"\n",
    "acsd_parameters.descriptors.bispectrum_twojmax = 10\n",
    "acsd_parameters.descriptors.bispectrum_cutoff = 4.67637\n",
    "acsd_parameters.descriptors.descriptors_contain_xyz = True\n",
    "acsd_parameters.targets.target_type = \"LDOS\"\n",
    "acsd_parameters.targets.ldos_gridsize = 11\n",
    "acsd_parameters.targets.ldos_gridspacing_ev = 2.5\n",
    "acsd_parameters.targets.ldos_gridoffset_ev = -5\n",
    "acsd_parameters.descriptors.descriptors_contain_xyz = True\n",
    "acsd_parameters.descriptors.acsd_points = 100\n",
    "\n",
    "hyperoptimizer = mala.ACSDAnalyzer(acsd_parameters)\n",
    "hyperoptimizer.add_hyperparameter(\"bispectrum_twojmax\", [2, 10])\n",
    "hyperoptimizer.add_hyperparameter(\"bispectrum_cutoff\", [0.5, 2.0, 4.67637])\n",
    "\n",
    "# Add raw snapshots to the hyperoptimizer. For the targets, numpy files are\n",
    "# okay as well.\n",
    "hyperoptimizer.add_snapshot(\"espresso-out\", pj(data_path, \"Be_snapshot1.out\"),\n",
    "                            \"numpy\", pj(data_path, \"Be_snapshot1.out.npy\"),\n",
    "                            target_units=\"1/(Ry*Bohr^3)\")\n",
    "hyperoptimizer.add_snapshot(\"espresso-out\", pj(data_path, \"Be_snapshot2.out\"),\n",
    "                            \"numpy\", pj(data_path, \"Be_snapshot2.out.npy\"),\n",
    "                            target_units=\"1/(Ry*Bohr^3)\")\n",
    "\n",
    "# If you plan to plot the results (recommended for exploratory searches),\n",
    "# the optimizer can return the necessary quantities to plot.\n",
    "plotting = hyperoptimizer.perform_study()\n",
    "hyperoptimizer.set_optimal_parameters()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Inference and prediction (hands-on / presentation)\n",
    "\n",
    "We now understand how we can build optimal models with MALA. Once this is done, we would like to use these models to predict properties of interest. For this, let us first see how MALA can be used to save and load models.\n",
    "\n",
    "We train a final model - based on the parameters idenitified above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot0.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot0.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking descriptor file  Be_snapshot1.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot1.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Consistency check successful.\n",
      "Initializing the data scalers.\n",
      "Input scaler parametrized.\n",
      "Output scaler parametrized.\n",
      "Data scalers initialized.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n",
      "Initial Guess - validation data loss:  0.056024612357082976\n",
      "Epoch 0: validation data loss: 0.007264077935588959, training data loss: 0.00581682218264227\n",
      "Time for epoch[s]: 0.599261999130249\n",
      "Epoch 1: validation data loss: 0.00719345378004797, training data loss: 0.003626645155693298\n",
      "Time for epoch[s]: 0.5895037651062012\n",
      "Epoch 2: validation data loss: 0.0071079567687152186, training data loss: 0.003570293454819074\n",
      "Time for epoch[s]: 0.5614831447601318\n",
      "Epoch 3: validation data loss: 0.0070283407490003055, training data loss: 0.003512501172279114\n",
      "Time for epoch[s]: 0.5882627964019775\n",
      "Epoch 4: validation data loss: 0.006942951515929339, training data loss: 0.003453353768614329\n",
      "Time for epoch[s]: 0.650747537612915\n",
      "Epoch 5: validation data loss: 0.006831633990213751, training data loss: 0.0033907416748673947\n",
      "Time for epoch[s]: 0.6137940883636475\n",
      "Epoch 6: validation data loss: 0.006719848881029103, training data loss: 0.0033147302392410906\n",
      "Time for epoch[s]: 0.6372406482696533\n",
      "Epoch 7: validation data loss: 0.006601111529624625, training data loss: 0.0032400546008593415\n",
      "Time for epoch[s]: 0.4927198886871338\n",
      "Epoch 8: validation data loss: 0.00646820569147258, training data loss: 0.0031544419184123002\n",
      "Time for epoch[s]: 0.4826834201812744\n",
      "Epoch 9: validation data loss: 0.00631572503477471, training data loss: 0.0030603827951161283\n",
      "Time for epoch[s]: 0.5329318046569824\n",
      "Final validation data loss:  0.00631572503477471\n"
     ]
    }
   ],
   "source": [
    "data_handler = mala.DataHandler(parameters)\n",
    "parameters.running.max_number_epochs = 10\n",
    "data_handler.clear_data()\n",
    "\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "trainer.train_network()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Saving such a model now becomes a one-liner. Models will be saved as `.zip` archives containing the network weights, parameter json file and coefficients for the data scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "trainer.save_run(\"Be_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Loading is equally simple, but dependent on the task. Two possible task can be done: Testing (as we have done above, to verify whether a model is performing well) or predicting (using the model to predict the electronic structure of arbitrary atomic configurations).\n",
    "\n",
    "We will have a quick look at the testing interface again to plot the density of states, and see how MALA can directly give us access to the electronic structure of a material.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the snapshots and your inputs for consistency.\n",
      "Checking descriptor file  Be_snapshot2.in.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "Checking targets file  Be_snapshot2.out.npy at /home/fiedlerl/data/mala_data_repo/Be2\n",
      "DataHandler prepared for inference. No training possible with this setup. If this is not what you wanted, please revise the input script. Validation snapshots you may have entered willbe ignored.\n",
      "Consistency check successful.\n",
      "Build datasets.\n",
      "Build dataset: Done.\n"
     ]
    }
   ],
   "source": [
    "parameters, network, data_handler, tester = mala.Tester.load_run(\"Be_model\")\n",
    "data_handler.clear_data()\n",
    "data_handler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                          \"Be_snapshot2.out.npy\", data_path, \"te\",\n",
    "                          calculation_output_file=pj(data_path, \"Be_snapshot2.out\"))\n",
    "data_handler.prepare_data(reparametrize_scaler=False)\n",
    "\n",
    "actual_ldos, predicted_ldos = tester.predict_targets(0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we can visualize the DOS via the internal LDOS calculator of the DataHandler object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7feff9804130>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAJNCAYAAACIkPmLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAB3XklEQVR4nO3dd5iU1cH+8e+Z2V5Y2lJ3l9477IIiIGDDBkqRYsMSk1iSN74x/U37pTdNTGJMYtQYpYqAChaKFKlL770tdanb+/n9MYNBpCyws2dm9v5c11zMPlO4x82yd87znHOMtRYRERERCX4e1wFEREREpHJU3ERERERChIqbiIiISIhQcRMREREJESpuIiIiIiFCxU1EREQkRES4DlAd6tevb5s3b+46hoiIiMhlrVq16ri1NvlCj9WI4ta8eXMyMzNdxxARERG5LGPMvos9plOlIiIiIiFCxU1EREQkRKi4iYiIiISIGnGNm4iIiFSd0tJSsrKyKCoqch0lpMXExJCSkkJkZGSlX6PiJiIiIlckKyuLxMREmjdvjjHGdZyQZK3lxIkTZGVl0aJFi0q/TqdKRURE5IoUFRVRr149lbZrYIyhXr16VzxqqeImIiIiV0yl7dpdzX9DFTcREREJSdOnT8cYw9atWy/5vBdeeIGCgoKr/ntee+01nn766at+fVVScRMREZGQNGHCBPr168eECRMu+bxrLW7BRMVNREREQk5eXh6LFy/mlVdeYeLEiQCUl5fzzW9+k86dO9O1a1defPFF/vSnP3Ho0CEGDRrEoEGDAEhISPjsfaZOncr48eMBePfdd+nTpw89evTg5ptv5ujRo9X+uS5Hs0pFREQk5MyYMYMhQ4bQtm1b6tWrx6pVq1ixYgV79+5l7dq1REREcPLkSerWrcsf/vAH5s+fT/369S/5nv369WPZsmUYY/jnP//Jb37zG37/+99X0yeqHBU3ERERuWo/eXcTmw/lVOl7dmxSix/d3emSz5kwYQJf//rXARgzZgwTJkxgz549fOUrXyEiwldv6tate0V/b1ZWFqNHj+bw4cOUlJRc0TId1UXFTURERELKyZMnmTdvHhs2bMAYQ3l5OcYYMjIyKvX6c2dznrscxzPPPMOzzz7L0KFD+eSTT/jxj39c1dGvmYqbiIiIXLXLjYwFwtSpU3nwwQd5+eWXPzt244030q1bN15++WUGDRr0uVOliYmJ5ObmfnaqtGHDhmzZsoV27drxzjvvkJiYCMCZM2do2rQpAK+//nq1f67K0OQEERERCSkTJkzg3nvv/dyxESNGcPjwYdLS0ujatSvdunXjrbfeAuCJJ55gyJAhn01O+NWvfsVdd91F3759ady48Wfv8eMf/5hRo0bRq1evy14P54qx1rrOEHDp6ek2MzPTdQwREZGwsGXLFjp06OA6Rli40H9LY8wqa236hZ6vETcRERGREKHiJiIiIhIiVNxEREREQoSKm4iIiEiIUHETERERCREqbiIiIiIhQsVNRORKZW+HiffD1Mdg2wdQVuI6kUiN4/V66d69O507d2bUqFEUFBRc9XuNHz+eqVOnAvD444+zefPmiz73k08+YcmSJVf8dzRv3pzjx49fdcazVNxERCqrvAwWPw9/6wd7F8OueTBhNPy+Lbz7P7D3U6iocJ1SpEaIjY1l7dq1bNy4kaioKP72t7997vGysrKret9//vOfdOzY8aKPX21xqyoqbiIilXF0M7xyC8z5MaWtbuHlLhN45boPOHPvm9D6Zlg/CV67A17oDB/9HxxeDzVggXORYNC/f3927tzJJ598Qv/+/Rk6dCgdO3akvLyc5557joyMDLp27frZFlnWWp5++mnatWvHzTffzLFjxz57r4EDB3J20f4PPviAnj170q1bN2666Sb27t3L3/72N55//nm6d+/OokWLyM7OZsSIEWRkZJCRkcGnn34KwIkTJ7j11lvp1KkTjz/+OFW14YH2KhURuZTyUlj8Aiz4NTYmiRXpf+CpNamcKDiFtaf4hcfDoHZfZvSd32MQmURsfhuW/RWW/Anqt4Ouo6DzSKjbwvUnEQlLZWVlzJ49myFDhgCwevVqNm7cSIsWLfj73/9OUlISK1eupLi4mBtuuIFbb72VNWvWsG3bNjZv3szRo0fp2LEjjz766OfeNzs7my996UssXLiQFi1afLbv6Ve+8hUSEhL45je/CcC4ceP4xje+Qb9+/di/fz+33XYbW7Zs4Sc/+Qn9+vXjhz/8Ie+//z6vvPJKlXxeFTcRkYs5vA5mPAVHNpDTehjfyBnL3MUVdE+N57VHOxMT6WHqqoNMW53FnC3HqBNXm2Hdf8jo+39Nh1PzYMNUmPcz3y0lA7qMgk73QkID159MpOrM/g4c2VC179moC9z+q0s+pbCwkO7duwO+EbfHHnuMJUuW0Lt3b1q08P0fpY8++oj169d/dv3amTNn2LFjBwsXLmTs2LF4vV6aNGnC4MGDv/D+y5YtY8CAAZ+9V926dS+YY86cOZ+7Ji4nJ4e8vDwWLlzItGnTALjzzjupU6fOlf03uAgVNxGR85UVw8LfwuLnqYitx7Q2v+Y7m9KIj/byy+GdGJ2eisdjAPjO7e355q1tWbTzOFNXZfHW8v28tqSCDo3bMLLXX7j3tgrq7nkPNkyB2d+CD74DLQf6Slz7uyCmltvPKhKizl7jdr74+PjP7ltrefHFF7nttts+95xZs2ZVWY6KigqWLVtGTExMlb3npai4iYic6+AqmP4UZG/hYLN7ePTwvWzbEMl96U359pD21EuI/sJLIrweBrVrwKB2DThdUMK76w4xdVUW/++9zfzSYxjU/gZG9h/N4Honidz0tq/ETf8qRHwD2t7mK3Gtb4HI6vmHX6RKXWZkzKXbbruNl156icGDBxMZGcn27dtp2rQpAwYM4OWXX+bhhx/m2LFjzJ8/n3Hjxn3utddddx1PPvkke/bs+dyp0sTERHJycj573q233sqLL77Ic889B8DatWvp3r07AwYM4K233uIHP/gBs2fP5tSpU1XymVTcREQASgvhk1/Ckhcpi2vInxr8jD9ta0n7RolMHdeZ9OYXPk1yvtpxUTx4fXMevL4524/m8vaqLKatOcjHm49SLz6KYd1HMHLU1+hYsd1X4DZNg80zIDoJOt7tK3HN+4PHG+APLBL+Hn/8cfbu3UvPnj2x1pKcnMz06dO59957mTdvHh07diQtLY3rr7/+C69NTk7m73//O8OHD6eiooIGDRrw8ccfc/fddzNy5EhmzJjBiy++yJ/+9CeeeuopunbtSllZGQMGDOBvf/sbP/rRjxg7diydOnWib9++pKWlVclnMlU1yyGYpaen27MzREREvmD/ct+1bCd2sKHhvTx88G6KPfE8e2s7Hr6+GRHea5uAX1ZewcId2UxdlcWczccoKa+gY+NajEpPYVjXhtQ9utR3PdyWd6EkFxIaQecR0GUkNOkBxlTRBxWpGlu2bKFDhw6uY4SFC/23NMasstamX+j5Km4iUnOV5PsmDix7iaL4Jny//AnePt2Gu7o25gd3dqRRUtWfujyVX8JM/6nUDQfPEOk1DG7fgJG9UhnYMoHIXR/7RuJ2fATlJVC3lW8UrstIqN+myvOIXA0Vt6qj4nYBKm4i8gV7FsHMZ+DUHhYk3cNXjw6lYf16/HRYJ/q3Sa6WCFuP5PD2qizeWXOQ43kl1E+I4p7uTRmZnkL7pArYMtNX4vYsAiw07u4rcZ2HQ60m1ZJR5EJU3KqOitsFqLiJyGeKc2HOj2HlPzkTm8rX8h9lWUUHnh7UmidubEl0RPVfW1ZaXsGCbb5TqXO3HqW03NK5aS1G9kxhWPem1Ck/4bsWbsMUOLQGMNC8n6/EdRwKsVWzzIBIZam4VR0VtwtQcRMRwLdF1cyvY88c4J3ooXzvzD1c3y6FnwztTFq9ONfpADiZX8LMtQeZsiqLTYdyiPQabu7QkJG9UrixbTIRp3bDxqmwfjKc3AWeSGhzq+9UatshEBUcn0PC25YtW2jfvj1G119eE2stW7duVXE7n4qbSA1XdAY++gGs/jdHo9L4au6jHKnVlR8N7cStHRsG7S+fzYdyeHt1FtPXHOREfgn1E6IZ3rMpI3qm0K5hAhxe65vUsGEq5B2BqATocLevxLUYCF4tHCCBsWfPHhITE6lXr17Q/vwEO2stJ06cIDc397NFfs9ScVNxE6m5tn+Iffd/sLlHeI27+V3JcB4e0J5nBrcmLio0ik1peQXztx5j6qos5m09RlmFpWtKEiN7pTC0WxNqx3hh36e+UbjNM6H4DMTV910L12WUb9cG/XKVKlRaWkpWVhZFRUWuo4S0mJgYUlJSiIyM/NxxFTcVN5Gap+AkfPBdWD+Rvd5mfK3gceJaZPD/hnWmTcNE1+mu2om8YmasPcSUVVlsOZxDlNfDzR0bMKpXKv3b1CfClsIO/8zU7R9AWRHUTvPPTB0FDXRdkkiwU3FTcROpWba8S8V7z2LzT/CXsmFMiL6Pb9/VlWHdm4TVaZ1Nh84wdVUWM9Ye4mR+CcmJ0Qzv0ZSRvVJ85bQoB7a+7ytxu+eDrYCGnX2nUjuP8BU6EQk6Km4qbiI1Q/5x7KznMJumsZUW/G/Jl0jvcyPP3tqOpNjIy78+RJWUVTB/m+9U6nz/qdRuqbV9p1K7NiEpLhLyjsGm6b4Sl7XC98K0630lruO9EF/P6WcQkf9ScVNxEwlv1sKmaZS/901sUQ7Plw5naaP7+cm9PeiSkuQ6XbU6nlfM9DUHmboqi61HcomK8HBLx4aM6pVC/zbJeD0GTu6Bjf49U7O3gjcK7nsD2g1xHV9EUHFTcRMJZ7lHKXv3G0Rsf591Fa34iecpRtx+M2My0nwlpYay1rLpUI7/VOpBThWU0rBWNPf2SGFkrxRaN0jwFd6jG2H6k3B6H3x5IdRp7jq6SI3nrLgZY4YAfwS8wD+ttb867/GvAE8B5UAe8IS1drP/se8Cj/kf+5q19sPKvOeFqLiJhCFrsesmUvb+t6koLeD3pSM53e1LfOuOztRPiHadLqgUl5V/Nit1/rZsyissPdJ8p1Lv6tqEpMIsePlGqNcSHv0QIvTfT8QlJ8XNGOMFtgO3AFnASmDs2WLmf04ta22O//5Q4Elr7RBjTEdgAtAbaALMAdr6X3bJ97wQFTeRMHPmIIXvfI3YvXPIrGjLS7W+wZdHDKF3i7qukwW97FzfqdQpqw6w/WgeUREehnRqxM/a76XWjPGQ8SW483euY4rUaJcqboFcxKg3sNNau9sfYiIwDPisZJ0tbX7xwNkWOQyYaK0tBvYYY3b634/LvaeIhDFrKc18nYoPvgdlpfzCPkyDm57hb/1aEen1uE4XEpITo/nSgJY83r8FGw/mMHXVASasPMDxvMb857qn8Sz7MzS73jfrVESCTiCLW1PgwDlfZwF9zn+SMeYp4FkgChh8zmuXnffapv77l31PEQlDp/dzauJXqXNkMUvLOzK75Xf56vBbaJwU6zpZSDLG0CUliS4pSXRqmsS3pq7nj6nj+EZqJsz8GjTqCvXbuI4pIudx/n9RrbV/sda2Ar4N/KCq3tcY84QxJtMYk5mdnV1Vbysi1a2igpxFf6Poj72JPJzJH6K+QskD0/npI0NV2qrIfempjOqVwh8/2cvSnr/xXeM2+SEoKXAdTUTOE8jidhBIPefrFP+xi5kI3HOZ11b6Pa21f7fWpltr05OTk68suYgEhbLsXRx68RZqzf02meWtmZwxhSef+wU3tmvoOlrY+emwzrRvlMiT7x7l+G1/gWNb4P3/9c08FZGgEcjithJoY4xpYYyJAsYAM899gjHm3HH4O4Ed/vszgTHGmGhjTAugDbCiMu8pImGgopwDs35H2V+uI+HkZv5Z939J/doHPHrXAGIiva7ThaXYKC9/vb8npeWWxxfXorz/c7DuLVjzhutoInKOgF3jZq0tM8Y8DXyIb+mOf1lrNxljfgpkWmtnAk8bY24GSoFTwMP+124yxkzGN+mgDHjKWlsOcKH3DNRnEJHqd/rAZk5P+BLNCzbyqacXJbf/gcfSu4XVVlXBqmVyAr8Z2ZUn31zNz1Pv5octV8Cs56BJD2jUxXU8EUEL8IpIkKgoK2X9lJ/TYdufKbRRLGz1TW667xniY8J3q6pg9eOZm3htyV7+MbwZtywaCZFx8MQnEFPLdTSRGuFSy4E4n5wgIrJz4wp2/aov3bc9z9roDI4/vIihDz2r0ubI9+7oQPfU2nzj/YMcvPmvcGovzHxa17uJBAEVNxFxJje/gHl//yZpU4ZQv+wIy3v9nt7ffp/WLVu5jlajRUV4+Mv9PYnwGh6bH0HpoP+DzTNg+cuuo4nUeCpuIlLtrLV8smAuh37bl8GH/sHWOoPwPr2CPnc/jvHon6Vg0LR2LC+M7s62o7l8/8hAaHs7fPQDyNJlJyIu6V9IEalWu46c4J0/PMkN80bRwJxiz01/p+v/vE2t+o1dR5PzDGzXgKcHtWby6kO80/wHUKsxTH4YCk66jiZSY6m4iUi1mT3nY8pfGsDw3LfY3/QOan1zDS36j3YdSy7hf25uS99W9fjOrAPsHvRXyD8G73wZKipcRxOpkVTcRKRa7Dp4hM6LvkoDbwFn7n2TVk/8B2+8NoUPdl6P4Y9jepAUG8ljH5dTdNPPYMdH8OnzrqOJ1EgqbiIScOUVli3/+RZNzXHsfa+T1O0u15HkCiQnRvPncT3Zf7KAZ3f3wnYeAfN+BnsWuY4mUuOouIlIwM2aPZM7Cmayp8UY6rQf4DqOXIXeLeryrdvaMWvjUd6o/yzUbQVTH4Xco66jidQoKm4iElD7j52m/YrvcSqiPi1H/8Z1HLkGTwxoyc0dGvLTj/azecCfoTgX3n4MystcRxOpMVTcRCRgrLWs/M8PaGOy4K7nMVp5P6QZY/j9qG40Sorh8Vn55N3yG9i7CD75hetoIjWGipuIBMzsefO5+8xb7Gl8B/V63O06jlSBpLhIXrq/F8fzSnhyU3tsjwdh0e9hx8euo4nUCCpuIhIQR07l03ThtyjyJtBs3J9cx5Eq1CUliR/e3ZGF27N5KfbL0LALTPsSnD7gOppI2FNxE5EqZ61l/hs/o5vZQfFNP8eTmOw6klSx+/ukMax7E343fz+r+jzvu85tyngoK3EdTSSsqbiJSJX7eMlKhp14hf31+pHc9wHXcSQAjDH84t4utExO4MuzTnP61hfgYCZ8/EPX0UTCmoqbiFSpE7lFJH78TYzHQ9MHXgJjXEeSAImPjuCl+3uSX1zOlzKbUN77K7D8Jdg03XU0kbCl4iYiVWrWmy9wPevI6fd9vHXSXMeRAGvTMJFfjejCyr2n+F3F/dA0HWY8DSd2uY4mEpZU3ESkyixYvYm7D/+JQ7W60XDQU67jSDUZ1r0p9/dJ46XFB1jU/XfgjfBtRl9a6DqaSNhRcRORKnGmsJSSd79JnCmm/ri/g0f/vNQk/3dXRzo3rcVT7x/j2M0vwtENMPtbrmOJhB39yyoiVeKdCf/gFruEk73+h6hG7V3HkWoWE+nlpft7AfDoktqU9X0WVv8b1k5wnEwkvKi4icg1W7ppD0P2/Yajca1pdMd3XMcRR1LrxvH7+7qz8WAOP8kbCs37w3vfgKObXUcTCRsqbiJyTfKLyzg67VskmzPUHv0yeCNdRxKHbunYkC8PaMkbKw4xu93PIDoRJj/k29dURK6ZipuIXJPJUydyT/lHHOv0GNHN0l3HkSDwzdva0bt5XZ6ddYSsm/4MJ3fBu18Ha11HEwl5Km4ictVW7TzEwG0/5URUUxoP+6nrOBIkIr0eXhzXg/hoL+M/iaFkwHdh49uQ+YrraCIhT8VNRK5KUWk52yd/nxaeo8SN+DNExbmOJEGkYa0Y/jimB7uz8/jWkZuwbW6FD74LB1e7jiYS0lTcROSqTJr5LqOKp3O41Shi2w12HUeC0A2t6/ONm9syfd0RpqT+ABIawpSHofCU62giIUvFTUSu2MYDx0lf90MKImvTeORvXceRIPbUoNbc2DaZH3x4iJ03/hlyDsP0J3W9m8hVUnETkStSWl7Byrd+SifPPrx3PQ+xdVxHkiDm8RieH92d+glRjP+4gsJBP4Fts2DJn1xHEwlJKm4ickUmzZ7HuIK3ONL0NuK73+M6joSAuvFR/Pn+nhzNKeKZXRnYDsNgzk9g3xLX0URCjoqbiFTajiNnaLviB5R7o2k05kXXcSSE9Eyrw/fu6MCcrdm8Wv9ZqNMMpj4Kedmuo4mEFBU3EamU8grLnDd/Q2/PFspv/hkkNnQdSULM+L7NubNLY34+9xAb+r7om6Qw7XGoKHcdTSRkqLiJSKVMnrec+3Ne4Vj960i8frzrOBKCjDH8akQX0urG8diHReQO/iXs/gQW/MZ1NJGQoeImIpe1/3g+DRd9j2hPBcnj/gbGuI4kISoxJpK/3t+TM4WlfHljByq6jYUFv4adc11HEwkJKm4icknWWqa/+SKDzSqK+n8XU7eF60gS4jo0rsX/u6czS3af5MWYr0KDDjDtS3DmoOtoIkFPxU1ELmnapxsYd/IvHE/qTNLAr7mOI2HivvRURvVK4fkFWSzLeB7KimHqI1Be6jqaSFBTcRORizp8ppCIj79PbZNP3TF/A4/XdSQJIz8d1pn2jRL5yuwcTt70OziwHOb+xHUskaCm4iYiF2StZcJbrzLMLCQv4xk8jbu4jiRhJjbKy0sP9KKs3PJoZhrlvR6DJS/C1vddRxMJWipuInJB76/awX1Hfs/puBbUvu17ruNImGpRP57fjOzK2gOn+UX5g9CkB7zzVTi5x3U0kaCk4iYiX3A8r5ic935IE3OCxNF/g4ho15EkjN3RpTGP3NCcV5YdYl6X34LBtxl9aZHraCJBR8VNRL7g9UmTGWM/4Eznh/E2u851HKkBvnt7B3qk1eZrH5zkyOAX4PA6+PC7rmOJBB0VNxH5nI/X7+fufb8kP6YBde7+mes4UkNERXj487ieRHoN45fUp/S6r0Hmv2D9FNfRRIKKipuIfOZMYSl7p/+Utp6DxA5/EaITXUeSGqRp7VieH92dbUdz+UHOPZB2Pbz7dcje5jqaSNBQcRORz/zz7fd4uHwap1rfS0S721zHkRpoYLsGPDOoNZNWH+HdNj+DyFiY/BCU5LuOJhIUVNxEBIBF244wePvPKItMpM69v3cdR2qwr9/clhta1+ObH2azb+CffCNu7z0L1rqOJuKcipuIkF9cxuopv6aHZycRd/0W4uu5jiQ1mNdjeGF0D5JiI3l4QRxF/b4F6yfC6tddRxNxTsVNRPjHzPl8qfRNTqcMJqrbKNdxREhOjObP43py4FQhzx6+BdtqMMz6lm+2qUgNpuImUsNl7jlB+oYf4/F6qT3qRTDGdSQRAHq3qMu3bmvHrE3HmND0BxBXDyY/DEVnXEcTcUbFTaQGKyotZ+6kF+jn2Yi55SeQlOI6ksjnPDGgJTd3aMgP5xxla/8X4cwBmPGUrneTGkvFTaQG+8fsZXy58BXOJGcQ3edx13FEvsAYw+9HdaNx7RgemWsoGPB/sOVdWPaS62giTqi4idRQGw+eoeXKnxDvKSVp9Evg0T8HEpyS4iL567henMgr4Su7rse2uxM+/j84sMJ1NJFqp3+pRWqg0vIKpr31Mnd6l1Pe7zmo38Z1JJFL6pKSxI+GdmThjuO8XPebvtP6U8ZD/gnX0USqlYqbSA30rzlr+XLeX8hJak/MwG+4jiNSKeN6p3FP9yb8+pPDrL3uj5B/HKZ9CSoqXEcTqTYqbiI1zI6judT+9P+RbHKoNfpv4I10HUmkUowx/PzeLrRKTuDxj0s5M/BnsGsuLNKC0VJzqLiJ1CDlFZbX3/oPoz3zKMr4KjTp4TqSyBWJj47gbw/0pKCknMc3dqKi8yj45Bewe4HraCLVQsVNpAb598ItPHbqefLiU4m75Qeu44hcldYNEvnl8C6s3HeaP0R/Feq1gbcfh9wjrqOJBJyKm0gNse9EPmXzfkELz1HiR/wFouJcRxK5asO6N+X+Pmn8+dMjfNrrD1CSB1MfhfIy19FEAkrFTaQGsNby8oRpPGreo6Dz/ZiWN7qOJHLN/u+ujnRpmsRXPszn+MBfw75PYf7PXMcSCSgVN5EaYOKy3Txw7LcUx9Qn7s5fuI4jUiViIr389f6eGGD86haUdX8IFr8ARze7jiYSMCpuImHu8JlCjn7wWzp69hF7z/MQW9t1JJEqk1o3jt/f152NB3P4ZeloiIqHhb9xHUskYFTcRMKYtZYXJ8/mq0wlv/VdmA53u44kUuVu6diQL9/YkldWnWFbs7GwaToc2+o6lkhAqLiJhLEZa7IYduDX2MhY4of9wXUckYB57tZ29G5el0e29cZGxmrUTcKWiptImDqeV8zmd1+gj2crUXf8AhIbuo4kEjARXg+/GtGFw6XxLKs/EjZOg+xtrmOJVDkVN5Ew9fzbn/C1ijcpSOmPp8cDruOIBFzL5ATu6tqE5w7294+6/dZ1JJEqp+ImEoY+3HiYQTt/SbTXEjfiz2CM60gi1eKpQa3IKolnVYMRsPFtOL7DdSSRKhXQ4maMGWKM2WaM2WmM+c4FHn/WGLPZGLPeGDPXGNPMf3yQMWbtObciY8w9/sdeM8bsOeex7oH8DCKh5kxBKQveeZmbvWvw3PR/UKe560gi1aZ9o1rc1qkh3zw4ABsRo1E3CTsBK27GGC/wF+B2oCMw1hjT8bynrQHSrbVdganAbwCstfOttd2ttd2BwUAB8NE5r3vu7OPW2rWB+gwioegPM5bwv2WvUJDcHe/1X3UdR6TaPT2oDXuL4lnbcARsmALHd7qOJFJlAjni1hvYaa3dba0tASYCw859gr+gFfi/XAakXOB9RgKzz3meiFzEwu3ZdNv8G2p78okb+VfweF1HEql2XVKSGNgumecO3Yj1RsOi37mOJFJlAlncmgIHzvk6y3/sYh4DZl/g+BhgwnnHfu4/vfq8MSb62mKKhIf84jKmT3md4d7F2BuehYadXEcSceaZwa3ZWRDHxiYjYP0kOLHLdSSRKhEUkxOMMQ8A6cBvzzveGOgCfHjO4e8C7YEMoC7w7Yu85xPGmExjTGZ2dnZAcosEkxfeX8X/lrxEYVJrIgY+5zqOiFO9mtWlb6t6PHdoINYbBQs16ibhIZDF7SCQes7XKf5jn2OMuRn4PjDUWlt83sP3Ae9Ya0vPHrDWHrY+xcCr+E7JfoG19u/W2nRrbXpycvI1fhSR4LZy70marP49TcwJYke+BBEaiBZ5enBrtubFsbXpSN+o28ndriOJXLNAFreVQBtjTAtjTBS+U54zz32CMaYH8DK+0nbsAu8xlvNOk/pH4TDGGOAeYGPVRxcJHUWl5fx70iQejviIsl6PQ+oF/7+MSI1zfct69GpWh28fGYT1RsLC37uOJHLNAlbcrLVlwNP4TnNuASZbazcZY35qjBnqf9pvgQRgin9pj8+KnTGmOb4RuwXnvfWbxpgNwAagPvCzQH0GkVDw4keb+Fr+i5TENSby1h+7jiMSNIwxPDO4NevPxLIjZQSsmwAn97iOJXJNIgL55tbaWcCs84798Jz7N1/itXu5wGQGa+3gKowoEtI2ZJ0hZtnztPEehHvfhugE15FEgsqNbZPp0jSJ7xy9ibc9UzGLfg/D/uw6lshVC4rJCSJy5UrKKvjLpBl81TuDkk73QZuL/v8gkRrLGMPTg1uz+lQMe9L8o26n9rmOJXLVVNxEQtTLn2znK2deoCI6iag7f+06jkjQuqVDQ9o1TOS72bdgjQcW6Vo3CV0qbiIhaPvRXHIW/Jnunl1E3fVbiKvrOpJI0PJ4fKNuy49Hs6/ZSFj7Jpze7zqWyFVRcRMJMeUVlt9P+pBnvZMoaXkrdB7hOpJI0LujS2Na1o/n/07c6h91+4PrSCJXRcVNJMS8ung3D2Q/T0REJFHDXgBjXEcSCXpej+HJQa1ZdDSKg81Hwpr/wOkDl3+hSJBRcRMJIXuP57N7zt/p791IxG3/D5IutYuciJxrWPcmpNaN5UenbsUCLNaom4QeFTeREFFRYfnllE/4jnmDkqbXY3o94jqSSEiJ9Hr46o2tmXsoiiMtR8LqN+BMlutYIldExU0kRExYuZ+hh14g3ltK1L1/Bo9+fEWu1IheTWmcFMP/OzPEd2Dx824DiVwh/csvEgIOnS5kxax/c6d3BZ6B34H6rV1HEglJ0RFevjygJbMORHKs1QhY/W8484VttEWCloqbSJCz1vKzt5fyfV6hpH5nzA1fcx1JJKSN6Z1G/YQofpl3B9gK+PQF15FEKk3FTSTIfbjpCP33/In6nhyihv8FvJGuI4mEtJhIL1/q35J39kRwvPVIWPU65Bx2HUukUlTcRILcwo9nMjZiPlz/NDTp7jqOSFi4/7pm1I6L5Lf5d4It16ibhAwVN5EgtmrfKW45+SaFUfXwDPqu6zgiYSMhOoLHbmjBpF1eTrUeAateg9wjrmOJXJaKm0gQe3/efAZ51+Ht8wRExrqOIxJWHurbnMToCP5QcjeUl8Knf3QdSeSyVNxEgtSBkwW03v0GZSaKqOsedx1HJOwkxUbycN/m/Gebh5y2IyDzXxp1k6Cn4iYSpCYvWMtwzyKKO90H8fVdxxEJS4/2a0FspJc/lQ71j7r9yXUkkUtScRMJQrlFpUStfY0YU0r8gGdcxxEJW3Xjo3jgumb8a4uHvLbDfaNuecdcxxK5KBU3kSA0dcVuxvABOSk3QoP2ruOIhLXH+7cg0uvhrxX3QnmxrnWToKbiJhJkyisshxa/QbI5Q62BX3cdRyTsNUiMYWzvNP6+yZDf7l5Y+QrkZbuOJXJBKm4iQebjTYcZXjSD3FptoNVg13FEaoQnBrTEGPgnI3yjbkt0rZsEJxU3kSCzfN50Onj2E3/j18AY13FEaoQmtWMZ2SuFv2z0UNjuXlj5T8g/7jqWyBeouIkEkXUHTtPv+GQKo+ri6Xqf6zgiNcpXb2xNeYXlVe9IKC2EJS+6jiTyBSpuIkHk3XkLuMm7Bk/vxyEyxnUckRolrV4cw7o34U/rDcXt74UV/4D8E65jiXyOiptIkDh8ppAWO/9NmYki+ronXMcRqZGeHNia4rIK3oi+D0oLYOmfXUcS+RwVN5EgMWnBeoZ7FlLUYSQkJLuOI1IjtW6QwB1dGvPCWg8l7YfBir9DwUnXsUQ+o+ImEgTyi8vwrP4XsaaEhBu14K6IS08Pak1ecRkTY8dASb5G3SSoqLiJBIF3Mvcw2n7AmSb9oWFH13FEarQOjWtxS8eG/H6tl9L2Q2G5Rt0keKi4iThWUWHZv/A/NDSnqTVIC+6KBINnBrfmTGEp0xLGQUkuLPur60gigIqbiHPzthxlWOE75CS2wrS+2XUcEQG6ptTmxrbJ/GaNl7J2d8Pyl6HwlOtYIipuIq4tmTedTp59xGnBXZGg8szg1pzIL2Fm7QehOAeWveQ6koiKm4hLmw6d4fpjkyiMrE1Et9Gu44jIOdKb1+W6lnX59Rov5e3ugmV/g8LTrmNJDafiJuLQzLkLucmzBpPxOETGuo4jIud5ZnAbjuYUM7vew1B8Bpb/zXUkqeFU3EQcOZZTROr216nwRBDT98uu44jIBfRtVY+eabX55eoIKtrd6ZukUHTGdSypwVTcRByZvNC/4G774ZDQwHUcEbkAYwzPDG7DwdOFzGkw3lfalr/sOpbUYCpuIg4UlZZTsepV4kwxCQO1BIhIMBvYLpnOTWvxi9WR2La3+xbk1aibOKLiJuLA9My93Fcxm9ONb4CGnVzHEZFLMMbw9KA27D1RwIImj/pH3f7uOpbUUCpuItWsosKye+F/aGROkaQFd0VCwq0dG9K2YQI/Xx2NbXObf9Qtx3UsqYFU3ESq2YLtx7g7fxq5CS0xrW9xHUdEKsHjMTw1qDU7juWxNPVxKDrt24BepJqpuIlUs0/nzqSLZy+xA54Bj34ERULFXV2b0KJ+PD9fG4ttc6tv1K0413UsqWH0W0OkGm07kkvvIxMojEgiosdY13FE5Ap4PYYnB7Zi06EcMpt/ybcF1op/uI4lNYyKm0g1mjF3ETd7VkPGY1pwVyQE3dOjKU1rx/LztfHY1rfAkhehOM91LKlBVNxEqsnxvGIab3uNCo+XWC24KxKSIr0evjqwFWsPnGZ9669A4UlY+U/XsaQGUXETqSZTFm9guPmEgnb3QmIj13FE5CqNSk+hYa1ofrEuHlrd5Bt1K8l3HUtqCBU3kWpQVFpO6YpXiTfF1NKCuyIhLTrCy5cHtGL5npNsbvskFByHla+4jiU1hIqbSDV4d80+RpbP4nSj66FRF9dxROQaje2dRv2EKH61qRa0HASf/lGjblItVNxEAsxay65P3qSJOakFd0XCRGyUl8f7t2Th9mx2dHzKN+qW+S/XsaQGUHETCbBPdxzn9rxp5MQ3x7S5zXUcEakiD1zXjKTYSH69qQ60uNE/6lbgOpaEORU3kQBbOPddunl2E9tfC+6KhJOE6AgevaEFc7YcZW/nZyA/G1a96jqWhDn9FhEJoJ3H8uh56C0KI5KI7DnOdRwRqWLj+zYnITqC326tB837+0bdSgtdx5IwpuImEkDT5y3mVk8mttcjEBXnOo6IVLGkuEge7tuMWRsPk9Xta5B3FFa95jqWhDEVN5EAOZVfQoPNr1Lh8RJ3w1dcxxGRAHn0hhbERHj5w/YGvlG3xS9AaZHrWBKmVNxEAmTqkk0MN5+Q32YY1GrsOo6IBEi9hGju75PGjHWHONLj65B3BFa/7jqWhCkVN5EAKCmroGDpv0gwRVoCRKQGeGJAS7wewws7G0KzG2Dx8xp1k4BQcRMJgPfX7mdk+fucatAHGndzHUdEAqxBrRjGZKTy9uosstO/AbmHYc0brmNJGFJxE6li1lq2zX+TpuYEtQdrtE2kpvjyja0A+POuRpB2PSz6A5QVO04l4UbFTaSKLd99giG5b5Mbl4Zpe7vrOCJSTZrWjmVEzxQmZGZxKuMbkHsIVv/bdSwJMypuIlXsk7nv092zixgtuCtS43x1YCvKyiv4674USO3ju9ZNo25ShfRbRaQK7T2eT9cD/6EwohaRve53HUdEqlmzevEM696U/yw/QE6f/4Wcg7DmP65jSRhRcROpQu/M+5TbPCup6PEwRMW7jiMiDjw1qBVFZeW8nJUGKRn+UbcS17EkTKi4iVSRM4Wl1N30KhgP8f2fdB1HRBxp3SCROzo35vWl+8m//jk4cwDWvuk6loQJFTeRKjJtySaGM5/c1kOhVhPXcUTEoacGtSavuIxXDreApun+GaYadZNrp+ImUgVKyyvIWfIqiaZQS4CICB2b1OLmDg3415K9FN7wHJzZD+smuI4lYUDFTaQKfLA+ixFl73EqOQOa9HAdR0SCwNOD23C6oJTXj7WGJj1h0e+gvNR1LAlxAS1uxpghxphtxpidxpjvXODxZ40xm40x640xc40xzc55rNwYs9Z/m3nO8RbGmOX+95xkjIkK5GcQuRxrLZvnv0WKOa7trUTkM91Ta9O/TX3+uXgPxf2+Baf3w7qJrmNJiAtYcTPGeIG/ALcDHYGxxpiO5z1tDZBure0KTAV+c85jhdba7v7b0HOO/xp43lrbGjgFPBaozyBSGav3n+Lm01PJiU3F0/4O13FEJIg8M7gNx/NKeOtkO99o/MLfatRNrkkgR9x6AzuttbuttSXARGDYuU+w1s631hb4v1wGpFzqDY0xBhiMr+QBvA7cU5WhRa7U3I/fp5dnBzH9ngSP13UcEQkivVvUpU+Lury8cA+l/Z6D0/tg/STXsSSEBbK4NQUOnPN1lv/YxTwGzD7n6xhjTKYxZpkx5h7/sXrAaWttWSXfUySgDpwsoOO+/1DkTSAq/SHXcUQkCD0zuA1HcoqYnNMJGneDhb+D8rLLv1DkAoJicoIx5gEgHfjtOYebWWvTgXHAC8aYVlf4nk/4i19mdnZ2FaYV+a935i/lds9yyno8DNEJruOISBC6oXU9uqfW5qUFuynr/y04tQc2THYdS0JUIIvbQSD1nK9T/Mc+xxhzM/B9YKi19rMN3ay1B/1/7gY+AXoAJ4DaxpiIS72n/3V/t9amW2vTk5OTr/3TiJwnt6iUWuv/BcaQoAV3ReQijDE8M7g1WacKmV7QDRp18V/rplE3uXKBLG4rgTb+WaBRwBhg5rlPMMb0AF7GV9qOnXO8jjEm2n+/PnADsNlaa4H5wEj/Ux8GZgTwM4hc1LSlWxjOXHJa3QVJl7w8U0RquMHtG9CxcS3++skuygd8G07uho1TL/9CkfMErLj5r0N7GvgQ2AJMttZuMsb81Bhzdpbob4EEYMp5y350ADKNMevwFbVfWWs3+x/7NvCsMWYnvmveXgnUZxC5mPIKy+klr1LLFFJn8P+4jiMiQe7sqNvu4/m8X9ITGvpH3SrKXUeTEGN8g1jhLT093WZmZrqOIWFk9vosOk8dSHxyGnWfmec6joiEgIoKy20vLMRjDLNvPYVnykMw/B/Q9T7X0STIGGNW+a/z/4KgmJwgEmo2zn2LVE82SdreSkQqyeMxPD24NduO5vJRRQY06AQLfqNRN7kiKm4iV2jtgdMMPDWF3NimeDvc5TqOiISQO7s0pnm9OP78yS7sjd+CEztg0zuuY0kIUXETuUJzPp5Fhmc7UTc8pQV3ReSKRHg9PDmwNRsP5vCJ9zpo0BEW/FqjblJpKm4iV+DQ6ULa7fk3Rd54ojO04K6IXLl7ejSlae1YXpy3CzvgOTi+XaNuUmkqbiJX4J35y7jds5zSbg9BdKLrOCISgqIiPHxlYCtW7z/N0uh+kNzeP8O0wnU0CQEqbiKVlF9cRuy6f2EMJA54ynUcEQlho3ql0CAxmhfn74YBz0H2Vtg83XUsCQEqbiKVNH35dkbaOZxpcQfUTr38C0RELiIm0ssTA1qydPcJMuNvhPrt/DNMNeoml6biJlIJ5RWW44tfoZYpoO5N33AdR0TCwLg+adSLj+LFT/bAjd+C7C2wZeblXyg1moqbSCXM3XyIe4pmcrJuD0i54JqIIiJXJC4qgsf6t2DB9mzWJw2C+m016iaXpeImUgnr506gmecYSYO04K6IVJ0Hr2tGUmwkf/5kj+9at2ObYOt7rmNJEFNxE7mMjQfP0P/EFHJjmuDteLfrOCISRhJjInnkhuZ8tPkoW+rdAvVaa9RNLknFTeQyPvp4Nn08W4ns+1XwRriOIyJhZnzf5iRER/CXBf5Rt6MbYNss17EkSKm4iVzC0ZwiWu36N8WeOGJ6j3cdR0TCUO24KB68vhnvbzjMrkZDoG5LWPArsNZ1NAlCKm4ilzDtk+Xc4VlGcdcHIaaW6zgiEqYe69eC6AgPf1mw1zfqdkSjbnJhKm4iF1FYUk706lfwGkutG7XgrogETv2EaO7v04wZaw+xv+ldUKcFLPqD61gShFTcRC5i5srtjLAfc6rZEKjTzHUcEQlzTwxoidcYXlq0D/p8BQ5mwuF1rmNJkFFxE7mAigrLkYX/IskUUPem/3EdR0RqgIa1YrgvI4Wpqw5wpMUwiIiFzFddx5Igc9EpcsaYZyvx+nxr7ctVmEckKCzYdpShhTM4Wa8bddP6uI4jIjXEV25sxcQVB/jb8pP8uPMI2DAFbv1/EJ3oOpoEiUuNuD0HJACJl7j9b6ADiriwZs5EWniOUksL7opINUqpE8fwnk2ZsGI/JzuOg5I8WD/ZdSwJIpdalOoNa+1PL/ViY0x8FecRcW7rkRz6Zk8kN7YRiZ2GuY4jIjXMkwNbM3VVFi/vqMN3G3XxnS5NfxSMcR1NgsClRtwuO53FWvutKswiEhQ++OgDrvNsIeJ6LbgrItWvef147ujSmAkrD1Da4xHfgrwHV7mOJUHiUsVtrTFmjjHmMWNM7eoKJOJSdm4xLXa+TrEnltjrHnEdR0RqqPv7NCOnqIz36QdRCZD5L9eRJEhcqrg1BX4L9AO2GWNmGGPGGGNiqyeaSPV7Z8FK7jBLKepyP8QkuY4jIjXUdS3r0jI5nn+vOg5d74ONb0PhKdexJAhctLhZa8uttR9aax8BUoF/AcOAPcaYN6sroEh1KSotJ2LVP/EaS9LAZ1zHEZEazBjDuN5prN5/mj3NR0NZEayb6DqWBIFKreNmrS0BNgNbgBygQyBDibjwXuYOhld8xKm0W6FOc9dxRKSGG9EzhagID6/tSoCUDN/pUu1fWuNdsrgZY1KNMc8ZY1YD7/mfP9Ra27Na0olUE2sthxa8Sm2TrwV3RSQo1ImP4o7OjZi25iAl3cfD8e2w71PXscSxixY3Y8wSYDHQAPiStbadtfbH1tqt1ZZOpJos3nGMuwqmc7J2F0zada7jiIgAMLZ3GrlFZbxb3sd33a0mKdR4lxpx+w7Q3Fr7nLVW85AlrGV+PImWniMkDvqa1koSkaDRu0VdWjdI4I3MY9D9ftg8E/KyXccShy41OWGhtdYaY9oaY+YaYzYCGGO6GmN+UH0RRQJr57Fc+hyZQG50QyI73+s6jojIZ4wxjO2dxtoDp9mZNgoqSmHtf1zHEocqMznhH8B3gVIAa+16YEwgQ4lUp1kff0xf72Y8fb4M3kjXcUREPmdEz6ZERXh4fXsUNOsHq16DigrXscSRyhS3OGvtivOOlQUijEh1O5lfQuq2Vyn2xBJ//WOu44iIfEHtuCju6tKY6WsOUtz9YTi1F3bPdx1LHKlMcTtujGkFWABjzEjgcEBTiVST6QtXcaf5lIJOYyG2tus4IiIXNK5PGrnFZcws7glx9TVJoQarTHF7CngZaG+MOQj8D/CVQIYSqQ7FZeXYlf8gwlRQZ5AW3BWR4NWrWR3aNEjgP5lHoMcDsG025BxyHUscuNRyINcbY4y1dre19mYgGWhvre1nrd1XfRFFAmP2qt0ML/+QEym3QN2WruOIiFyUMYZxfdJYl3WGbSkjwJbD6jdcxxIHLjXi9hCwyhgz0RgzHki01uZWTyyRwLLWcmDBq9QxedS/+X9cxxERuazhPVKIjvDw+lYDrW6C1a9DuS45r2kutRzIV/07JPwYqAO8ZoxZaoz5hTFmgDHGW10hRarasl3HuSNvGieSOmGa9XUdR0TkspLiIrmraxNmrDlIUfeHIecg7PjIdSypZpe9xs1au9Va+7y1dggwGN9uCqOA5YEOJxIoKz6eRCvPYRIHfl0L7opIyBjXJ438knKm53eBxMaapFADVWqT+bOstYXW2lnAd6216QHKJBJQe47n0+vwW+RGNSCq63DXcUREKq1nWm3aN0rkzZWHoOfDsHOOb3kQqTGuqLidY3OVphCpRrPmfEw/zyZMnye04K6IhJSzOylsOHiGLU3u9Z0xWPW661hSjSIu9oAx5tmLPQQkBCaOSGCdKSil8eZ/UeKNIaHv467jiIhcsXt6NOWXs7fw700l/LLtEFjzBgz8LkREuY4m1eBSI26/wDcpIfG8W8JlXicStN5ZtJo7zafkdRwDsXVcxxERuWJJsZHc3bUJM9YeoqDrQ5CfDdvedx1LqslFR9yA1cB0a+2q8x8wxmioQkJOaXkF5cv/QaQpp+7gr7mOIyJy1cb1SWPKqiym53ZgXO003ySFTve6jiXV4FIjZ48AF1toVxMTJOR8sHYP95R/wImmg6FeK9dxRESuWvfU2nRoXIs3VxzE9hwPexbC8R2uY0k1uNQ6btustccv8tjRwEUSqXrWWvbNf5V6Jpd6N33DdRwRkWtijGFc71Q2Hcphc6Oh4ImAVa+5jiXV4FJbXv34ci+uzHNEgkHm3pPcljONE7U64GnRz3UcEZFrNqxHU2Ijvfx7fSF0uBvWvgmlha5jSYBd6hq3x40xOZd43ABj8O2sIBLUln40ma95DlI88AdacFdEwkKtmEiGdmvCzHWH+OEDDxG/6R3YPAO6jXEdTQLoUte4/YMvzig9f3bpPwIdUORaHThZQI+Db5IbmUx015Gu44iIVJlxfdIoLC1n2smWUK+1dlKoAS464mat/Ul1BhEJlPfmzOWrng3k9v6e1jkSkbDSNSWJTk1q8daKAzyQ8Qjmo+/DkY3QqLPraBIgWo9NwlpOUSkNNr1CsYkh8YYvuY4jIlKljDGM65PGlsM5bEi+E7zRsOpV17EkgFTcJKy9u3gtd7GYvPb3QVxd13FERKrc0G5NiIvy8sbaHN9abusmQXGe61gSICpuErbKyisoWvoPok0p9W76uus4IiIBkRgTybDuTXh3/SHyujwEJbmw8W3XsSRALlvcjDG/McbUMsZEGmPmGmOyjTEPVEc4kWvx8fr9DCubzbHGg6B+a9dxREQCZlzvZhSVVvD2sSbQoJMmKYSxyoy43WqtzQHuAvYCrYHnAhlKpCrsnvcv6psc6t2sBXdFJLx1SUmiS9MkJqw8gE1/BA6vhYOrXceSAKhMcTs78/ROYIq19kwA84hUiY1Zp7n5zNucSGiHt+UA13FERAJuXJ80th7JZW2d2yAyXqNuYaoyxe09Y8xWoBcw1xiTDBQFNpbItVmx8H3aebKI7f+0FtwVkRphaLcmxEd5+c+aU9BlpO86t8LTrmNJFbtscbPWfgfoC6Rba0uBAmBYoIOJXK3isnLqbp9CoSeOuB4jXMcREakW8dERDOvRlPfWHyK384NQWgDrJ7uOJVWsMpMT4oAngZf8h5oA6YEMJXItFmzYwy12Caeb3wlR8a7jiIhUm3G90yguq2Dq4frQpKfvdKm1rmNJFarMqdJXgRJ8o24AB4GfBSyRyDXa/+kk4k0xDQY85jqKiEi16tw0iW4pSby1fL9vkkL2Fti/zHUsqUKVKW6trLW/AUoBrLUF+DaYFwk6x3KK6HTsPU7FpOBtdp3rOCIi1W5cnzR2HMtjTeJgiE7SJIUwU5niVmKMiQUsgDGmFVAc0FQiV2nOkhVc79mM7TZOkxJEpEa6u1sTEqIj+M/q49BtNGyeDvknXMeSKlKZ4vZj4AMg1RjzJjAX+HYgQ4lcDWstpWveogJD3b4Pu44jIuJEXFQE9/RownsbDpPT6QEoL4F1b7mOJVWkMrNKPwKGA+OBCfhml84PcC6RK7Z2/0kGF87hWP3rICnFdRwREWfG9W5GSVkFUw7UgrTrIfNVqKhwHUuqQGVmlc611p6w1r5vrX3PWnvcGDO3OsKJXInMhe+R6skm6XqNtolIzdaxSS26p9bmreX7sL0egZO7YO9C17GkCly0uBljYowxdYH6xpg6xpi6/ltzoGll3twYM8QYs80Ys9MY850LPP6sMWazMWa9fx/UZv7j3Y0xS40xm/yPjT7nNa8ZY/YYY9b6b92v9ENL+CkqLafBrrcp9MQT20XLDIqIjOuTxq7sfFbG9YfYupqkECYuNeL2ZWAV0N7/59nbDODPl3tjY4wX+AtwO9ARGGuM6Xje09bgO/XaFZgK/MZ/vAB4yFrbCRgCvGCMqX3O656z1nb339ZeLouEv7lrd3GLXcaZlndDVJzrOCIizt3dtQmJMRG8teoo9Lgftr4PuUdcx5JrdNHiZq39o7W2BfBNa21La20L/62btfayxQ3oDey01u621pYAEzlvxwVr7Xz/8iIAy4AU//Ht1tod/vuHgGNA8hV/OqkxDi6ZSJwppkH/R11HEREJCrFRXob3aMqsjUc40/F+qCiDNW+4jiXXqDKTE140xnQ2xtxnjHno7K0S790UOHDO11lc+hTrY8Ds8w8aY3oDUcCucw7/3H8K9XljTHQlskgYO3i6kG4n3udkbDM8ab1dxxERCRpj+6T5JinsiYaWA2HV61BR7jqWXIPKTE74EfCi/zYI3+nMoVUZwhjzAL5ttH573vHGwBvAI9bas9Nhvovv9G0GUJeLLE1ijHnCGJNpjMnMzs6uyrgSZOZ+uow+nq14etyvtdtERM7RvlEteqbV5q0V+32TFM4cgJ1zXMeSa1CZddxGAjcBR6y1jwDdgKRKvO4gkHrO1yn+Y59jjLkZ+D4w1FpbfM7xWsD7wPettZ/t12GtPWx9ivFtx3XBIRZr7d+ttenW2vTkZJ1lDVfWWuzat6jAQ+3rHnQdR0Qk6Izr04zd2fksj7oOEhpqkkKIq0xxK/SPdpX5y9QxPl/ILmYl0MYY08IYEwWMAWae+wRjTA/gZXyl7dg5x6OAd4B/W2unnveaxv4/DXAPsLESWSRMrdxzgptK5nEs+Xqo1cR1HBGRoHNX18bUiongrczD0PMh2PERnD5w+RdKUKpMccv0z+j8B75ZpauBpZd7kbW2DHga+BDYAky21m4yxvzUGHP2VOtvgQRgin9pj7PF7j5gADD+Ast+vGmM2QBsAOqjDe9rtDULZpJijlPnhkdcRxERCUoxkV6G90zhg41HON1+LFgLq//tOpZcJWOtrfyTfWu41bLWrg9YogBIT0+3mZmZrmNIFcsvLmPuL+7hFu8aYr+7CyJjXEcSEQlK24/mcuvzC/neHe154sB34fA6+MZG8Ea6jiYXYIxZZa1Nv9Bjldo54ex9a+1ea+167ZwgweDjNTu5heXkthmm0iYicgltGyaS3qwOE1YcwKY/AnlHYNsXFnKQEBDQnRNEAuno0reINSUka+02EZHLGtcnjT3H81lqekKtFE1SCFEB2zlBJJD2nyig56nZnIxrgWnay3UcEZGgd0eXxiTFRvJm5kHoNR52z4cTuy77Ogkugdw5QSRg5ny6hAzPdiJ6au02EZHKiIn0MqJnCh9tOsKJdqPBeGHVa65jyRW61KnSDGNMI2vti/6vHzLGzDDG/Ml/ClXEiYoKi3edb+22Wn20dpuISGWN65NKabllyrZSaH8nrPkPlBVf/oUSNC51qvRloATAGDMA+BXwb+AM8PfARxO5sKU7j3FL2XyyG/aDxEau44iIhIzWDRLp3bwuE1bsp6LXo1B4EjbPvPwLJWhcqrh5rbUn/fdHA3+31r5trf0/oHXgo4lc2PpFM2liTlLnhvGuo4iIhJxxfdLYd6KAJRWdoG5LTVIIMZcsbsaYCP/9m4B55zwWcYHniwRcTlEpqfumUeBNJKrjXa7jiIiEnCGdG1EnLpK3Vh6AXo/A/iVwbIvrWFJJlypuE4AFxpgZQCGwCMAY0xrf6VKRavfRqm3cbFaS3/ZeiIh2HUdEJOT8d5LCUbJbjwBvlCYphJBLzSr9OfC/wGtAP/vfLRY8wDOBjybyRdnLJhJjSqnfT1tciYhcrbF90iirsEzZUggdh8HaCVBS4DqWVMIld06w1i6z1r5jrc0/59h2a+3qwEcT+bxd2Xn0OTObE/GtMU16uI4jIhKyWiUncF3LukxccYCKno9A8RnYNM11LKmEymwyLxIU5i1eTE/PTqLSH9DabSIi12hs7zT2nyxgcUkbSG6vSQoh4lLruOkCIgka5RWWqA0TKcdDYvo413FERELekM6NqBsfxVsrDkD6o3BwFRxa6zqWXMalRtyWAhhj3qimLCIXtXDbEW4r/4QTjQdAYkPXcUREQl50hJeRvVL4eMtRslvcAxGxsOpV17HkMi5V3KKMMeOAvsaY4effqiugCMDmxTNoZE5R5wZtKC8iUlXG9k6jvMIyaWMOdBkB66dAUY7rWHIJlypuXwH6A7WBu8+7aQEtqTanC0pofmA6Bd4kItvf7jqOiEjYaFE/nr6t6jFhxQEqej4KpfmwYbLrWHIJF11I11q7GFhsjMm01r5SjZlEPueDzK3cazLJa38/cRFRruOIiISVsb3TeGbCGhbmd2Jg426w8l+Q/pgmgQWpyswqfcMY8zVjzFT/7RljTGTAk4n4nVw+gWhTSr1+Ok0qIlLVbuvUiHrnTlI4tgmyVrqOJRdRmeL2V6CX/8+/Aj2BlwIZSuSsrUdy6Jv7IScT2kKjrq7jiIiEnagIDyPTU5i79RjHmt0NUYmQqUkKwaoyxS3DWvuwtXae//YIkBHoYCIAnyxaRHfPLqIzHtSwvYhIgIzN8E1SmLjuJHS9z7cYb8FJ17HkAipT3MqNMa3OfmGMaQmUBy6SiE9peQVxmydRjpf4XmNdxxERCVvN68fTr3V9Jq08QHnP8VBWBOsmuo4lF1CZ4vYcMN8Y84kxZgEwD98epiIBtWDLYYZULOBEk4GQkOw6johIWBvXJ42DpwtZmNMIUnr7dlL4bJtyCRaXLW7W2rlAG+Br+DaXb2etnR/oYCKbF0+ngTlNvRu0obyISKDd3KEh9ROieHP5ft8khRM7YO9i17HkPJXaq9RaW2ytXe+/FQc6lMjxvGLaHJpBfkQdvO2HuI4jIhL2oiI8jEpPZd7WoxxOvQ1iamv/0iCkTeYlKM1esYnBZhUlHUeAV6vPiIhUh7EZaVRYmLTmOHS/H7a8C3nHXMeSc6i4SdCx1pKzchLRpow6fce7jiMiUmOk1YujfxvfJIWyHg9DRSms+Y/rWHKOyxY3Y8w0Y8ydxhiVPKkWmw7l0D//Q07Uag+NuriOIyJSo9zfJ43DZ4pYcLI2NO/v23i+osJ1LPGr7AK844AdxphfGWPaBTiT1HALFn1CV88e4jIech1FRKTGualDQ5ITo3nr7CSF0/th1zzXscSvMrNK51hr78e3Y8JeYI4xZokx5hFtfSVVrbisnIStUygjgtieY1zHERGpcSK9Hu5LT2H+tmMcanwTxCf7Rt0kKFTq9Kcxph4wHngcWAP8EV+R+zhgyaRGmrfpIHfYBZxKvQni67mOIyJSI43JSMMCE1cfhR4PwLbZcOag61hC5a5xewdYBMQBd1trh1prJ1lrnwESAh1QapZti98h2eRQV5MSREScSa0bx4A2yUxauZ+y7g+BrYA1b7iOJVRuxO0f1tqO1tpfWmsPAxhjogGstekBTSc1ytGcItofeZf8yDp4297iOo6ISI02rk8aR3OKmXc0DlrfBKteh/Iy17FqvMoUt59d4NjSqg4iMmvZRgZ7VlPW6T6t3SYi4thN7RvQIDGaCSv8kxRyD8GOD13HqvEuWtyMMY2MMb2AWGNMD2NMT/9tIL7TpiJVxlpL/qoJRJlykq5/2HUcEZEaL8LrYXRGKp9szyYruT8kNtFOCkHgUiNutwG/A1KAPwC/99+eBb4X+GhSk6w5cJqBhXM4mdQRGnZyHUdERIDRGakATFp1GHo9DDvnwsk9jlPVbBctbtba1621g4Dx1tpB59yGWmunVWNGqQEWLZpPZ89e4vtotE1EJFik1IljYNtkJq08QGm3B8B4YPXrrmPVaJc6VfqA/25zY8yz59+qKZ/UAIUl5dTdPoUyE0l09/tcxxERkXOM69OMY7nFzD0YAe1u922BVVbiOlaNdalTpfH+PxOAxAvcRKrExxv3cweLOJ16M8TVdR1HRETOMahdMo1qxfDWiv2Q/gjkZ8PW91zHqrEiLvaAtfZl/58/qb44UhPt+vQdhppcKrR2m4hI0Dk7SeFP83ZwYNiNpNZu5puk0Hm462g1UmUW4P2NMaaWMSbSGDPXGJN9zmlUkWty8HQhnY69R35kPTxtbnYdR0RELmB0RioGmJiZBb3Gw95FkL3ddawaqTLruN1qrc0B7sK3V2lr4LlAhpKaY/bS9QzyrKW8y2jwXnQAWEREHGpSO5ZB7RowOTOL0q7jwBMJq15zHatGqkxxO/vb9E5girX2TADzSA1iraVozUQiTTm1rtNsUhGRYDauTxrZucXM2W+hw92w9k0oLXQdq8apTHF7zxizFegFzDXGJANFgY0lNcGK3Se4qehjTtbuAg3au44jIiKXMLBdA5oknZ2k8CgUnYZN013HqnEuW9ystd8B+gLp1tpSIB8YFuhgEv4+/XQ+HTwHSOjzkOsoIiJyGV6PYXRGGot2HGd/Yk+o10Y7KThQmRE3gPbAaGPMQ8BI4NbARZKaIL+4jAa7plJqoojqPsp1HBERqYTRGal4DEzIPOAbdctaAUc2uI5Vo1RmVukb+La+6gdk+G/pAc4lYe6Dtfu4k8XkNrsVYuu4jiMiIpXQKCmGwe0bMiXzACWdR0NEDGS+6jpWjVKZaXzpQEdrrQ10GKk59iydRh2Th71hvOsoIiJyBe7vk8acLUf5eE8Jd3YaDusnwy0/hegE19FqhMqcKt0INAp0EKk59p3Ip/uJ98mLSsa0Guw6joiIXIEBbZNpWjuWt1bs850uLcmFjVNdx6oxKlPc6gObjTEfGmNmnr0FOpiEr9lL1zLQsw66jQGP13UcERG5Al6PYUxGKp/uPMHemA7QsAusfAV0Yq5aVOZU6Y8DHUJqjooKS+nayUSYChJ6azapiEgoui8jlRfm7mBC5gG+mz4e3v9fOLQamvZyHS3sVWY5kAX4dkyI9N9fCawOcC4JU0t2HufWkjmcrNMNktu6jiMiIlehYa0YbmrfgKmZWZR0HAmR8VoapJpUZlbpl4CpwMv+Q02B6QHMJGFs2adzaOfJIlE7JYiIhLRxfdI4kV/ChzsLoOso2PA2FJ52HSvsVeYat6eAG4AcAGvtDqBBIENJeMopKqXRnmmUmigiu45wHUdERK7BgDbJpNSJ5a3l/p0Uygph/STXscJeZYpbsbW25OwXxpgIQFcgyhWbtXofd5lPyWsxBGJru44jIiLXwOMxjO2dxtLdJ9gd0cp3fVvmvzRJIcAqU9wWGGO+B8QaY24BpgDvBjaWhKMDy6ZS2+RTu+8jrqOIiEgVGJWeQoTHMOHs/qXZW2H/Utexwlplitt3gGxgA/BlYBbwg0CGkvCz81guvU7NJi+6Iablja7jiIhIFWiQGMMtHRsydVUWxe2HQXSSJikEWGVmlVbgm4zwpLV2pLX2H9pFQa7U7KXruNGzDtNda7eJiISTsb3TOFVQygfbcqD7WNg8A/JPuI4Vti5a3IzPj40xx4FtwDZjTLYx5ofVF0/CQVl5BaybiNdY4ntrNqmISDjp17o+aXXjfJMUej0C5SWw9k3XscLWpUbcvoFvNmmGtbautbYu0Ae4wRjzjWpJJ2Fh0fZshpTN41S9nlCvles4IiJShTwew5jeqSzfc5KdpECzG2DVq1BR4TpaWLpUcXsQGGut3XP2gLV2N/AAoCXvpdJWLplDG89Brd0mIhKmRvVK/fwkhZO7Yc8C17HC0qWKW6S19vj5B6212UBk4CJJODldUELK3mmUmmgiugx3HUdERAIgOTGa2zo14u3VWRS1vgPi6mmSQoBcqriVXOVjIp95d9Ue7vIsIb/VnRBTy3UcEREJkHF90jhdUMoHW09B9/th6/uQc9h1rLBzqeLWzRiTc4FbLtClugJKaDu0fCq1TAG1++o0qYhIOLu+ZT2a1Ts7SWE82HJY8x/XscLORYubtdZrra11gVuitbZSp0qNMUOMMduMMTuNMd+5wOPPGmM2G2PWG2PmGmOanfPYw8aYHf7bw+cc72WM2eB/zz8ZY8yVfmipHlsO53DdmQ/Ii2kMzQe4jiMiIgF0dieFFXtPsqOsAbQcBKteg4py19HCSmUW4L0qxhgv8BfgdqAjMNYY0/G8p60B0q21XfFtZP8b/2vrAj/CN4u1N/AjY0wd/2teAr4EtPHfhgTqM8i1+XDpavp5NuDtMQ48AfufmoiIBImRvVKI9BreOjtJIScLdnzsOlZYCeRv097ATmvtbv9epxOBYec+wVo731pb4P9yGZDiv38b8LG19qS19hTwMTDEGNMYqGWtXeZfBPjfwD0B/AxylUrLK/BumIzXWGIzHnAdR0REqkH9BP8khVVZFLW8FRIaaZJCFQtkcWsKHDjn6yz/sYt5DJh9mdc29d+v7HuKI/O2HOWO8nmcTk6Hui1dxxERkWoyrk8aOUVlzNp8HHo+BDs+gtP7XccKG0Fx/soY8wCQDvy2Ct/zCWNMpjEmMzs7u6reVipp9ZKPaOU5TOJ1411HERGRanR9y3q0qB/vm6TQ8yEwBlb/23WssBHI4nYQSD3n6xT/sc8xxtwMfB8Yaq0tvsxrD/Lf06kXfU8Aa+3frbXp1tr05OTkq/4QcuWO5xXT/MAMSjwxeDvf4zqOiIhUI2MMY3unkrnvFNuLa0Prm2HNm5qkUEUCWdxWAm2MMS2MMVHAGGDmuU8wxvQAXsZX2o6d89CHwK3GmDr+SQm3Ah9aaw8DOcaY6/yzSR8CZgTwM8hVeDdzJ3d6llDU5i6ITnQdR0REqtnIXqlEeT3/HXXLPQQ757qOFRYCVtystWXA0/hK2BZgsrV2kzHmp8aYof6n/RZIAKYYY9YaY2b6X3sS+H/4yt9K4Kf+YwBPAv8EdgK7+O91cRIErLUcXf42tUwhtXSaVESkRqobH8WQzr6dFAqb3wLxybD6ddexwkJEIN/cWjsLmHXesR+ec//mS7z2X8AXpqJYazOBzlUYU6rQxoM59M37iLyEJiQ0u8F1HBERcWRcnzRmrjvEe5uyGdVtLCz7K+QehcSGrqOFtKCYnCDh46OlK+nn2UhEz/u1dpuISA3Wp0VdWibH+zae7/kQVJTBugmuY4U8/WaVKlNUWk7Upil4jCUmXWu3iYjUZMYYxvVOY/X+02wtawhpfX2zS611HS2kqbhJlZm7+Sh3VczndIM+UKe56zgiIuLYiJ4pREV4mHB2ksLJXbBvietYIU3FTarM2iUf0MJzlFrXa0N5ERGBOvFRDOnUiHfWHKSo7V0QXQvWvOE6VkhTcZMqceRMEa0PzaDEE4en0z2u44iISJAYk5FKTlEZH2zLgS6jYNN0KDztOlbIUnGTKjEzcwd3eJZT3O5uiIp3HUdERILEdS3r0axe3H8nKZQVwsaprmOFLBU3uWbWWk6snEqiKSSxj06TiojIf3k8hvvSU1m+5yS7I1tDoy7aAusaqLjJNVu9/zQD8j8mLy4FmvV1HUdERILMqF4peD2GyZlZ0PNhOLwODq11HSskqbjJNZuzdCU3eDcR2etB32bCIiIi52hQK4bB7RswdVUWpR1HQESMJilcJRU3uSaFJeXEb5lCBYboXuNcxxERkSA1JiOV43nFzN1bAh2HwfopUFroOlbIUXGTa/LhxkMMtfPJaXQ91E5zHUdERILUjW2TaVgrmkkr/ZMUis/A5pmuY4UcFTe5JhuWfkCaJ5ta12lSgoiIXFyE18OoXqks2J7NoaSeULelJilcBRU3uWpZpwpof2Qmxd54PB2Huo4jIiJBbnRGKhUWpqw6CD0ehH2L4fhO17FCioqbXLWZK3xrt5W2HwZRca7jiIhIkEutG0e/1vWZnHmA8q5jwXg1SeEKqbjJVamosJxaNYV4U0yC1m4TEZFKGp2RysHThXx6NALaDoG1b0F5qetYIUPFTa7Kir0nualoDnnxaZDax3UcEREJEbd2akiduEgmrTzgm6SQfwy2f+g6VshQcZOrMnfJCq7zbCE6XWu3iYhI5UVHeBneM4WPNh/hROP+kNhYp0uvgIqbXLG84jKStk+lAkNkT63dJiIiV2Z0Riql5ZZpa49C9/thx0eQc8h1rJCg4iZXbNb6gwxjAXlNboCkFNdxREQkxLRtmEjPtNpMXLkf2+MBsBWw9k3XsUKCiptcsS1LZpPqySZRa7eJiMhVGtM7jV3Z+azKSYIWA2D1G1BR4TpW0FNxkyuy93g+nY+/R7E3HtPhbtdxREQkRN3ZpTEJ0RFMWHHAt/H86X2wd6HrWEFPxU2uyMyV27jds4LyjsMhMtZ1HBERCVHx0RHc3a0J7284RE6L2yCmtnZSqAQVN6m08gpL7qqpxJli4no/5DqOiIiEuDEZqRSVVjBz40noNga2vAsFJ13HCmoqblJpS3Yd5+aSueQltICUDNdxREQkxHVNSaJD41pMXLnftwVWeQmsn+w6VlBTcZNKm79kGX08W4nO0NptIiJy7YwxjMlIZePBHDaWp0KTnrD6dbDWdbSgpeImlXKmsJT6u96mAg+RPca6jiMiImHinu5NiYrw/HcnhWOb4eBq17GCloqbVMp767IYZhaSl9IfajVxHUdERMJEUlwkd3RuxPS1Bylsdw9ExvlG3eSCVNykUrYtnUVTc4JEbSgvIiJVbEzvNHKLypi1PR86DYeNb0NxnutYQUnFTS5r57Fcepx8n+KIREz7O13HERGRMNOnRV2a14v77+nSkjzY9I7rWEFJxU0ua+byrQzxrKSi0wiIjHEdR0REwowxhtEZaazYe5JdMR2hfjttPH8RKm5ySWXlFRSsfZtYU0JshtZuExGRwBjRqykRHsPkzCzfqNuB5XBsq+tYQUfFTS5p4Y5shpTOJS+xFTTt6TqOiIiEqQaJMdzUoQFTV2VR0uk+8ERq1O0CVNzkkhYsWUq6ZzsxvR/S2m0iIhJQYzLSOJFfwtz95dD+Dlg3AcqKXccKKipuclGn8ktouOcdKvAQ0X2M6zgiIhLmBrRNpnFSDBPPTlIoOAHbZrmOFVRU3OSiZq7Zz72ehRSkDYTERq7jiIhImPN6DKPSU1m4I5usOn0gKVUbz59HxU0uasfy92lsTpKgtdtERKSajOqVAsCU1YehxwOwaz6c3u84VfBQcZML2nwoh4zTH1AcUQva3e46joiI1BCpdePo17o+UzIPUN5tnO/gmjfdhgoiKm5yQe8u38JtnpXYLiMhItp1HBERqUHGZKRx6EwRi47FQKvBsOY/UFHuOlZQUHGTLygpq6Bs/VRiTCkxWrtNRESq2S0dG1I3PoqJK/yTFHKyfKdMRcVNvmje1mPcXj6PvKS20Li76zgiIlLDREV4GNGzKXO2HCW76U0QV08bz/upuMkXfLr0U3p6dhKb8aDWbhMRESdGZ6RSVmGZtu4YdBvrWxYkL9t1LOdU3ORzsnOLabp/OhV48XYb7TqOiIjUUK0bJJLRvA6TVh7A9ngQKsp8C/LWcCpu8jkzVu/nHs8iCpsPhsSGruOIiEgNNjojjd3H81mRlwyp1/m2wLLWdSynVNzkM9Za9ix/l0bmFPG9NSlBRETcuqNLIxKjI5i08gD0fBCOb/dtPl+DqbjJZ9ZnneH63A8piqwNbYe4jiMiIjVcXFQEQ7s34f0NhznT8k6ISqzxOymouMln3lu+iVs9mZiu90FElOs4IiIijO2dRnFZBTM2n4EuI2DTO1B0xnUsZ1TcBICi0nLMxqlEmTKitXabiIgEic5Nk+jUpBYTVhzA9ngISgtg49uuYzmj4iYAfLjpCHdXzCOvTido1MV1HBERkc+MyUhly+EcNtpW0LBzjT5dquImAKxYuoAunr3EaUN5EREJMkO7NyUm0sPETP9OCofWwJENrmM5oeImHDxdSOuD0ykzkXi6jnIdR0RE5HOSYiO5o0tjZqw9REH74eCNhtVvuI7lhIqbMH3lboZ5P6Wk9e0QV9d1HBERkS8Yk5FGXnEZ7+8ogg53w/qJUFroOla1U3Gr4SoqLEdWTqeuydNpUhERCVoZzevQMjnev6bbQ76ZpVvecx2r2qm41XAr9p5kcOGHFMQ0gpaDXMcRERG5IGMMYzJSydx3ip3x3aFO8xq58byKWw334dLVDPCsJ7Ln/eDxuo4jIiJyUcN7phDhMUxceRB6PAh7F8GJXa5jVSsVtxosr7iMxG1T8RpLZK/7XccRERG5pPoJ0dzSsSHT1hykuMsYMB5Y8x/XsaqVilsN9v66g9zLfHIb9YZ6rVzHERERuazRGamczC9hzgEvtLkN1r4F5WWuY1UbFbcabP3SD2nhOUpCn/Guo4iIiFRK/zbJNK0dy8SV+30bz+cdgZ0fu45VbVTcaqhd2Xl0y36PEm8cptM9ruOIiIhUitdjGJWewqIdxzlQrx8kNKxROymouNVQM1Zs507vMso73ANR8a7jiIiIVNqo9FSMgSlrjkD3cbD9Q8g57DpWtVBxq4HKyivIWz2VeFNMbO/xruOIiIhckaa1YxnQJpnJmVmUd3sAbDmse8t1rGqh4lYDLdpxnCGlc8hLbAGpvV3HERERuWJje6dyJKeIBScSoXl/3xZYFRWuYwWcilsNtGDpEnp7thGT8RAY4zqOiIjIFRvcviH1E6KYuMK/k8KpPbBvsetYAafiVsOczC+h4e5pVOAlosc413FERESuSlSEhxE9U5i79RjHUm6B6KQasfF8QIubMWaIMWabMWanMeY7F3h8gDFmtTGmzBgz8pzjg4wxa8+5FRlj7vE/9poxZs85j3UP5GcINzNX7+Nez0Ly0wZBYiPXcURERK7afRmplFdY3l5/ErreB5tnQOEp17ECKmDFzRjjBf4C3A50BMYaYzqe97T9wHjgc1cUWmvnW2u7W2u7A4OBAuCjc57y3NnHrbVrA/MJwtPu5e/SyJwi8TptKC8iIqGtVXICvVvUZdLK/dieD0J5Mayf4jpWQAVyxK03sNNau9taWwJMBIad+wRr7V5r7XrgUlcTjgRmW2sLAhe1Zth48Ax9zsymKLIOtB3iOo6IiMg1G5ORyt4TBSwrSIHG3X0bz1vrOlbABLK4NQUOnPN1lv/YlRoDTDjv2M+NMeuNMc8bY6KvNmBNM2v5Rm7xrIJuoyEiynUcERGRa3Z758YkxkQwaeV+3ySFoxvh0BrXsQImqCcnGGMaA12AD885/F2gPZAB1AW+fZHXPmGMyTTGZGZnZwc8a7ArLiuHDZOJMuW+2aQiIiJhIDbKy709mjJr4xFOtx4GEbFhvZNCIIvbQSD1nK9T/MeuxH3AO9ba0rMHrLWHrU8x8Cq+U7JfYK39u7U23VqbnpycfIV/bfiZu/kod5fPI6duF2jYyXUcERGRKjM6I5WSsgqmb86FTvfAxrehJN91rIAIZHFbCbQxxrQwxkThO+U58wrfYyznnSb1j8JhjDHAPcDGa48a/pYvmU8Hz34SrhvvOoqIiEiV6tQkiS5Nk5i48gC2x4NQnOObYRqGAlbcrLVlwNP4TnNuASZbazcZY35qjBkKYIzJMMZkAaOAl40xm86+3hjTHN+I3YLz3vpNY8wGYANQH/hZoD5DuDhypohWB9+hzETh6TLCdRwREZEqNzojla1Hclnv6Qj1Woft6dKIQL65tXYWMOu8Yz885/5KfKdQL/TavVxgMoO1dnDVpgx/MzJ3MdqzhKLWd5AQW8d1HBERkSo3rHsTfv7+FiZmHqBbz4fg4x9C9nZIbus6WpUK6skJcu2stRxdMY3aJl+nSUVEJGwlxkRyZ9fGzFx7iPwOo8ATAWvCb9RNxS3Mrdp3ioEFH5If2xha3Og6joiISMCMyUglv6Sc93eVQ7vbYe0EKCtxHatKqbiFuY+WrKKfZyORPR8Aj77dIiISvno1q0PrBglMXLkfej4MBcdh+2zXsaqUfpOHsYKSMhK2TsZjLFHpD7iOIyIiElDGGMZkpLJ6/2m2J2RAraZht/G8ilsYm7X+EMP4hJxG10Od5q7jiIiIBNy9PZoS6TVMzDwE3e+HnXPgTJbrWFVGxS2MbVwym2aeYyReP951FBERkWpRLyGaWzs2YtqaLIq7jvMdXPOm21BVSMUtTO07kU+X7Hcp9sZjOgx1HUdERKTajOmdyumCUj48GA0tB8KaN6Ci3HWsKqHiFqZmLt/K7Z4VlHUcDlFxruOIiIhUmxta1adp7dj/bjx/5gDs/sR1rCqh4haGyisseaunEGeKie8z3nUcERGRauXxGEZnpPLpzhPsbzAIYuuGzU4KKm5haMmu49xaMofcxFbQtJfrOCIiItVuVHoKHgOT1xyDbmNg6/uQf9x1rGum4haGFnz6Kb08O4jp/RAY4zqOiIhItWucFMvAdg2YsuoAZd3uh4pSWD/JdaxrpuIWZs4UlNJg91TK8RLZY5zrOCIiIs6MzkjlaE4xn5xKhpQM3+lSa13HuiYqbmHm3bX7uccsIr/ZYEho4DqOiIiIM4PbN6B+QjQTVx7wTVLI3gpZK13HuiYqbmFm97IZNDCnSdSG8iIiUsNFej2M7JXC/G3HOJZ2J0QlwOrXXce6JipuYWTbkVx6n5pFYVRdTNvbXMcRERFxbnRGKuUVlikbTkHn4bBxGhTluI511VTcwsj7S9dxk2c1dB0N3kjXcURERJxrUT+e61rWZdLKA1R0fxBKC2DTNNexrpqKW5goLa/Arp9MpCkntvfDruOIiIgEjTEZaew/WcCy4haQ3CGkN55XcQsT87Yc5a7yuZyp2w0adHAdR0REJGgM6dyIpNhIJmRm+SYpHMyEo5tcx7oqKm5hYuWSubTzZJFwnUbbREREzhUT6eXeHk35cOMRTre+F7xRITvqpuIWBo7lFtH8wDuUmmi8XUe6jiMiIhJ0RmekUlJewbRtRdD+Llg/EUqLXMe6YipuYeDdzF0M9SyhsM2dEJPkOo6IiEjQ6dC4Ft1Skpi08gC250NQeAq2vuc61hVTcQtx1lqOrnibWqaAWlq7TURE5KLG9E5j29Fc1kR0hdppIbnxvIpbiFuXdYb+eR+SF9sUmvd3HUdERCRo3d2tCXFRXiatPAg9HoQ9C+DkHtexroiKW4j7aMkKbvBsIrLXA+DRt1NERORiEqIjuKtrY95df4j8jqPBeGDtm65jXRH9pg9hRaXlxG+ZDAai0x9wHUdERCTojc5Io6CknHf3GGh9M6x5E8rLXMeqNBW3EPbhxkMMrfiEnEZ9fefqRURE5JJ6ptWmbcMEJpzdeD73EOya6zpWpam4hbCNn75PqiebWn0fcR1FREQkJBhjGJ2RxroDp9mS2Bfik0NqkoKKW4jKOlVAp6MzKfYm4Olwl+s4IiIiIePeHk2J8nqYtPoIdB8H22ZD7lHXsSpFxS1Evbt8G0M8KyjtOAIiY13HERERCRl146O4rXMj3llzkOIu48CWw7q3XMeqFBW3EFRRYclbNZEYU6otrkRERK7CmIxUzhSW8sGRREjr69sCy1rXsS5LxS0ELdtzgluKP+ZMYhto0tN1HBERkZBzfct6pNaNZeIK/ySFk7tg3xLXsS5LxS0ELfp0Ed09u4jr8zAY4zqOiIhIyPF4DKPTU1m6+wT7Gt4M0bVCYpKCiluIyS0qJXnnFMrxEtljrOs4IiIiIWtUeioeAxPXnYAuo2DzdCg87TrWJam4hZj31+7nbrOI3GY3Q3x913FERERCVsNaMQxu34Cpq7Io7f4glBXBhimuY12SiluI2bP0HZJNDklau01EROSajc5IIzu3mPlnGkOjLkF/ulTFLYTsPJZH+slZFETVx7S+xXUcERGRkDeoXTINEqOZtPIA9HwYjqyHQ2tdx7ooFbcQMmvpWgZ51mC7jQFvhOs4IiIiIS/C62FUegrztx3jSLO7ICIG1rzhOtZFqbiFiLLyCuy6SUSYCuJ7a+02ERGRqnJfeioVFqZszIOOw2D9FCgpcB3rglTcQsTC7ce4o2wup+r1gOS2ruOIiIiEjWb14unbqh6TMg9Q0f1BKD4DW2a6jnVBKm4hYsXiObTxHCTx+vGuo4iIiISdMb3TyDpVyKdl7aBuy6CdpKDiFgJO5pfQfP80SjwxRHQe7jqOiIhI2Lm1Y0Nqx0UyMTPLt5PCvk/h+E7Xsb5AxS0EvJu5kzs9SyhsfRfE1HIdR0REJOzERHq5t0dTPtp0hFNtRoLxwprgG3VTcQty1lqOLp9CoinU2m0iIiIBNCYjjdJyy9vbS6HtEFg7AcpLXcf6HBW3ILfpUA79cj8gJzYVmt3gOo6IiEjYatcokR5ptZm48gC254OQfwy2f+g61ueouAW5OZ8uo693M1G9HtCG8iIiIgE2JiOVncfyWB3VCxIbB90kBRW3IFZUWk7M5slUYIjJeMB1HBERkbB3V9cmxEd5mZh5GLrfDzs/hjMHXcf6jIpbEJuz+RB32/mcbtwPklJcxxEREQl78dER3N2tCe+tP0xepzFgK2DtW65jfUbFLYhtXvweTc0JamtSgoiISLUZ0zuNwtJyZuyLghYDfLNLKypcxwJU3ILW4TOFtD8yk8KIWnja3+k6joiISI3RLSWJ9o0S/7vx/On9sGeB61iAilvQen/5Fm7zrKS04wiIjHEdR0REpMYwxjA6I5X1WWfYnNQfYmoHzcbzKm5ByFpLbuYEok0pta4b7zqOiIhIjXNvj6ZERXiYuCYbuo2BLe9CwUnXsVTcglHmvlMMLprD6VrtoHE313FERERqnNpxUdzeuRHvrDlIcZf7obwE1k9yHUvFLRgtWvQJ3Ty7ievzsNZuExERcWR0Riq5RWXMyq4LTXv51nSz1mkmFbcgk19cRr2dUygzEUR1H+M6joiISI11XYt6NKsXx8QVB6DHg3BsMxxc5TSTiluQmb1uP3exiJy0WyC+nus4IiIiNZbH45uksHzPSfY0HgK3/xbqtXabyenfLl+wd8nb1DO51LlBa7eJiIi4NrJnCl6PYeL609DnCYit7TSPilsQ2Xs8nx4n3icvKhnT6ibXcURERGq8BrViGNy+AW+vyqK03P0ivCpuQWT20jUM9KyFbmPBG+E6joiIiODbeP54XglztxxzHUXFLViUV1gq1k7AaywJfR52HUdERET8bmybTKNaMUxcud91FBW3YLF4RzZDSudysl4vqO/2wkcRERH5rwivh1HpKSzYns2h04VOs6i4BYnMxR/QynOYxOvHu44iIiIi57kvPZWBbZPJLy5zmkMXUgWB0wUlpO2bRklELFFdhruOIyIiIudJrRvHq4/0dh1DI27BYNaqndxulpLf+m6ITnAdR0RERIJUQIubMWaIMWabMWanMeY7F3h8gDFmtTGmzBgz8rzHyo0xa/23meccb2GMWe5/z0nGmKhAfobqcHTZZBJMEXVueNR1FBEREQliAStuxhgv8BfgdqAjMNYY0/G8p+0HxgNvXeAtCq213f23oecc/zXwvLW2NXAKeKzKw1ejLYdz6Js7mzNxaZB2nes4IiIiEsQCOeLWG9hprd1trS0BJgLDzn2CtXavtXY9UKkV7YwxBhgMTPUfeh24p8oSOzBn8VL6eLYS2etBbSgvIiIilxTI4tYUOHDO11n+Y5UVY4zJNMYsM8bc4z9WDzhtrT07peNK3zOolJRVELt5EhV4iMt4wHUcERERCXLBPKu0mbX2oDGmJTDPGLMBOFPZFxtjngCeAEhLSwtQxGszb8th7qyYz6km/alXq4nrOCIiIhLkAjnidhBIPefrFP+xSrHWHvT/uRv4BOgBnABqG2POFs6Lvqe19u/W2nRrbXpycvKVp68GmxfPoLE5Se2+2lBeRERELi+QxW0l0MY/CzQKGAPMvMxrADDG1DHGRPvv1wduADZbay0wHzg7A/VhYEaVJ68Gx3KLaHdoBgURSXg73OE6joiIiISAgBU3/3VoTwMfAluAydbaTcaYnxpjhgIYYzKMMVnAKOBlY8wm/8s7AJnGmHX4itqvrLWb/Y99G3jWGLMT3zVvrwTqMwTSrOWbudmTSUnHkRAR7TqOiIiIhICAXuNmrZ0FzDrv2A/Pub8S3+nO81+3BOhykffcjW/Gasiy1pKbOYFoU0a0TpOKiIhIJWnnBAfWHDjNoIKPOFmrAzS6YD8VERER+QIVNwcWL5pPZ89e4vo87DqKiIiIhBAVt2pWWFJO3e2TKTORxPQY7TqOiIiIhBAVt2r20fp93MkiTqfdBnF1XccRERGREKLiVs32LplKHZNH3Rs0KUFERESujIpbNTpwsoCu2e+RG9UQT+tBruOIiIhIiFFxq0YfLFnFAM96bPex4PG6jiMiIiIhRsWtmlRUWCrWTsBrLLX6POQ6joiIiIQgFbdqsmzXcW4tmcPxehlQr5XrOCIiIhKCVNyqSeaiWbTwHKVW3/Guo4iIiEiIUnGrBjlFpTTd+zbFnliiutzrOo6IiIiEKBW3avDBqp0MMcvIaz0MouJdxxEREZEQFdBN5sXn2LKJxJti4vpp7TYRERG5ehpxC7Cdx3Lpc2YWp+OaY1L7uI4jIiIiIUzFLcDmLlpChmc7Eb0eBGNcxxEREZEQpuIWQKXlFURtnEA5HhJ6P+A6joiIiIQ4FbcAWrjlMHdUzOdk4xshsZHrOCIiIhLiVNwCaPPi6TQ0p6nT71HXUURERCQMqLgFyIm8Ytocmk5+RG0i2g1xHUdERETCgIpbgMxavpHBZhUlHUdBRJTrOCIiIhIGtI5bAFhrycucQJQpJ+oGnSYVERGRqqERtwDYmHWGgfkfcLxWJ2jY0XUcERERCRMqbgGweNEcOngOEH/dw66jiIiISBhRcatiRaXl1N4+mRITRWyP0a7jiIiISBhRcaticzfs5w67iNNpt0FsbddxREREJIyouFWxvZ9OIckUUK//Y66jiIiISJhRcatCh04X0iV7JjlRjfC2vNF1HBEREQkzKm5V6KMlmfQzG7HdxoFH/2lFRESkaqldVBFrLWVr3sJjLEl9NZtUREREqp6KWxVZsfs4txTP4Vj9PlCnues4IiIiEoZU3KrI6kXv08xzjKS+411HERERkTCl4lYF8orLaLznbYo88UR3vsd1HBEREQlTKm5V4KNVO7iNZeS2HgpRca7jiIiISJjSJvNVIGn3u8SaEmK0dpuIiIgEkIpbFbgpzYMt6IFJSXcdRURERMKYTpVWhQHfxDw+D4xxnURERETCmIpbVdGCuyIiIhJgahsiIiIiIULFTURERCREqLiJiIiIhAgVNxEREZEQoeImIiIiEiJU3ERERERChIqbiIiISIhQcRMREREJESpuIiIiIiFCxU1EREQkRKi4iYiIiIQIFTcRERGREKHiJiIiIhIiVNxEREREQoSKm4iIiEiIUHETERERCREqbiIiIiIhQsVNREREJESouImIiIiECBU3ERERkRCh4iYiIiISIlTcREREREKEsda6zhBwxphsYF+A/5r6wPEA/x1y5fR9CT76ngQnfV+Cj74nwae6vifNrLXJF3qgRhS36mCMybTWprvOIZ+n70vw0fckOOn7Enz0PQk+wfA90alSERERkRCh4iYiIiISIlTcqs7fXQeQC9L3JfjoexKc9H0JPvqeBB/n3xNd4yYiIiISIjTiJiIiIhIiVNyqkDHmx8aYg8aYtf7bHa4z1VTGmCHGmG3GmJ3GmO+4ziM+xpi9xpgN/p+PTNd5aiJjzL+MMceMMRvPOVbXGPOxMWaH/886LjPWRBf5vuh3ikPGmFRjzHxjzGZjzCZjzNf9x53+vKi4Vb3nrbXd/bdZrsPURMYYL/AX4HagIzDWGNPRbSo5xyD/z4eWOXDjNWDIece+A8y11rYB5vq/lur1Gl/8voB+p7hUBvyvtbYjcB3wlP93idOfFxU3CUe9gZ3W2t3W2hJgIjDMcSaRoGCtXQicPO/wMOB1//3XgXuqM5Nc9PsiDllrD1trV/vv5wJbgKY4/nlRcat6Txtj1vuHvXW6wY2mwIFzvs7yHxP3LPCRMWaVMeYJ12HkMw2ttYf9948ADV2Gkc/R75QgYIxpDvQAluP450XF7QoZY+YYYzZe4DYMeAloBXQHDgO/d5lVJAj1s9b2xHca+yljzADXgeTzrG+pAS03EBz0OyUIGGMSgLeB/7HW5pz7mIufl4jq/MvCgbX25so8zxjzD+C9AMeRCzsIpJ7zdYr/mDhmrT3o//OYMeYdfKe1F7pNJcBRY0xja+1hY0xj4JjrQALW2qNn7+t3ihvGmEh8pe1Na+00/2GnPy8acatC/m/gWfcCGy/2XAmolUAbY0wLY0wUMAaY6ThTjWeMiTfGJJ69D9yKfkaCxUzgYf/9h4EZDrOIn36nuGWMMcArwBZr7R/Oecjpz4sW4K1Cxpg38A1pW2Av8OVzzoNLNfJPm38B8AL/stb+3G0iMca0BN7xfxkBvKXvS/UzxkwABgL1gaPAj4DpwGQgDdgH3Get1YXy1egi35eB6HeKM8aYfsAiYANQ4T/8PXzXuTn7eVFxExEREQkROlUqIiIiEiJU3ERERERChIqbiIiISIhQcRMREREJESpuIiIiIiFCxU1EwoIxptwYs/acW1BslH5OriaXeM6PjDG/PO9Yd2PMFv/9+caYPGNMeqDzikhw03IgIhIWjDF51tqEKn7PCGtt2TW+x2VzGWPaAh9Ya1uec+xXQIG19qf+rz8BvmmtzbyWPCIS2jTiJiJhzRiz1xjzE2PMamPMBmNMe//xeP/G3SuMMWv8+w1jjBlvjJlpjJkHzDXGxBljJhtjNhtj3jHGLDfGpBtjHjXGvHDO3/MlY8zzlchzqzFmqT/PFGNMgrV2O3DKGNPnnKfeB0yo0v8YIhLyVNxEJFzEnneqdPQ5jx33b27/EvBN/7HvA/Ostb2BQcBv/VtxAfQERlprbwSeBE5ZazsC/wf08j9nMnC3fy9DgEeAf10qoDGmPvAD4GZ/nkzgWf/DE/Btz4Yx5jrgpLV2x5X/ZxCRcKZN5kUkXBRaa7tf5LGzm0OvAob7798KDDXGnC1yMfi2sAH4+JwtbPoBfwSw1m40xqz338/zj8rd5b8WLdJau+EyGa8DOgKf+rZBJApY6n9sErDEGPO/+AqcRttE5AtU3ESkJij2/1nOf//dM8AIa+22c5/oP12ZX8n3/Se+vQu3Aq9W4vkGXykce/4D1toDxpg9wI3ACOD6SmYQkRpEp0pFpKb6EHjG+Ie+jDE9LvK8T/Fdb4YxpiPQ5ewD1trlQCowjsqNkC0DbjDGtPa/X7x/YsJZE4Dngd3W2qwr+zgiUhOouIlIuDj/GrdfXeb5/w+IBNYbYzb5v76QvwLJxpjNwM+ATcCZcx6fDHxqrT11uYDW2mxgPDDBf8p1KdD+nKdMATqh06QichFaDkRE5BKMMV58168VGWNaAXOAdtbaEv/j7wHPW2vnXuT1VbJMiZYDERHQiJuIyOXEAYuNMeuAd4AnrbUlxpjaxpjt+CZFXLC0+eVcbgHeyzHGzAdaAqVX+x4iEh404iYiIiISIjTiJiIiIhIiVNxEREREQoSKm4iIiEiIUHETERERCREqbiIiIiIhQsVNREREJET8f+BFwZj3R6FpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ldos_calculator = data_handler.target_calculator\n",
    "ldos_calculator.read_additional_calculation_data(data_handler.get_snapshot_calculation_output(0))\n",
    "\n",
    "\n",
    "fig = plt.figure(figsize=(10,10))\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "ldos_calculator.read_from_array(actual_ldos)\n",
    "ax.plot(ldos_calculator.energy_grid, ldos_calculator.density_of_states.copy(), label=\"Actual\")\n",
    "ldos_calculator.read_from_array(predicted_ldos)\n",
    "ax.plot(ldos_calculator.energy_grid, ldos_calculator.density_of_states.copy(), label=\"Predicted\")\n",
    "\n",
    "ax.set_xlabel(\"Energy [eV]\")\n",
    "ax.set_ylabel(\"Density of States [1/eV]\")\n",
    "ax.legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This finally leads us to the prediction part - the part, to which such efforts eventually should culminate too. Since this part again requires LAMMPS, it will not be hands-on.\n",
    "\n",
    "Instead I will demonstrate how MALA can be used to predict the electronic density and electronic structure of a system far larger the atomic configurations it was trained on. As we have demonstrated in a recently submitted research article, this is feasible if the training data is big enough to capture characteristic electronic effects. This is technically not the case with this reduced data set, but it is the case for production-level models and the syntax is the same regardless.\n",
    "\n",
    "Before we predict models it should be noted that for production settings, it is very useful to save some simulation output alongside the model, so the predictor knows about, e.g., grid coarseness and such.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "trainer.save_run(\"Be_model\", additional_calculation_data=pj(data_path, \"Be_snapshot2.out\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor calculation: had to enforce periodic boundary conditions on 0 atoms before calculation.\n",
      "Had to readjust batch size from 40 to 48\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "could not broadcast input array from shape (48,11) into shape (48,0)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Input \u001b[0;32mIn [32]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m parameters, network, data_handler, predictor \u001b[38;5;241m=\u001b[39m mala\u001b[38;5;241m.\u001b[39mPredictor\u001b[38;5;241m.\u001b[39mload_run(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mBe_model\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      2\u001b[0m Be16 \u001b[38;5;241m=\u001b[39m read(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mBe16.vasp\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 3\u001b[0m predicted_ldos \u001b[38;5;241m=\u001b[39m \u001b[43mpredictor\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpredict_for_atoms\u001b[49m\u001b[43m(\u001b[49m\u001b[43mBe16\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/codes/mala/mala/network/predictor.py:185\u001b[0m, in \u001b[0;36mPredictor.predict_for_atoms\u001b[0;34m(self, atoms, gather_ldos, temperature)\u001b[0m\n\u001b[1;32m    182\u001b[0m snap_descriptors \u001b[38;5;241m=\u001b[39m \\\n\u001b[1;32m    183\u001b[0m     torch\u001b[38;5;241m.\u001b[39mfrom_numpy(snap_descriptors)\u001b[38;5;241m.\u001b[39mfloat()\n\u001b[1;32m    184\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdata\u001b[38;5;241m.\u001b[39minput_data_scaler\u001b[38;5;241m.\u001b[39mtransform(snap_descriptors)\n\u001b[0;32m--> 185\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_forward_snap_descriptors\u001b[49m\u001b[43m(\u001b[49m\u001b[43msnap_descriptors\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/codes/mala/mala/network/predictor.py:216\u001b[0m, in \u001b[0;36mPredictor._forward_snap_descriptors\u001b[0;34m(self, snap_descriptors, local_data_size)\u001b[0m\n\u001b[1;32m    213\u001b[0m     inputs \u001b[38;5;241m=\u001b[39m snap_descriptors[i \u001b[38;5;241m*\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparameters\u001b[38;5;241m.\u001b[39mmini_batch_size:\n\u001b[1;32m    214\u001b[0m                               (i\u001b[38;5;241m+\u001b[39m\u001b[38;5;241m1\u001b[39m)\u001b[38;5;241m*\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparameters\u001b[38;5;241m.\u001b[39mmini_batch_size]\n\u001b[1;32m    215\u001b[0m     inputs \u001b[38;5;241m=\u001b[39m inputs\u001b[38;5;241m.\u001b[39mto(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparameters\u001b[38;5;241m.\u001b[39m_configuration[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdevice\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[0;32m--> 216\u001b[0m     predicted_outputs[i \u001b[38;5;241m*\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparameters\u001b[38;5;241m.\u001b[39mmini_batch_size:\n\u001b[1;32m    217\u001b[0m                               (i\u001b[38;5;241m+\u001b[39m\u001b[38;5;241m1\u001b[39m)\u001b[38;5;241m*\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparameters\u001b[38;5;241m.\u001b[39mmini_batch_size] \\\n\u001b[1;32m    218\u001b[0m         \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdata\u001b[38;5;241m.\u001b[39moutput_data_scaler\u001b[38;5;241m.\u001b[39m\\\n\u001b[1;32m    219\u001b[0m         inverse_transform(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mnetwork(inputs)\u001b[38;5;241m.\u001b[39m\n\u001b[1;32m    220\u001b[0m                           to(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcpu\u001b[39m\u001b[38;5;124m'\u001b[39m), as_numpy\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[1;32m    222\u001b[0m \u001b[38;5;66;03m# Restricting the actual quantities to physical meaningful values,\u001b[39;00m\n\u001b[1;32m    223\u001b[0m \u001b[38;5;66;03m# i.e. restricting the (L)DOS to positive values.\u001b[39;00m\n\u001b[1;32m    224\u001b[0m predicted_outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdata\u001b[38;5;241m.\u001b[39mtarget_calculator\u001b[38;5;241m.\u001b[39m\\\n\u001b[1;32m    225\u001b[0m     restrict_data(predicted_outputs)\n",
      "\u001b[0;31mValueError\u001b[0m: could not broadcast input array from shape (48,11) into shape (48,0)"
     ]
    }
   ],
   "source": [
    "parameters, network, data_handler, predictor = mala.Predictor.load_run(\"Be_model\")\n",
    "Be16 = read(\"Be16.vasp\")\n",
    "predicted_ldos = predictor.predict_for_atoms(Be16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now that we have the LDOS we can calculate the total energy and electronic density. The latter can be visualized using e.g. VESTA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# For the total energy calculation we need to tell MALA where to find a pseudopotential.\n",
    "\n",
    "parameters.targets.pseudopotential_path = \"./data_generation/\"\n",
    "ldos_calculator = predictor.target_calculator\n",
    "ldos_calculator.read_from_array(predicted_ldos)\n",
    "print(ldos_calculator.band_energy)\n",
    "density_calculator = mala.Density.from_ldos_calculator(ldos_calculator)\n",
    "density_calculator.write_to_cube(\"Be16_density.cube\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "That all seems nice and well, but a bit complicated in practice. Some or many of you may have worked with ASE calculators in the past. ASE is an incredibly useful library that provides direct interfaces to a lot of electronic structure codes. By providing a unified interface to access properties of interest (such as the total energy) it is easy to investigate the electronic structure of compounds.\n",
    "\n",
    "Thus, MALA also provides an interface for this.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor calculation: had to enforce periodic boundary conditions on 0 atoms before calculation.\n",
      "MALA: QuantumEspresso is already running. Except for the atomic positions, no new parameters will be used.\n",
      "time used by set_positions:  0.048365369000748615\n",
      "time used by set_rho_of_r:  0.03205540999988443\n",
      "-582.515180723934\n"
     ]
    }
   ],
   "source": [
    "calculator = mala.MALA.load_model(\"Be_model\", pj(data_path, \"Be_snapshot2.out\"))\n",
    "calculator.mala_parameters.targets.pseudopotential_path = \"./data_generation/\"\n",
    "Be16.set_calculator(calculator)\n",
    "print(Be16.get_potential_energy())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Such calculators can now in principle be used like any electronic structure tool.\n",
    "Currently, MALA cannot evaluate ionic forces. This is a topic of ongoing implementation, planned to be finished this summer. Thermodynamic sampling is possible via Monte-Carlo sampling, the limitations of which are currently researched.\n",
    "For a set of ionic positions (obtained e.g. via a standard ML-IAP) MALA can be used to predict DOS, electronic density, LDOS, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Making MALA performant (presentation only)\n",
    "\n",
    "Now that we know how to build and use MALA models the important question of scalability and performance arises.\n",
    "\n",
    "There are two important things to test out here. One is optimizing performance of fitting models.\n",
    "For this, the common practice (not just for MALA, but for ML in general) is using GPUs. I will shortly demonstrate how much this can help. Let us consider a model with 1000 neurons in one single layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Consistency check successful.\n",
      "Data scalers initialized.\n",
      "Build dataset: Done.\n",
      "Final validation data loss:  5.188130906649998e-06\n",
      "Training time: 80.2793219089508\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 0\n",
    "parameters.manual_seed = 2023\n",
    "parameters.running.max_number_epochs = 5\n",
    "\n",
    "data_handler = mala.DataHandler(parameters)\n",
    "data_handler.clear_data()\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                          \"Be_snapshot2.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot3.in.npy\", data_path,\n",
    "                          \"Be_snapshot3.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "parameters.network.layer_sizes = [data_handler.input_dimension,\n",
    "                                  1000, 1000,\n",
    "                                  data_handler.output_dimension]\n",
    "\n",
    "start_time = time()\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "trainer.train_network()\n",
    "print(\"Training time:\", time()-start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "And now let's do the same with GPU enabled!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Consistency check successful.\n",
      "Data scalers initialized.\n",
      "Build dataset: Done.\n",
      "Final validation data loss:  5.18813037446567e-06\n",
      "Training time: 16.596766710281372\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 0\n",
    "parameters.manual_seed = 2023\n",
    "parameters.running.max_number_epochs = 5\n",
    "parameters.use_gpu = True\n",
    "\n",
    "data_handler = mala.DataHandler(parameters)\n",
    "data_handler.clear_data()\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                          \"Be_snapshot2.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot3.in.npy\", data_path,\n",
    "                          \"Be_snapshot3.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "parameters.network.layer_sizes = [data_handler.input_dimension,\n",
    "                                  1000, 1000,\n",
    "                                  data_handler.output_dimension]\n",
    "\n",
    "start_time = time()\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "trainer.train_network()\n",
    "print(\"Training time:\", time()-start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "That's already a fair bit of speed-up! The speed-up may not seem all to substantial here, which is due to the fact that we are using small amounts of data here. For larger data sets, the speed-up will be much bigger.\n",
    "\n",
    "Advanced GPU utilization (beyond what pytorch supports by default, kindly implemented for us by experts from Nvidia) can give an even better speed-up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Consistency check successful.\n",
      "Data scalers initialized.\n",
      "Build dataset: Done.\n",
      "Final validation data loss:  5.223024828312599e-06\n",
      "Training time: 3.5216217041015625\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 0\n",
    "parameters.manual_seed = 2023\n",
    "parameters.running.max_number_epochs = 5\n",
    "parameters.use_gpu = True\n",
    "parameters.running.use_graphs = True\n",
    "parameters.data.use_fast_tensor_data_set = True\n",
    "\n",
    "data_handler = mala.DataHandler(parameters)\n",
    "data_handler.clear_data()\n",
    "\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                          \"Be_snapshot2.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot3.in.npy\", data_path,\n",
    "                          \"Be_snapshot3.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "parameters.network.layer_sizes = [data_handler.input_dimension,\n",
    "                                  1000, 1000,\n",
    "                                  data_handler.output_dimension]\n",
    "\n",
    "start_time = time()\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "trainer.train_network()\n",
    "print(\"Training time:\", time()-start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Secondly, we'd like to optimize inference. Ideally, we would like to perform the entire inference on GPU. For MALA, this includes the calculation of the bispectrum descriptors as well as the LDOS integration / DFT energy evaluation. This is currently in development, so for now acceleration of inference is done via CPU parallelization. This is realized via MPI and since this is not easily shown in a jupyter notebook, this repository contains the `test_inference_mpi.py` script to this end, at which we will have a look now.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "A final concept we may need is lazy loading. Lazy loading means that not all of the data can be loaded into RAM at the same time. This may happen sooner than you'd think or like, since data sets can be quite extensive. Of course for our small training examples this technically would not be a problem, but it can very well happen for huge data sets, so let us see what it entails.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Consistency check successful.\n",
      "Data scalers initialized.\n",
      "Build dataset: Done.\n",
      "Final validation data loss:  2.9018478734152658e-06\n",
      "Training time: 120.87365913391113\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 0\n",
    "parameters.manual_seed = 2023\n",
    "parameters.network.layer_sizes = [100]\n",
    "parameters.running.max_number_epochs = 100\n",
    "parameters.input_rescaling_type = \"feature-wise-standard\"\n",
    "parameters.output_rescaling_type = \"normal\"\n",
    "parameters.data.use_lazy_loading = True\n",
    "\n",
    "data_handler = mala.DataHandler(parameters)\n",
    "data_handler.clear_data()\n",
    "\n",
    "data_handler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                          \"Be_snapshot0.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                          \"Be_snapshot2.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot3.in.npy\", data_path,\n",
    "                          \"Be_snapshot3.out.npy\", data_path, \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "parameters.network.layer_sizes.insert(0, data_handler.input_dimension)\n",
    "parameters.network.layer_sizes.append(data_handler.output_dimension)\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "\n",
    "start_time = time()\n",
    "trainer.train_network()\n",
    "print(\"Training time:\", time()-start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Of course, that can lead to problems with data shuffling. Ideally, within ML, we'd like to randomize the order in which data points are processed. In all of the models we have trained so far this had been the case, since they had all been loaded into RAM and randomized there. It had been done under the hood. But if we never load all of the data at the same time, how do we ensure random access?\n",
    "\n",
    "The best solution for this problem is to shuffle your data before you load it. MALA provides a distinct class for this, but you may handle this however you'd like.\n",
    "\n",
    "Just remember: Always randomize your data!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Disabling XYZ-cutting from descriptor data for shuffling. If needed, please re-enable afterwards.\n",
      "Data shuffler will generate 3 new snapshots.\n",
      "Shuffled snapshot dimension will be  (18, 18, 27)\n"
     ]
    }
   ],
   "source": [
    "parameters.data.shuffling_seed = 1234\n",
    "\n",
    "data_shuffler = mala.DataShuffler(parameters)\n",
    "data_shuffler.clear_data()\n",
    "data_shuffler.add_snapshot(\"Be_snapshot0.in.npy\", data_path,\n",
    "                           \"Be_snapshot0.out.npy\", data_path)\n",
    "data_shuffler.add_snapshot(\"Be_snapshot2.in.npy\", data_path,\n",
    "                           \"Be_snapshot2.out.npy\", data_path)\n",
    "data_shuffler.add_snapshot(\"Be_snapshot3.in.npy\", data_path,\n",
    "                           \"Be_snapshot3.out.npy\", data_path)\n",
    "data_shuffler.shuffle_snapshots(complete_save_path=\"./\",\n",
    "                                save_name=\"Be_shuffled*\")\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Consistency check successful.\n",
      "Data scalers initialized.\n",
      "Build dataset: Done.\n"
     ]
    }
   ],
   "source": [
    "parameters = mala.Parameters()\n",
    "parameters.verbosity = 0\n",
    "parameters.manual_seed = 2023\n",
    "parameters.network.layer_sizes = [100]\n",
    "parameters.running.max_number_epochs = 100\n",
    "parameters.input_rescaling_type = \"feature-wise-standard\"\n",
    "parameters.output_rescaling_type = \"normal\"\n",
    "parameters.data.use_lazy_loading = True\n",
    "\n",
    "data_handler = mala.DataHandler(parameters)\n",
    "data_handler.clear_data()\n",
    "\n",
    "data_handler.add_snapshot(\"Be_shuffled0.in.npy\", \"./\",\n",
    "                          \"Be_shuffled0.out.npy\", \"./\", \"tr\")\n",
    "data_handler.add_snapshot(\"Be_shuffled1.in.npy\", \"./\",\n",
    "                          \"Be_shuffled1.out.npy\", \"./\", \"tr\")\n",
    "data_handler.add_snapshot(\"Be_shuffled2.in.npy\", \"./\",\n",
    "                          \"Be_shuffled2.out.npy\", \"./\", \"tr\")\n",
    "data_handler.add_snapshot(\"Be_snapshot1.in.npy\", data_path,\n",
    "                          \"Be_snapshot1.out.npy\", data_path, \"va\")\n",
    "\n",
    "# This already loads data into RAM!\n",
    "data_handler.prepare_data()\n",
    "parameters.network.layer_sizes.insert(0, data_handler.input_dimension)\n",
    "parameters.network.layer_sizes.append(data_handler.output_dimension)\n",
    "network = mala.Network(parameters)\n",
    "trainer = mala.Trainer(parameters, network, data_handler)\n",
    "\n",
    "start_time = time()\n",
    "trainer.train_network()\n",
    "print(\"Training time:\", time()-start_time)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}