======================= Introduction to hfcnn

A convolutional neural network designed to extract plasma parameters from thermal camera heat load images.

Acronym: Heat flux cnn

Features

• The Heat Flux NN package is built using the Hydra framework and uses PyTorchLightning.

Setup

The environment used to run the code can be reproduced with the environment.yaml file in the root directory. This will create a conda environment called PyTorch. To install this environment simply install anaconda and run the following command:

    $ conda env create -f environment.yaml

In the future this will be better handled but this is a placeholder for now.

Data:

If you run the package the main will instantiate the action defined in the config/config.yaml folder.

By default, data is stored in the following directory structure:

data/
    processed/
        train.pkl       ########################
        test.pkl        # generated by package #
        validation.pkl  ########################
    raw/
        51/ <- port number of the data
            100343888387473.pkl <- timestamp of data
        data_table.pkl <- name defined in configs/config.yaml

The processed data is generated when running the preprocessing actions.

DVC & Datashare:

Datashare is a common tool used at the institute and is compatible with DVC as a remote share.

• On Datashare, create a folder to store the data and share the folder. Under the sharing tab go to Public Links > Create public link. Select the Download / View / Edit option and provide a secure password. Create the share URL. The url should be formatted as https://datashare.mpcdf.mpg.de/s/<PublicShareLinkEnd>, which is a series of numbers and characters. Save both the password and the <PublicShareLinkEnd> portion of the URL.

• Inside the project folder the .dvc/config file should already exist and contain the following:

[core]
    remote = datashare
['remote "datashare"']
    url = webdavs://datashare.mpcdf.mpg.de/public.php/webdav

• In the same directory should be a file called config.local.example. Copy this file to the same directory as config.local. Inside the file there should be the following:

['remote "datashare"']
    password = PublicLinkPassword
    user = PublicShareLinkEnd

• Replace PublicLinkPassword with the password you set. Replace PublicShareLinkEnd with the <PublicShareLinkEnd> you generated with the link.

• Important: Make sure to add .git\config.local to your .gitignore file.

• Now the data repository is set to your Datashare account. Optionally, if you know an existing repository that contains data you're interested in working with you can use the username and password for that Datashare repository.

Configuration:

The code is designed with modular yaml config files. Instead of directly modifying each file per run it's suggested to create experiments which are run. Experiments will use the default settings unless a setting is overridden. (see Use Example)

By default, config files are stored in the following directory structure:

config/
    actions/
    callbacks/
    criterion/
    datamodule/
    experiments/
    filters/
    logger/
    metric/
    model/
    optimizer/
    trainer/
    config.yaml

Each folder contains optional modules that can be called easily from the command line or an experiment. The core config file is the config.yaml in the root of the configs folder.

Use Example:

Let's start with a simple example. If you want to prepare the test set by itself you can run the action prepare_test. Actions like this can be defined by actions contained in the /config/actions folder. Instead of using the default action of info, we want to change the default action to prepare_test.

One method for doing this is directly from the command line. .. code-block:: console $ python hfcnn action=prepare_test ...

You can set any number of settings using command line arguments. But if you have a operation that requires setting many different parameters it's better to use experiments. You can create a new experiment in the config/experiments folder. Inside these yaml files you can override any of the default settings. For example, if you wanted to prepare_test data using a different default filename:

experiments/prepare_test_new_file_path .. code-block:: yaml stuff goes here ...

Then you'd simply run it using the following command: .. code-block:: console $ python hfcnn experiment=prepare_test_new_file_path ...

Training:

Training is no different than other actions. Simply set the action to train.

.. code-block:: console $ python hfcnn action=train ...