Add automated benchmarks (#256)

* Added initial benchmarks

* Heat benchmarks

* Moved file logging

* Added a basic pipeline for benchmarks.

* Full pipeline to markdown table

* Better log organization and keeps tracks of solver steps

* Added support for multiple sims

* Delete Manifest.toml

* Added projects to julia scripts

* Explicitly match the period char

* Overhauled config generation

* Removed generated files

* Overhauled configuration generation

* Support for CUDA benchmarking

* Deleted old benchmarking suite

* Integrated DrWatson into benchmarks

* Stronger config generation

* Much stronger data collection

* Added full README

* Remove my email

* Added compat entries

* Moved more code into src

* Shuffle main code around

* More config tests

* Clean up result processing

* Testing of data aggregation

* Add post processing scripts

* Use Brusselator from the Canon

* Add support for running tagged simulations

* Add back brussel canon

* Support full runs and single runs

* Update config gen to support tagging

* Added automated config generation

* Added Cahn-Hilliard files

* Cleaning up and more tests

* Better parsing of config files

* Better file accessing, more data collection

* Support for main args for slurm

* Updated readme for new workflow

* Cleaning up

---------

Co-authored-by: George Rauta <[email protected]>
Co-authored-by: George Rauta <[email protected]>
Co-authored-by: Luke Morris <[email protected]>

benchmarks/Project.toml

@@ -0,0 +1,46 @@
+name = "benchmarks"
+uuid = "b465d61d-1350-4ef5-9abc-112f9dc9b757"
+version = "0.0.1"
+
+[deps]
+ACSets = "227ef7b5-1206-438b-ac65-934d6da304b8"
+BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
+CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
+CombinatorialSpaces = "b1c52339-7909-45ad-8b6a-6e388f7c67f2"
+ComponentArrays = "b0b7db55-cfe3-40fc-9ded-d10e2dbeff66"
+DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
+Decapodes = "679ab3ea-c928-4fe6-8d59-fd451142d391"
+DiagrammaticEquations = "6f00c28b-6bed-4403-80fa-30e0dc12f317"
+DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1"
+GeometryBasics = "5c1252a2-5f33-56bf-86c9-59e7332b4326"
+JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
+LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
+MLStyle = "d8e11817-5142-5d16-987a-aa16d5891078"
+OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
+PrettyTables = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
+TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
+
+[compat]
+ACSets = "0.2"
+BenchmarkTools = "1.5"
+CUDA = "5.4"
+CombinatorialSpaces = "0.6.7"
+ComponentArrays = "0.15"
+DataFrames = "1.6"
+Decapodes = "0.5.5"
+DiagrammaticEquations = "0.1.6"
+DrWatson = "2.15.0"
+GeometryBasics = "0.4"
+JLD2 = "0.4"
+LinearAlgebra = "1.10"
+MLStyle = "0.4"
+OrdinaryDiffEq = "6.86"
+PrettyTables = "2.3"
+TOML = "1.0"
+julia = "1.10.0"
+
+[extras]
+Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
+
+[targets]
+test = ["Test"]

benchmarks/README.md

@@ -0,0 +1,89 @@
+# Benchmarks
+
+## DrWatson Initialization
+
+This code base is using the [Julia Language](https://julialang.org/) and
+[DrWatson](https://juliadynamics.github.io/DrWatson.jl/stable/)
+to make a reproducible scientific project named
+> benchmarks
+
+To (locally) reproduce this project, do the following:
+
+0. Download this code base. Notice that raw data are typically not included in the
+   git-history and may need to be downloaded independently.
+1. Open a Julia console and do:
+
+   ```julia
+   julia> using Pkg
+   julia> Pkg.add("DrWatson") # install globally, for using `quickactivate`
+   julia> Pkg.activate("path/to/this/project")
+   julia> Pkg.instantiate()
+   ```
+
+This will install all necessary packages for you to be able to run the scripts and
+everything should work out of the box, including correctly finding local paths.
+
+You may notice that most scripts start with the commands:
+
+```julia
+using DrWatson
+@quickactivate :benchmarks
+```
+
+which auto-activate the project, enable local path handling from DrWatson and provide several helper functions. Note that `@quickactivate :benchmarks` is only usable from within the `benchmarks` directory.
+
+## Establishing Simulation Configurations
+
+To establish a set of configurations for a simulation, you should create a file `src/$physics/config.toml` with the below structure.
+
+1. For a given physics (heat, brusselator, etc), entries are structured as `[$physics.$architecture.$tag]`, e.g. `[heat.cpu.example]`.
+2. Under an entry, list all the parameters desired. This should be structured as either `$param_name = [val1, val2, ...]` or `$param_name = val`.
+3. You should always include a `code_target`, which takes either `CPUTarget` or `CUDATarget` as a string.
+
+## Creating a Simulation File
+
+These benchmarks depend upon you to create the simulation files to be benchmarked. For a certain simulation named ```example```, the simulation file would be ```src/example/example.jl```.
+
+Always start the file with the following, as it provides you access to helper functions located in ```src/helpers```.
+
+```julia
+using DrWatson
+@quickactivate :benchmarks
+```
+
+Additionally, this file should contain the following functions.
+
+1. ```setup_config```, which will take in a ```Dict{String, Any}``` with the provided parameter names pointing to that task's provided configuration values. It's up to you to take these values, process them and then organize them to be passed along.
+2. ```setup_benchmark```, which will create the Decapode and run ```eval(gensim())``` on it and return that evaluated function.
+3. ```create_mesh```, which will create the mesh upon which the simulation will run and also initialize the initial conditions and any constants/parameters. Return the mesh, initial conditions and constants/parameters in that order.
+4. ```create_simulate```, which will take the generated mesh and evaluated function and run the simulate function. Return the resulting function.
+5. ```run_simulation```, which will take the resulting simulation function, initial conditions and constants/parameters and run the solve. Return the result of the solve.
+
+## Running the Benchmarks
+
+Use the `main_config.toml` to list out which physics configuration entries you would like to be run. These entries correspond one-to-one with the entries in the physics configurations. However, these `main_config.toml` entries can take optional arguements to customize how the simulations are run. Currently supported arguments are:
+
+1. `slurm_args`, passed as a vector of strings for each seperate `sbatch` argument. These arguments will be added to each job for that entry.
+2. `concur_jobs`, passed as an integer that determines how many jobs can run at a time for that configuration.
+
+Once done, simply run `scripts/main.sh`. 
+
+As another option, the name of a specific configuration entry in the `main_config.toml` can be passed to `main.sh` to only run that job, e.g. `main.sh heat cpu example`.
+
+**Warning**: The caller of this script should be in the `benchmarks` directory.
+
+## Data Collection and Processing
+
+Once a simulation is run, their outputs will be saved in `data/sims/$physics`. The benchmark JSON files will contain the full result of the benchmarking run while the stat JLD2 files will contain the solve result statistics from DEStats from OrdinaryDiffEq.jl.
+
+These files will then be processed and have their data stored in `data/exp_pro/$physics/$slurm_job_id/autogen`. Each result file in that directory will contain the processed results from one single task.
+
+These result files can then be collected with `collect_results` on the `autogen` directory and post-processed with `DataFrames.jl` functions to create the desired tables. An example of this kind of post-processing script is included in `scripts/post_processing`.
+
+Because these files are expected to be collected using DrWatson's `collect_results` , there is no intended correlation between a task and its result file name.
+
+**Warning**: Not all information from the benchmarking run is saved to the result files and any files in `data/sims/$physics`, specifically the JSON and JLD2 files mentioned above, will be deleted upon the next benchmark run for that physics. On the other hand, result files in the `autogen` directory mentioned before will never be deleted by the benchmarking.
+
+## Testing
+
+The benchmarks have a few tests that can be used to establish the robustness of the system. To run them, activate the `benchmarks` environment and then enter `test`.

benchmarks/scripts/array.jl

@@ -0,0 +1,59 @@
+using DrWatson
+@quickactivate :benchmarks
+
+using BenchmarkTools
+using TOML
+
+const task_key = ARGS[1]
+const physics = ARGS[2]
+const arch = ARGS[3]
+const tag = ARGS[4]
+
+sim_namedata = SimNameData(physics, arch, tag, task_key)
+
+function extract_task_config(config_data)
+  if !haskey(config_data, task_key)
+    error("Warning: Task with key $(task_key) could not find config data")
+  end
+
+  task_config_data = all_config_data[task_key]
+  @info string(task_config_data)
+
+  task_config_data
+end
+
+@info "Running $physics on $arch, tagged as $tag, array id is $task_key"
+
+# Extract data
+all_config_data = TOML.parsefile(simconfig_path(sim_namedata))
+task_config_data = extract_task_config(all_config_data)
+
+# Grab user's physics file
+include(physicsfile_path(sim_namedata))
+
+sim_instance = pass_simulation_instance()
+
+config = sim_instance.setup_config(task_config_data)
+
+# Get intermediate variables to use in benchmarking
+sim = sim_instance.setup_benchmark(config);
+sd, u0, cnst_param = sim_instance.create_mesh(config);
+fm = sim_instance.create_simulate(config, sd, sim);
+result = sim_instance.run_simulation(config, fm, u0, cnst_param);
+
+# Save solver statistics
+stats_data = tostringdict(struct2dict(result.stats))
+wsave(statsfile_path(sim_namedata), stats_data)
+
+# Setup and run benchmarking
+simulation_suite = BenchmarkGroup()
+
+stages = solver_stages()
+simulation_suite[task_key][stages[1]] = @benchmarkable sim_instance.setup_benchmark($config) gctrial=true
+simulation_suite[task_key][stages[2]] = @benchmarkable sim_instance.create_mesh($config) gcsample=true
+simulation_suite[task_key][stages[3]] = @benchmarkable sim_instance.create_simulate($config, $sd, $sim) gctrial=true
+simulation_suite[task_key][stages[4]] = @benchmarkable sim_instance.run_simulation($config, $fm, $u0, $cnst_param) gcsample=true
+
+tune!(simulation_suite)
+deca_sim_results = run(simulation_suite; verbose = true)
+BenchmarkTools.save(benchfile_path(sim_namedata), deca_sim_results)

benchmarks/scripts/array.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+#SBATCH --job-name=PRINT
+#SBATCH --output=printlog_%A_%a.txt
+#SBATCH --mem=16GB
+#SBATCH --time=01:00:00
+
+pwd; hostname; date
+
+module load julia
+
+SIMNAME=$1
+ARCH=$2
+TAG=$3
+
+julia array.jl $SLURM_ARRAY_TASK_ID $SIMNAME $ARCH $TAG
+
+date

benchmarks/scripts/clean.jl

@@ -0,0 +1,13 @@
+using DrWatson
+@quickactivate :benchmarks
+
+# TODO: Can improve by seperating bench and final logs
+file_regex = r"^.*log_(\d+)(?:_\d+)?\.txt$"
+
+file_matches = filter(!isnothing, map(x -> match(file_regex, x), readdir(scriptsdir())))
+
+for file in file_matches
+  tgt_dir = scriptsdir("slurm_logs", "logs_"*file[1])
+  mkpath(tgt_dir)
+  mv(scriptsdir(file.match), joinpath(tgt_dir, file.match))
+end

benchmarks/scripts/final.jl

@@ -0,0 +1,13 @@
+using DrWatson
+@quickactivate :benchmarks
+
+include(helpersdir("data_aggr_helper.jl"))
+
+using TOML
+
+const slurm_id = ARGS[1]
+const physics = ARGS[2]
+
+aggregate_data(slurm_id, physics)
+
+run(`julia --threads=auto $(postprocessdir("default_out.jl")) $slurm_id $physics`)

benchmarks/scripts/final.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+#SBATCH --job-name=FINAL
+#SBATCH --output=finallog_%j.txt
+#SBATCH --mem=1GB
+#SBATCH --time=01:00:00
+
+pwd; hostname; date
+
+module load julia
+
+SIMNAME=$1
+
+julia final.jl $SLURM_JOB_ID $SIMNAME
+
+date
+
+julia clean.jl
+

benchmarks/scripts/main.jl

@@ -0,0 +1,32 @@
+using DrWatson
+@quickactivate :benchmarks
+
+@info "Precompiling Julia"
+using Pkg
+Pkg.instantiate()
+Pkg.precompile()
+@info "Finished precompiling Julia"
+
+using TOML
+
+include(helpersdir("main_source.jl"))
+
+if length(ARGS) == 0
+    @info "Running all sims"
+    generate_all_configs()
+    run_all_physics()
+
+elseif length(ARGS) == 3
+    @info "Running single sim"
+    generate_all_configs()
+
+    const physics = ARGS[1]
+    const arch = ARGS[2]
+    const tag = ARGS[3]
+
+    run_physics = SimNameData(physics, arch, tag)
+    is_valid_config_instance(run_physics)
+    run_single_physics(physics, [run_physics])
+else
+    error("Usage: ['physics' 'architecture' 'tag']")
+end

benchmarks/scripts/main.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+DIR=scripts
+cd $DIR
+
+module load julia
+
+if [ $# == 3 ]
+then
+    SIMNAME=$1
+    ARCH=$2
+    TAG=$3
+    julia --threads=auto main.jl $SIMNAME $ARCH $TAG
+elif [ $# == 0 ]
+then
+    julia --threads=auto main.jl
+else
+  echo "Usage: ['physics' 'architecture' 'tag']"
+  exit 1
+fi
+

benchmarks/scripts/post_processing/default_out.jl

@@ -0,0 +1,44 @@
+using DrWatson
+@quickactivate :benchmarks
+
+include(helpersdir("data_aggr_helper.jl"))
+
+using TOML
+using DataFrames
+using PrettyTables
+
+const slurm_id = ARGS[1]
+const physics = ARGS[2]
+
+sims_to_process = collect_mainconfig_simentries(physics)
+
+# TODO: Have meta config information be in a seperate toml
+config_data = load_simconfig(first(sims_to_process))
+meta_config = meta_config_info(config_data)
+const meta_field_names = split(meta_config["fields"], ",")
+
+# TODO: Meant as a basic data processing pipeline
+# Can create multiple scripts to roughly process data in general ways
+pretty_results = collect_results(aggdatadir(physics, slurm_id))
+
+median_times = map(stage -> benchmark_headername(stage, "Median", "time"), solver_stages())
+table_header = vcat(["Task ID", "statsfile", "benchfile"], meta_field_names, median_times, ["nf"])
+
+select!(pretty_results, table_header)
+
+for time in median_times
+  transform!(pretty_results, [time] => ByRow(x -> x / 1e9) => [time])
+end
+
+# TODO: Can choose other sorting methods
+for field_name in reverse(meta_field_names)
+  sort!(pretty_results, Symbol(field_name))
+end
+
+mkpath(tablesdir(physics, slurm_id))
+
+# TODO: Can change this backened to be different from markdown
+open(tablesdir(physics, slurm_id, "default_output.md"), "w") do results_file
+  conf = set_pt_conf(tf = tf_markdown)
+  pretty_table_with_conf(conf, results_file, pretty_results; header = table_header)
+end

benchmarks/src/benchmarks.jl

@@ -0,0 +1,9 @@
+module benchmarks
+
+include(joinpath("helpers", "constants.jl"))
+include(joinpath("helpers", "paths.jl"))
+include(joinpath("helpers", "param_parsing.jl"))
+include(joinpath("helpers", "sim_interface.jl"))
+include(joinpath("helpers", "main_config_helper.jl"))
+
+end

benchmarks/src/helpers/constants.jl

@@ -0,0 +1,11 @@
+export solver_stages, supported_arches, is_supported_arch, meta_config_id, mainsim_config_path
+
+const solver_stages_list = ["Setup", "Mesh", "Simulate", "Solve"]
+solver_stages() = return solver_stages_list
+
+const supported_architectures_list = ["cpu", "cuda"]
+supported_arches() = return supported_architectures_list
+is_supported_arch(arch) = return arch in supported_arches()
+
+const meta_key = string(0)
+meta_config_id() = return meta_key