Large-Scale_Studies.md

Running Large-Scale Studies

If you have not already done so, install pymongo; see: https://pypi.org/project/pymongo/.

Microarchitecture research tends to involve large state-space searches comprised of many thousands or even millions of different configurations of the caches, the branch predictor, decoder, etc. Manually launching such large numbers of experiments is a significant challenge.

Therefore, to facilitate large-scale studies, Nebula includes executor.py, which takes as input a JSON-formatted expression of the state space to be explored. Consider the included executor.py script l1ic_size.exec:

{
    "pipelines/pompia": {
        "config": {
            "watchdog:result_name": ["retire"],
            "watchdog:result_cycles": [1000],
            "fetch:l1ic_nsets": [16, 64],
            "fetch:l1ic_nways": [2, 4],
            "fetch:l1ic_nbytesperblock": [16, 32],
            "fetch:l1ic_evictionpolicy": ["random", "lru"],
            "l2:nbytesperblock": [64],
            "mainmem:capacity": [4297967296]
        },
        "max_cycles": 100000,
        "script": "localhost.nebula",
        "service": ["../../components/simplemainmem/mainmem.py:localhost:-1:-1"],
        "command": "../../examples/bin/sum 2 3 -5 7"
    },
    "pipelines/rangpur": {
        "config": {
            "watchdog:result_name": ["retire"],
            "watchdog:result_cycles": [1000],
            "fetch:l1ic_nsets": [16, 64],
            "fetch:l1ic_nways": [2, 4],
            "fetch:l1ic_nbytesperblock": [16, 32],
            "fetch:l1ic_evictionpolicy": ["random", "lru"],
            "decode:max_bytes_to_decode": [32],
            "l2:nbytesperblock": [64],
            "mainmem:capacity": [4297967296],
            "_comment": "decode:max_bytes_to_decode must be >= fetch:l1ic_nbytesperblock"
        },
        "max_cycles": 100000,
        "script": "localhost.nebula",
        "service": ["../../components/simplemainmem/mainmem.py:localhost:-1:-1"],
        "command": "../../examples/bin/sum 2 3 -5 7"
    }
}

This script spawns runs of two pipelines: Pompia and Rangpur, the only pipelines that have L1 instruction caches. Separate runs will be spawned for the cross product of two configurations of the number of sets (16 or 64), the number of ways (2 or 4), the number of bytes/block (16 or 32), the eviction policy (LRU or random); for a total of 16 configurations for each pipeline. To execute the all 64 runs, from the top-level directory of the source tree, execute:

mkdir /tmp/runs
python3 ./executor.py \
    --purge_successful \
    --basepath /tmp/runs \
    --max_cpu_utilization 80 \
    --timeout 3600 \
    -- \
    l1ic_size.exec

The first parameter, --purge_successful, will delete from the file system the artifacts (e.g., log files, the mainmem file, stats.json) produced by each execution of the simulator that returns error code 0. The --basepath parameter points to the place for all the file system artifacts to be deposited during the execution of the simulations; default: /tmp. The --max_cpu_utilization parameter caps the CPU utilization so that many thousands of simulations do not launch simultanesouly and cripple the host; default: 90. The --timeout parameter caps the amount of time each simulation is allowed to run. Here, any simulation that does not end in an hour (i.e., 3,600 seconds) will be forcibly terminated, preventing an errant simulation from preventing executor.py from successfully concluding; default: no limit. And l1ic_size.exec is the input to executor.py.

Finally, since wrangling all the output generated by the simulations is also a significant challenge, exeuctor.py also includes simple support for inserting artifacts of each simulation into a MongoDB (see: https://www.mongodb.com/) database; to wit:

mkdir /tmp/runs
python3 ./executor.py \
    --purge_successful \
    --basepath /tmp/runs \
    --max_cpu_utilization 80 \
    --timeout 3600 \
    --mongodb mongodb://citrus.local:27017 nebula experiments \
    -- \
    l1ic_size.exec

This command line operates identically to the preceding, but also inserts documents into the MongoDB server running on my internal network named citrus.local at port 27017, in a database named nebula, in a collection named experiments. It inserts one document per configuration, and the documents contain all the data generated by the simulation execution and important metadata about the simulation including the date, time, the Git SHA of the source tree, the branch of the source tree, the exit code, the log files generated by the execution, the configuration used by the simulation, the path within which the execution artifacts were deposited during the execution, the pipeline that executed, the simulator execution script (e.g., localhost.nebula), and the command line to invoke the simulation.

For an example of how to use Jupyter Notebooks and the Pandas data anaylsis software to study data generated by executor.py and placed into a MongoDB database, load the MongoDB_Data_Analysis_Example.ipynb into a Jupyter Notebook interpreter, e.g., https://code.visualstudio.com/docs/datascience/jupyter-notebooks.