diff --git a/session_management/README.md b/session_management/README.md new file mode 100644 index 0000000..1c6626b --- /dev/null +++ b/session_management/README.md @@ -0,0 +1,113 @@ +# Janelia Session Manager + + +## Summary + +This Python script runs an experiment consisting of a number of sessions, with each session involving the running of a stand-alone executable built with Unity. The details of the sessions are described in a "spec" file in the [JSON format](https://en.wikipedia.org/wiki/JSON). + +## Installation + +The Python script itself has no dependencies and requires no special installation (assuming that Python is installed). It does assume that the stand-alone executables built with Unity that it is running have certain capabilities, though, that require the [org.janelia.logging package](https://github.com/JaneliaSciComp/janelia-unity-toolkit/tree/master/org.janelia.logging) and the [org.janelia.general package](https://github.com/JaneliaSciComp/janelia-unity-toolkit/tree/master/org.janelia.general). To install those packages, follow the [installation instructions in the main repository](https://github.com/JaneliaSciComp/janelia-unity-toolkit/blob/master/README.md#installation). + +## Usage + +To run an experiment described by the file `experiment.json`: +``` +python session-manager.py --input-paradigm experiment.json +``` +The `--input-paradigm` argument can be abbreviated as `-ip`. + +To make trial-specific additions to the log filename and/or header: +``` +python session-manager.py --input-paradigm experiment.json --input-trial trial.json +``` +The `--input-trial` argument can be abbreviated as `-it`. + +To start in the middle of an experiment, at session 3 for example (where the first experiment is 1): +``` +python session-manager.py -ip experiment.json --start 3 +``` +The `--start` argument can be abbreviated as `-s`. + +To print what would be done in an experiment without actually doing it: +``` +python session-manager.py -ip experiment.json --dry-run +``` + +## JSON Spec File + +Here is an example of the JSON file describing a simple three-session experiment: +```json +{ + // The executable is a shortcut (link), as produced by the + // AdjoiningDisplaysCamera in org.janelia.camera-utilities. + "exe": "C:/Users/scientist/SessionA/unity-standalone.lnk", + "logDir": "C:/Users/scientist/AppData/LocalLow/institution/SessionA", + // Specifies the session length. + "sessionParams": { + "timeoutSecs": 120 + }, + // Pause between sessions. + "pauseSecs": 5, + "sessions": [ + { + // Suffix for the log file name. + "logFilenameExtra": "_session1.1", + // Extra text at the start of the log file. + "logHeader": "First experiment, first session." + }, + { + "log_filename_extra": "_session1.2", + "logHeader": "First experiment, second session: longer initial delay.", + "pauseSecs": 10 + }, + { + "logFilenameExtra": "_session1.3", + "logHeader": "First experiment, third session: uses a different executable.", + "exe": "C:/Users/scientist/SessionB/unity-standalone.lnk", + // The log directory will be different, too. + "logDir": "C:/Users/scientist/AppData/LocalLow/institution/SessionB" + } + ] +} +``` + +The general structure of the file is: +* a group of zero or more "global" parameters that apply in all sessions unless overridden in a particular session; +* a key `"sessions"` whose value is a list of objects, one for each session, with optional parmeters specific to that session. + +In the example, above, there is a global parameter `"exe"` that is overridden in the third session. + +The supported parameters are: +* `"exe"` [required]: The path to the stand-alone executable built with Unity. For the other parameters to work, this executable's project must include the [org.janelia.logging package](https://github.com/JaneliaSciComp/janelia-unity-toolkit/tree/master/org.janelia.logging) and the [org.logging.general package](https://github.com/JaneliaSciComp/janelia-unity-toolkit/tree/master/org.janelia.general). The executable can be a link (e.g., on Windows, a "shortcut" file with the extension `.lnk`, though that extension is often hidden), like the one produced when using the `AdjoiningDisplaysCamera` class from the [org.janelia.camera-utilities package](https://github.com/JaneliaSciComp/janelia-unity-toolkit/tree/master/org.janelia.camera-utilities). +* `"logDir"` [required]: The log directory corresponding to the executable. For now, at least, this parameter cannot be used to _change_ the log directory that the executable has been built to use; this parameter merely makes that directory known to the session manager script. +* `"pauseSecs"` [optional, default: 0]: The number of seconds to pause before starting a session. +* `"sessionParams"` [optional, default: `{}`]: A JSON object of special parameters to use in the session, as implemented by the [org.janelia.general package](https://github.com/JaneliaSciComp/janelia-unity-toolkit/tree/master/org.janelia.general). + - `"timeoutSecs"` [optional, default: unlimited]: probably the most useful session parameter, specifying the length of time for the session. +* `"logFilenameExtra"` [optional, default: `""`]: An extra string to go at the end of the log file name (e.g., a standard log file name would be something like `Log_2022-09-15_14-28-54.json`, and `"logFilenameExtra": "_extra"` would change it to `Log_2022-09-15_14-28-54_extra.json`). +* `"logHeader"` [optional, default: `""`]: Text to go in a special header record at the start of the log, as described in the [org.janelia.logging package](https://github.com/JaneliaSciComp/janelia-unity-toolkit/tree/master/org.janelia.logging). + +The keys for the parameters are "normalized" when the JSON file is read, meaning they are converted to lowercase and underscore characters (i.e., `_`) are removed, and only first part is tested for a match. Hence, various altneratives keys are equivalent, such as: +* `"exe"`, `"executable"`, `"Exe"` (and also `"prog"`, `"Program"`, or `"bin"`, `"binary"`) +* `"logDir"`, `"logDirectory"`, `"log_dir"`, `"logdir"` +* `"pauseSecs"`, `"pauseSeconds"`, `"pauseseconds"`, `"pause_secs"`, `"pause"` +* `"sessionParams"`, `"session_parameters`" + - But note that the session parameters themselves do not support alternatives (e.g., `"timeoutSec"` is supported but not `"timeout"`, `"timeout_seconds"`, etc.). +* `"logFilenameExtra"`, `"log_filename_extra"`, `"logfilenameextra"` +* `"logHeader"`, `"log_header"`, `"logheader"`, `"loghead"` + +If the sessions do not override any of the global parameters, the `"sessions"` list can contain empty JSON objects (i.e., `{}`) to indicate the number of sessions in the experiment: +```json +{ + "exe": "C:/Users/scientist/SessionA/unity-standalone.lnk", + "logDir": "C:/Users/scientist/AppData/LocalLow/institution/SessionA", + "sessionParameters": { + "timeoutSecs": 5 + }, + "pauseSecs": 2, + // Four sessions with no overrriding parameters. + "sessions": [{}, {}, {}, {}] +} +``` + +Unlike traditional JSON, comments lines starting with `//` or `#` are allowed, and are removed before the file is processed. Not all structured text editors handle JSON with comments, but [Visual Studio Code](https://code.visualstudio.com) does, when the "Select Language Mode" control in the bottom right is changed to "JSON with Comments". diff --git a/session_management/session_manager.py b/session_management/session_manager.py new file mode 100644 index 0000000..8f83b14 --- /dev/null +++ b/session_management/session_manager.py @@ -0,0 +1,294 @@ +# Usage: +# python session-manager.py -ip /path/to/paradigm_input.json -it /path/to/trial_input.json +# python session-manager.py -ip /path/to/paradigm_input.json -it /path/to/trial_input.json --start 3 +# python session-manager.py -ip /path/to/paradigm_input.json -it /path/to/trial_input.json --dry-run + +import argparse +import json +import os +import platform +import time +import sys + + +def remove_comments(file): + output = "" + with open(file) as f: + for line in f: + line_stripped = line.lstrip() + if line_stripped.startswith("#") or line_stripped.startswith("//"): + # Replace a comment line with a blank line, so the line count stays the same in error messages. + output += "\n" + else: + output += line + return output + + +def get_sessions(json_all): + return json_all["sessions"] + + +def normalize_key(key): + return key.lower().replace("_", "") + + +def absolutize_path(path): + if os.path.isabs(path): + return path + else: + return os.path.join(os.path.dirname(__file__), path) + + +def get_key_value(json_session, json_all, is_key_func): + val = None + for key in json_all: + if is_key_func(key): + val = json_all[key] + for key in json_session: + if is_key_func(key): + session_val = json_session[key] + # if json_all[key] and json_session[key] are both dicts + # overwrite dictionary values with those from json_session version + if isinstance(session_val, dict) and isinstance(val, dict): + val = val.copy() + val.update(session_val) + # if either the json_all or the json_session value are not dictionaries + # simply set the return value to the session value + else: + val = session_val + return val + + +def is_executable_key(key): + key = normalize_key(key) + return key.startswith("exe") or key.startswith("bin") or key.startswith("prog") + + +def get_executable(paradigm_json_session, paradigm_json_all): + exe = get_key_value(paradigm_json_session, paradigm_json_all, is_executable_key) + + if not os.path.exists(exe): + if platform.system() == "Windows": + if not exe.endswith(".lnk"): + exe2 = exe + ".lnk" + if os.path.exists(exe2): + exe = exe2 + if not exe.endswith(".exe"): + exe2 = exe + ".exe" + if os.path.exists(exe2): + exe = exe2 + + return absolutize_path(exe) + + +def is_log_filename_extra_key(key): + key = normalize_key(key) + return key.startswith("logfilenameextra") + + +def get_log_filename_extra(json_session, json_all): + return get_key_value(json_session, json_all, is_log_filename_extra_key) or "" + + +def is_log_header_key(key): + key = normalize_key(key) + return key.startswith("logheader") + + +def get_log_header(json_session, json_all): + return get_key_value(json_session, json_all, is_log_header_key) or "" + + +def is_log_dir_key(key): + key = normalize_key(key) + return key.startswith("logdir") + + +def get_log_dir(paradigm_json_session, paradigm_json_all): + log_dir = get_key_value(paradigm_json_session, paradigm_json_all, is_log_dir_key) + return absolutize_path(log_dir) + + +def is_session_parameters_key(key): + key = normalize_key(key) + return key.startswith("sessionparam") + + +def get_session_parameters(paradigm_json_session, paradigm_json_all): + return get_key_value( + paradigm_json_session, paradigm_json_all, is_session_parameters_key + ) + + +def is_pause_key(key): + key = normalize_key(key) + return key.startswith("pause") + + +def get_pause(paradigm_json_session, paradigm_json_all): + return get_key_value(paradigm_json_session, paradigm_json_all, is_pause_key) + + +def process_log_dir(log_dir, dry_run): + if not os.path.exists(log_dir): + child = log_dir[:-1] if log_dir.endswith("/") else log_dir + parent = os.path.dirname(child) + if os.path.exists(parent): + print("Creating log directory: {}".format(log_dir)) + if not dry_run: + os.mkdir(log_dir) + else: + print( + "Warning: log directory {} does not exists, nor does the parent directory".format( + child + ) + ) + + +def process_log_filename_extra(log_filename_extra, cmd, dry_run): + if log_filename_extra: + cmd += " -logFilenameExtra {}".format(log_filename_extra) + print("Appending to the log file name: {}".format(log_filename_extra)) + return cmd + + +def process_log_header(log_header, cmd, dry_run): + if log_header: + if log_dir: + log_header_file = os.path.join(log_dir, "logHeader.txt") + print("Writing to {}:\n{}".format(log_header_file, log_header)) + if not dry_run: + if os.path.exists(log_header_file): + os.remove(log_header_file) + with open(log_header_file, "w") as f: + f.write(log_header + "\n") + cmd += " -addLogHeader" + return cmd + + +def process_session_params(session_params, cmd, dry_run): + if session_params: + if log_dir: + session_params_file = os.path.join(log_dir, "SessionParameters.json") + pretty = json.dumps(session_params, indent=4) + print("Writing to {}:\n{}".format(session_params_file, pretty)) + if not dry_run: + if os.path.exists(session_params_file): + os.remove(session_params_file) + with open(session_params_file, "w") as f: + f.write(pretty + "\n") + return cmd + + +def process_pause(pause_secs, cmd, dry_run): + if pause_secs: + print("Pausing {} secs".format(pause_secs)) + if not dry_run: + time.sleep(pause_secs) + return cmd + + +def process_missing_exe(exe): + if platform.system() == "Windows": + if not exe.endswith(".exe") and not exe.endswith(".lnk"): + print("The executable may need an explicit '.exe' or '.lnk' extension") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--input-paradigm", + "-ip", + dest="input_paradigm_json_file", + help="path to the JSON file describing the paradigm(s)s", + ) + parser.add_argument( + "--input-trial", + "-it", + dest="input_trial_json_file", + help="path to the JSON file describing the trial(s)", + ) + parser.set_defaults(start=1) + parser.add_argument( + "--start", + "-s", + type=int, + dest="start", + help="session to start at (1 is the first)", + ) + parser.set_defaults(dry_run=False) + parser.add_argument( + "--dry-run", + "-dry-run", + dest="dry_run", + action="store_true", + help="only print what would be run", + ) + args = parser.parse_args() + + print("Using input paradigm JSON file: {}".format(args.input_paradigm_json_file)) + print("Using input trial JSON file: {}".format(args.input_trial_json_file)) + print("Dry run: {}".format(args.dry_run)) + + paradigm_json_all = json.loads(remove_comments(args.input_paradigm_json_file)) + trial_json_all = json.loads(remove_comments(args.input_trial_json_file)) + + paradigm_json_sessions = get_sessions(paradigm_json_all) + trial_json_sessions = get_sessions(trial_json_all) + + if len(paradigm_json_sessions) != len(trial_json_sessions): + print( + "the input-paradigm and input-trial files must have the same number of sessions!" + ) + print(f"the input-paradigm file has {len(paradigm_json_sessions)} sessions") + print(f"the input-trial file has {len(trial_json_sessions)} sessions") + sys.exit() + + for i in range(args.start - 1, len(paradigm_json_sessions)): + print(f"Session {i+1} / {len(paradigm_json_sessions)}: ") + + paradigm_json_session = paradigm_json_sessions[i] + trial_json_session = trial_json_sessions[i] + + exe = get_executable(paradigm_json_session, paradigm_json_all) + cmd = exe + + log_dir = get_log_dir(paradigm_json_session, paradigm_json_all) + process_log_dir(log_dir, args.dry_run) + + paradigm_log_filename_extra = get_log_filename_extra( + paradigm_json_session, paradigm_json_all + ) + trial_log_filename_extra = get_log_filename_extra( + trial_json_session, trial_json_all + ) + full_log_filename_extra = paradigm_log_filename_extra + trial_log_filename_extra + cmd = process_log_filename_extra(full_log_filename_extra, cmd, args.dry_run) + + paradigm_log_header = get_log_header(paradigm_json_session, paradigm_json_all) + trial_log_header = get_log_header(trial_json_session, trial_json_all) + full_log_header = paradigm_log_header + "\n" + trial_log_header + cmd = process_log_header(full_log_header, cmd, args.dry_run) + + session_params = get_session_parameters( + paradigm_json_session, paradigm_json_all + ) + cmd = process_session_params(session_params, cmd, args.dry_run) + + pause_secs = get_pause(paradigm_json_session, paradigm_json_all) + cmd = process_pause(pause_secs, cmd, args.dry_run) + + print(cmd) + + if not os.path.exists(exe): + print("Cannot find executable {}".format(exe)) + process_missing_exe(exe) + print("Skipping") + continue + + if not args.dry_run: + os.system(cmd) + + print("") + + print("Done")