BQ_module_generator Design Choices

Overview

BQ_module_generator has 3 main components:

• bqmod (A command-line interface): Generates the module xml.
• PythonScriptaWrapper: Parses module xml, establishes communication with Bisque, runs module.
• BQ_run_module.py: User-defined file called by PythonScriptWrapper that reads data from the module container, pre-processes it, runs inference, and returns output paths to PythonScriptWrapper.

BQ_module_generator is structured such that users don't need to edit the PythonScriptWrapper or xml files to create modules. These files are the most tedious and time-consuming part of the process. The PythonScriptWrapper in this repo has been written such that it can establish communication between Bisque and modules that follow certain naming conventions and guidelines. This allows users to build modules by only dealing with their own code rather than learning to use the Bisque API. Furthermore, users can utilize a command-line interface (CLI) to generate a standardized xml file of their module that PythonScriptWrapper can parse and establish communication with Bisque. This CLI sets the configuration of the module with easy-to-use commands in the terminal. Finally, the BQ_run_module.py is the only code that the user needs to write in order to create their module. This file will have a run_module function which will be called by PythonScriptWrapper at runtime. This function will have to follow some specific naming conventions and structure in order to correctly parse the arguments being passed by PythonScriptWrapper as well as return the appropriate data.

bqmod

Bqmod is the command-line interface that helps users set the configuration settings needed to generate the module xml file. This file is read by both Bisque and PythonScriptWrapper. Bqmod is implemented in bqmodule.py with the help of the click library. To get familiar with this library, read the click library documentation.

The main steps for bqmod are:

1. Create bqconfig file with 'bqmod init' in {ModuleName} folder
2. Set module name, author, and short description with 'bqmod set'
3. Set input resource types and names with 'bqmod inputs'
4. Set output types and names with 'bqmod outputs'
5. Review module configurations with 'bqmod summary'
6. Check BQ_run_module dictionary keys match bqmowd input/output names with 'bqmod check_config'
7. Create module files with 'bqmod create_module'
8. Generate help.html from help.md with 'bqmod gen_help_html'

There are a number of functions in bqmodule.py that are used for dealing with each of the bqmod commands. Most are very well documented in the code and you can read more about them there, some that might need a little more attention are:

• bqmod: This is what click refers to as the group command since all commands under it will need to be prefixed by bqmod command. Thus, anytime we call a command, the code in the bqmod function will run first. We can control the behavior of this bqmod function with respect to which subcommand is called by using the ctx.invoked_subcommand attribute, if ctx.invoked_subcommand == 'init': pass. By default, this function will always load in the bqconfig.json file present in the current directory into a dictionary saved as the context object ctx.obj. Only when the invoked subcommand is init, will this not happen since it probably means there is no bqconfig.json created yet or the user means to overwrite it. As you may read in the click documentation, any command you wish to be part of this group will use the decorator @bqmod.command("name_of_command"). The ctx.obj mentioned earlier is an object passed to all commands of the bqmod group that have the decorator @click.pass_context. We use it in bqmod to pass the dictionary stored in bqconfig.json in order to make changes, append to it, and eventually to created the module xml.
• download_files: This function is used to pull the necessary files into the module folder, if the URL for any file changes, it must be reflected here.
• check_config_main: As explained in the README of the project, the dictionary keys used in BQ_run_module must match the input and output names set in the CLI. In order to avoid bugs, this function checks whether there are any inconsistencies between these.

PythonScriptWrapper.py

PythonScriptWrapper reads and parses the module xml to identify the number, types, and names of the inputs and outputs in a module. This is how it is able to establish communication with Bisque for modules with a wide variety of input/output types and quantities. PythonScriptWrapper has x main components.

• main: parses arguments, ensures all necessary credentials are passed, initializes Bisque session, calls run function, and calls upload_results function.
  • setup: updates mex message, calls mex_parameter_parser, and instantiates self.output_resources list
    • self.mex_parameter_parser: Parses inputs from the mex XML which is different from the module XML, and adds the input uris to the options attribute so they can be loaded by fetch_input_resources.
  • run: Sets the container's inputs_dir_path and outputs_dir_path. By default, they are set to the current working directory /module. If changed, must make sure that the fetch_input_resources function is pulling inputs to the correct folder. Then has a series of try catches that call fetch_input_resources, run_module, and upload_results.
    • fetch_input_resources: Instantiates input_path_dict dictionary which will contain the paths within the container to the inputs. It then finds the tag within the module xml with the name inputs and iterates through its child tags that have the type resource. It then finds each input name from the module xml and uses bq.load(getattr(self.options, input_name)) to create a bqapi.bqclass.BQResource object which has the following attributes: name, uri, ts, and resource_unique. We then use this object to set input_path_dict[input_name] = os.path.join(inputs_dir_path, resource_obj.name) and fetch_blob_output = fetch_blob(bq, resource_obj.uri, dest=input_path_dict[input_name]) . Important to note that input_name refers to the name set with bqmod inputs -{i,f,t} -n {input_name} while resource_obj.name refers to the name of the file uploaded to bisque for inference, eg. 'whale.jpeg', 'yolov5s.pt', 'input_matrix_file.npy'. The function fetch_blob(bq, resource_obj.uri, dest=input_path_dict[input_name]) pulls the resource with URI resource_obj.uri from Bisque and puts it inside the container with the path determined by input_path_dict[input_name]. Finally, it returns the input_path_dict which has the container paths to each input.
    • run_module: This will be the user-defined function imported from BQ_run_module.py. It will take the input_path_dict created in fetch_input_resources and the output_folder_path defined in __run__. It will return a dictionary self.output_data_path_dict which will contain the paths to each output within the container indexed by the output name set with bqmod inputs -{i,f,t} -n {output_name}.
    • upload_results: Instantiates output_resources which is a list of xml tag strings for each output defined in the module xml.
  • teardown: takes the output xml tags set in run, creates an output xml tag that contains all output tags with their respective uri, and passes this to finish_session which updates the mex with this output tag.