bag_converter

A package for converting rosbags from grinder tests using the data_gathering package, and compiling them into datasets suitable for modelling.

This is done using two main scripts:

• convert_bag.py
• compile_data.py

convert_bag.py

This script converts the rosbag into a .csv file. The rosbag contains timestamped messages of various topics. The .csv contains a row for each unique timestamp that exists in the headers of the messages. Initially, fields will be set to None, until a message containing information for that field occurs. After this, each row will contain the most recent information. This means that at each row, some value will be updated compared to the previous, but most values will remain the same.

compile_data.py

This script iterates over the 'csv bags' generated by convert_bag.py, and extracts datapoints from them. The csv bags contain a time series of information from a single test. However, for modelling single values should be extracted from this time series, such as the average force during the test, the average RPM etc. This is what compile_data.py does.

Dataflow and usage

Rosbags are recorded using data_gathering. These contain messages on a couple of topics as specified in the launch file. Aside from these messages, information such as the test settings, the belt grit and test plate thickness are stored in the filename of the rosbag. The filename also has some identifier as specified by the sample variable in the launch file. This identifier can be used by compile_data.py and convert_bag.py to filter out other tests.

After the tests are performed, the rosbags are converted to .csvs using convert_bag.py. In this script, the variable test_identifiers can be used to find the specific rosbags belonging to these tests. There are two other variables that can be set. REPROCESS_PCLS can be set to True, to ignore the published grind_volme messages in the rosbag, and instead reprocess the raw pointcloud data. This is useful if something went wrong with the volume calculation code. This can then be updated and used to reprocess the source data for more accurate volume information. OVERWRITE_AREA can be set to manually enter a surface area for the test, instead of using the message in the rosbag. The purpose of this is to process older rosbags from before a message containing the surface area was published.

After converting the rosbags to csv bags, compile_data.py is used to compile the time series data into a collection of data points. Similar to the variable in convert_bag.py, the test_identifiers variable can be used to target specific rosbags.

Alteration

To expand or adjust the functionality of these scripts to different topics, the convert_bag should be altered.

• The path to each present message type should be added to msg_paths
• A dictionary should be made containing information on how to process this message. It should contain:
  • 'parser`: A function which returns a tuple containing the relevant information extracted from the message. For examples, see those already present.
  • 'column_headers': A list of headers which should be present in the csv for this topic, corresponding to the entries in the tuple.
  • 'timetype': Either 'ros' or 'plc', depending on whether the timestamp on the message originates from ROS or the PLC.
• Add 'topic': dict to the topic_dict, where dict is the dictionary created in the previous step.

Note that some single-valued information which therefore does not fit into a csv well, such as resulting removed material, is communicated between the rosbag, convert_bag.py, and compile_data.py through the filename. If the naming scheme changes, this should be adjusted into the scripts as well.