Getting started MOTIF

Download Git repository

# Download this pipeline repository into `MOTIF` directory.
# We assume the directory `MOTIF` is the default working directory unless we mention the location.
git clone https://github.com/SNTSVV/MOTIF.git MOTIF
or
git clone https://gitlab.uni.lu/faqas/MutationTestingWithFuzzing.git MOTIF

Contents of archive (repository) description

case_studies : directory expected to contain case study data
containers : directory containing virtual machine images and its environment setting
configs : directory containing configuration files for each subject
pipeline : directory containing main source codes for the MOTIF pipeline
scripts : directory containing scripts for supporting experiments
test : directory containing test cases of MOTIF
tools : directory containing scripts for supporting experiments (managing result files)
motif.py : main entry point of MOTIF
Vagrantfile : configuration file for the vagrant box (for Windows and Mac OS users)

Pre-requisition

Below, we provide two alternative solutions to install MOTIF requirements: (1) execute a pre-defined virtual environment with a Vagrant box, (2) execute a pre-defined virtual environment for Docker, (3) install MOTIF and its dependencies from scratch. The above environments only includes default dependencies for executing MOTIF. Users who want to execute R scripts follows the instructions in Section R dependencies

Using Docker virtual environment

We provide pre-compiled resources and configuration files for virtual environments

Users can build and use docker image using containers/Dockerfile as following:

# These commands work for Ubuntu. 
sudo apt update -y
sudo apt install -y docker.io
sudo docker build -t motif/ubuntu22 -f containers/Dockerfile .            # build docker image
sudo docker run -v $(pwd):$(pwd) -w $(pwd) -it motif/ubuntu22 /bin/bash   # connect to the docker

Using vagrant box

Users who work on Windows and MacOS need to install Vagrant and virtual machine.
For the installation, please follow the guidelines from the official website.
We provide all the configuration to set up the vagrant instance for Singularity in Vagrantfile

Please use the commands below in the root repository of MOTIF

# The command below creates a virtual machine instance according to the Vagrantfile based on the root repository.
# This will bind automatically the root repository to the directory /vagrant inside of the container
vagrant up 
# Connect into the the vagrant box instance
vagrant ssh
# Move to the bound directory, which is sharing between vagrant container and the host OS
cd /vagrant

Now, we assume that you are in the directory /vagrant in the vagrant instance, which is the same as the root repository of the MOTIF.

Requirements to install MOTIF on host, from scratch

We recommend using Ubuntu 22.04 (some pre-compiled dependencies are included in this repository)
Install MOTIF dependencies

sudo apt-get update -y
sudo apt-get install -y git vim build-essential gcc g++ gdb

# install Python 3.10 or later
sudo apt-get install -y python3.10 python3-pip

# Upgrade pip version
sudo pip3 install --upgrade pip

# Install python dependencies
pip3 install -U -r containers/requirements.txt

Install dependencies for installing AFL++

sudo apt-get install -y build-essential python3-dev automake cmake git flex bison 
sudo apt-get install -y libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev
# try to install llvm 14 and install the distro default if that fails
sudo apt-get install -y lld-14 llvm-14 llvm-14-dev clang-14 || sudo apt-get install -y lld llvm llvm-dev clang
export GCC_VERSION=$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')
sudo apt-get install -y gcc-${GCC_VERSION}-plugin-dev libstdc++-${GCC_VERSION}-dev
sudo apt-get install -y ninja-build # for QEMU mode

install AFL++ (Users can use the pre-compiled version, see the next bullet item)

git clone https://github.com/AFLplusplus/AFLplusplus AFL++
cd AFL++
git checkout tags/v4.09c
make all LLVM_CONFIG=llvm-config-14    # to include QEMU, use "make distrib LLVM_CONFIG=llvm-config-14"
sudo make install
cd ..
rm -rf AFL++

Use precompiled AFL++ on Ubuntu 22.04 (it is compiled by gcc-10)

sudo tar xf containers/AFL++-4.09c-ALL-22.04.tar 
cd AFL++
sudo make install
cd ..
rm -rf AFL++

R dependencies

# execute provided scripts 
#   - This script is working, as default, with the pre-compiled archive.
#   - Users who wants to change the version of R or install manually, please modify the script  
./containers/vagrant-install_R.sh

Preparing target subjects

We have enclosed the subject data for the EXAMPLE project in the directory case_studies/EXAMPLE. It contains following files:

src.tar: Source code of software under test
mutants.tar: Source code of mutants
list: list of target mutants
- filtered_live: a list of live mutants generated by MASS (Mutation analysis tool)
- test: a list of live mutants selected for this tutorial

Configuration

We also have enclosed the configuration file configs/config-c.py.

Executing MOTIF for a mutant

By executing motif.py, you can do mutation testing for a mutant. We target a mutant rectangle.mut.13.1_2_18.ICR.rectangle_surface.c for this demo.

# ./motif.py [-c <CONFIG_FILE>] [-J <EXP_NAME>] [--timeout <INT>] [--phase <PHASE>] <MUTANT_NAME>
# -c <CONFIG_FILE>: providing configuration, default value is config.py 
# -J <EXP_NAME>   : name of experiment and also become the name of the output directory
# --timeout <INT> : maximum execution time of the fuzzer in seconds
# --phase <PHASE> : execution phase of MOTIF {'all', 'preprocess', 'build', 'fuzzing', 'gen'}, 
#                   'all' proceeds all following phases at once
# <MUTANT_NAME>   : the name of the target mutant
./motif.py -c configs/config-c.py -J demo --timeout 10000 rectangle.mut.13.1_2_18.ICR.rectangle_surface.c

Execute MOTIF for each phase (for a mutant)

# `preprocess`: generate source code of fuzzing drivers and seed inputs
./motif.py -c configs/config-c.py -J demo --phase preprocess rectangle.mut.13.1_2_18.ICR.rectangle_surface.c
# `build`: compile SUT and fuzzing drivers
./motif.py -c configs/config-c.py -J demo --phase build rectangle.mut.13.1_2_18.ICR.rectangle_surface.c
# `fuzzing`: executing fuzzer for the mutant
./motif.py -c configs/config-c.py -J demo --timeout 10000 --phase fuzzing rectangle.mut.13.1_2_18.ICR.rectangle_surface.c
# `gen`: After the fuzzing, you can generate test cases and check execution results with the generated inputs killing the mutants 
./motif.py -c configs/config-c.py -J demo --timeout 10000 --phase gen rectangle.mut.13.1_2_18.ICR.rectangle_surface.c

Execution results

Result file location: ./case_studies/EXAMPLE/demo/6-testcases/ProtocolLayer/TMFrameBuilder/Source/TMFrameBuilder/TMFrameBuilder.mut.489.1_4_21.ROR.TMFillHeaderValues
Take a look at the test cases: testcases/**/*.test.c
Open the execution result files
- execution result with fuzzing driver: input.exec_results.N*.presenter_driver.txt
- execution result with false-positive driver: input.exec_results.N*.false_driver.txt
- execution result of test cases: testcases.result.N*.txt

Executing MOTIF with the list of mutants

By executing motif.py with list keyword, you can do mutation testing for all the mutants that are listed a file. The following is the example usage and example command for the target subjects.

# ./motif.py list [-c <CONFIG_FILE>] [-J <EXP_NAME>] [--timeout <INT>]  [--phase <PHASE>] <MUTANT_LIST>
# -c <CONFIG_FILE> : location of configuration file 
# -J <EXP_NAME>    : name of experiment and also become the name of the output directory
# --timeout <INT>  : maximum execution time of the fuzzer
# --phase <PHASE>  : execution phase of MOTIF {'all', 'preprocess', 'build', 'fuzzing', 'gen'},
#                    'all' proceeds all following phases at once
# <MUTANT_LIST>    : a text list file of target mutants
./motif.py list -c configs/config-c.py -J demo_list --timeout 10000 case_studies/EXAMPLE/list/test

Check results for a list of mutants

# ./tools/RunCollector.py [-b <BASE_DIRECTORY>] [-J <EXP_NAME>] [--time <INT>] <MUTANT_LIST>
# -b <BASE_DIRECTORY>: WORKING DIRECTORY for the experiment, it is used for determining actual path of experiment results path and demo 
# -J <EXP_NAME>      : name of experiment 
# --time <INT>       : maximum time of experiment, should be the same value of --timeout for run.py
# <MUTANT_LIST>      : the file path for the list of mutants
./tools/RunCollector.py -b case_studies/EXAMPLE -J demo_list --time 10000 list/test

The command above will generate a summary file at case_studies/EXAMPLE/demo_list/summary.csv and print out the following output. Users can read how many mutants had been killed (KILLED), remained live (LIVE), or led to failures in the original function (FAILED_IN_ORIGIN), or were not killed because the function under test is not deterministic or the provided inputs do not satisfy preconditions (LIVE_ASSUMED). The numbers below are the example (not the result of list/test)

...
Total mutants: 100
Total LIVE mutant: 15
Total KILLED mutant: 78
Total FAILED_IN_ORIGIN mutant: 1
Total LIVE_ASSUMED mutant: 6
Done.

General guideline of MOTIF

We provide this guide for users who apply the MOTIF pipeline to a new subject.

Preparing a target subject

Create a subject directory

# You can make the directory anywhere, but we recommend you to locate it in this repository 
# so that you can easily mount the directory to the virtual machines without additional settings. 
$ mkdir -p case_studies/_SUBJECT

Copy source codes for the software under test (SUT) as a tar file.

MOTIF works with tar archive files for parallel work

# Move to the directory of the SUT
$ cd /path/from/SUT
# Make an archive in the _SUBJECT directory from the current directory, including all hidden files
#    Please make sure that all symbolic link files use relative paths and do not link outside of the SUT directory
/path/from/SUT$ tar -cf /path/to/_SUBJECT/src.tar .

Copy mutants files as a tar file
- MOTIF works with tar archive files for parallel work
- We assume that the mutants are generated by a mutation analysis tool (e.g., MASS) and the directory of the mutants has to have the same structure as the source code repository. See the example below:
  
  /path/from/SUT (source code repository)
  ./time.c
  ./main.c
  
  /path/from/MUTANTS (mutants directory)
  ./time/time.mut.12.1_1_1.ROR.system_time.c
  ./time/time.mut.13.1_1_2.UOI.system_time.c
  ./main/main.mut.14.1_1_3.SDL.main.c
- The name of the mutant file has the structure below:
  <FILE_NAME>.mut.<LINE_NO>.<EXTEND_INFO>.<MUTATION_OPERATOR>.<FUNCTION_NAME>.c
  - <FILE_NAME>: the mutated source code
  - <LINE_NO>: the mutated line number in the file
  - <EXTEND_INFO>: the extended information for the mutation (e.g., mutated column and order of the mutant))
  - <MUTATION_OPERATOR>: type of mutation operator that is applied to this mutant
  - <FUNCTION_NAME>: the name of the mutated function
- You can copy them as an archive using the following commands:
```
# Move to the directory of the mutants
$ cd /path/from/MUTANTS
# Make an archive in the _SUBJECT directory from the current directory
/path/from/SUT$ tar -cf /path/to/_SUBJECT/mutants.tar .
```
Make a list of target mutants

Among the mutants files that you copied from mutation analysis results, select and list mutants that you want to do mutation testing. Each item consists of a mutant name and a list of input filter. The mutant name can be specified with a relative path. The input filter is optional and can be specified multiple values with a delimiter ';', among {A: all, N: negative, Z: zero, P: positive}.

[relative/path/from/MUTANTS/]<mutant_name>[;input_filter]

Actual list could be one of the examples below:
```
# A simple list of mutant file name
time.mut.87.3_2_1.ICR.time_to_timestamp.c
time.mut.88.2_4_2.ROD.time_to_timestamp.c
...
```
```
# A relative mutant file path from the '_SUBJECT/mutants' directory.
./time/time.mut.87.3_2_1.ICR.time_to_timestamp.c
./time/time.mut.88.2_4_2.ROD.time_to_timestamp.c
...
```
```
# A mutant file name (also can contain relative path) and input filter.
time.mut.87.3_2_1.ICR.time_to_timestamp.c;A
time.mut.88.2_4_2.ROD.time_to_timestamp.c;Z;P
time.mut.89.2_1_3.LCR.time_to_timestamp.c;N;Z;P
...
```

Changing the configuration for a subject

The config.py is the default configuration file for the pipeline. If you want to use the other file, you can create and provide it to the pipeline using "-c" argument. Please take a look at the config.py file and change it according to your need. At least you need to provide the following information correctly:

EXP_BASE: path of the _SUBJECT directory (relative path from the root directory of MOTIF or the absolute path)
TEMPLATE_CONFIG: template configuration for the test drivers
compile configurations: INCLUDES, COMPILE_SUT_CMDS, and etc.

Working with Singularity

We provide the singularity image that we used for our experiments. To use the singularity image, users need to install the software Singularity. For the installation, please follow the guidelines from the official website. Note that we used Singularity 3.8 CE version since HPC in the University of Luxembourg uses this version.

Connecting to vagrant box (for Windows or Mac OS users)

We provide all the configuration to set up the vagrant instance for Singularity in Vagrantfile
Please use the commands below in the root repository of MOTIF

# TODO: Replace vm.box as the following configuration in `Vagrantfile`
    #config.vm.box = "generic/ubuntu2204"
    config.vm.box = "exatoa/singularity-ce-3.8-ubuntu-22.04" 
  
# The command below creates a virtual machine instance according to the Vagrantfile based on the root repository.
# This will bind automatically the root repository to the directory /vagrant inside of the container
$ vagrant up 
# Connect into the vagrant box instance
$ vagrant ssh
# Move to the bound directory, which is sharing between the vagrant container and the host OS
[vagrant]$ cd /vagrant

Now, we assume that you are in the directory /vagrant in the vagrant instance, which is the same as the root repository of the MOTIF.

Preparing singularity container

We provide the definition file ./containers/singularity_motif_22.04-llvm.def containing commands to install all the dependencies and environment

# Create a singularity container
$ sudo singularity build ./containers/motif_default_22.04-llvm.sif ./containers/singularity_motif_22.04-llvm.def

# You can test the singularity image with the following commands:
# Please note that we are binding the root repository in the directory `/expr` in the container.  
# Connect a shell in the singularity container
$ singularity shell --bind /vagrant:/expr -H /expr ./containers/motif_default_22.04-llvm.sif  
# Execute a command in the singularity container
$ singularity exec --bind /vagrant:/expr -H /expr ./containers/motif_default_22.04-llvm.sif pwd

Configuration of the Singularity

In the config.py, you need to specify the path of the singularity image for the parameter SINGULARITY_IMAGE in config.py. For example, for the motif_default_22.04-llvm.sif, you can set as follows:

SINGULARITY_IMAGE = "containers/motif_default_22.04-llvm.sif"

Installing requirements (minimum requirements to execute `./motif.py list`)

The command ./motif.py list is dedicated to the mutation testing for a list of mutants. This command will generate a list of commands using ./motif.py for a single mutant. To execute each command MOTIF will run them inside the Singularity image. Thus, on the host machine, it requires to install subset of MOTIF dependencies. Please follow the instructions:

sudo apt update -y
sudo apt install -y gcc g++ python python3 python3-pip    # Python 3.6.9 for Ubuntu 18.04, Python3.10 for Ubuntu 22.04

# python-pip upgrade
sudo pip3 install --upgrade pip

# install python dependencies
pip3 install psutil

Executing MOTIF

To make MOTIF work in the specified singularity image, you need to use --singularity argument for motif.py list. The script will then generate a list of commands with motif.py for each mutant specified in the given list of mutants and execute the commands in the singularity image. See the following examples:

$ ./motif.py list --singularity --timeout 600 case_studies/_SUBJECT/live_mutants

# The above command will call `run.py` for each mutant in the `live_mutants` in the singularity image as below:
# ./motif.py --timeout 600 <mutant_1>
# ./motif.py --timeout 600 <mutant_2>
# ./motif.py --timeout 600 <mutant_3>
# ...

This argument --singularity does not subject to the motif.py for a single mutant. To do it in the singularity image, you need to enter inside the singularity environment and execute motif.py. See the following examples:

# enter singularity environment
$ singularity shell --bind ./:/expr -H /expr ./containers/motif_default_22.04-llvm.sif
[singularity]~/$ cd /expr
[singularity]/expr$ ./motif.py --timeout 600 <mutant_1>

Working with HPC (High Performance Computing)

Users who are available to use HPC can apply the following arguments: --hpc and --parallel. Only motif.py list is subject to these parameters. To execute the commands, please consider that you have the correct working environments in HPC.

For the execution of the pipeline, you need to upload all the input files and pipeline files into the HPC. We assume that ~/<workpath> directory is the working directory for this mutation testing and has the same files and structure as the local machine above.

Mutation Testing on HPC (Sequential)

The argument --hpc is for the MOTIF pipeline to work with a job scheduler on HPC (SLURM). With the argument, the pipeline will generate a list of commands (i.e. list of motif.py commands) in a <command_file> and request a job to the SLURM by providing launcher.sh (located in the directory ./scripts/HPC/) , which is a script dedicated for SLURM job scheduler to execute sequentially each line of <command_file>. See the example below:

# run_list.py [--hpc] [--timeout <int>] <mutants_list_file> <phase>
[HPC]~/<workpath>$ ./motif.py list --hpc --runs 2 --timeout 100 case_studies/_SUBJECT/live_mutants

#### <command_file>
#./motif.py --runID 1 --timeout 100 <mutant1>
#./motif.py --runID 2 --timeout 100 <mutant1>
#./motif.py --runID 1 --timeout 100 <mutant2>
#./motif.py --runID 2 --timeout 100 <mutant2>
#./motif.py --runID 1 --timeout 100 <mutant3>
#./motif.py --runID 2 --timeout 100 <mutant3>
# launcher.sh will execute them sequentially

Mutation Testing on HPC (Parallel)

The argument --parallel will allow you to execute your commands in multiple nodes in HPC. Since --hpc argument is automatically on the argument --parallel, you do not need to provide --hpc argument together. With --parallel argument, the pipeline will generate a list of commands (i.e. list of motif.py commands) in a <command_file> and request multiple jobs to SLURM job scheduler using parallel.sh (located in the directory ./scripts/HPC/), by providing the <command_file> and line numbers that indicate the parallel.sh need to process commands. As like launcher.sh, parallel.sh is also a script dedicated for SLURM job scheduler to execute in parallel each line of <command_file>. The parallel.sh will call launcher.sh with the line number of <command_file>. See the following examples:

# run_list.py [--parallel] [--runs <int>] [--timeout <int>] <mutants_list_file> <phase>
[HPC]~/<workpath>$ ./motif.py list --phase preprocess --parallel case_studies/_SUBJECT/live_mutants
[HPC]~/<workpath>$ ./motif.py list --phase build      --parallel case_studies/_SUBJECT/live_mutants
[HPC]~/<workpath>$ ./motif.py list --phase fuzzing    --parallel --runs 10 --timeout 10000 case_studies/_SUBJECT/live_mutants
[HPC]~/<workpath>$ ./motif.py list --phase gen        --parallel --runs 10 case_studies/_SUBJECT/live_mutants

We recommend you execute each phase separately with --parallel argument, because it may cause a collision issue. For example, assume that you have 10 mutants for a function under test. Since we assume that mutants do not mutate the function prototype, we developed MOTIF to share the same fuzzing drivers for those mutants. Therefore, MOTIF will generate one set of fuzzing drivers for the 10 mutants. However, without --phase phase argument, MOTIF will generate a set of fuzzing drivers for each mutant regardless the mutated function and share the same location to store them. This may lead to a collision in that when a node builds executable fuzzing drivers, the other node overwrites the fuzzing driver in the middle. To prevent this issue, please do not use motif.py without the --phase argument.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
README		README
case_studies		case_studies
configs		configs
containers		containers
pipeline		pipeline
scripts		scripts
test		test
tools		tools
.gitignore		.gitignore
Diag-class.drawio		Diag-class.drawio
LICENSE.txt		LICENSE.txt
NOTICE.txt		NOTICE.txt
README.md		README.md
Vagrantfile		Vagrantfile
motif.py		motif.py
setup.py		setup.py
shell.sh		shell.sh
vagrant		vagrant
vagrant.18.04.Vagrantfile		vagrant.18.04.Vagrantfile

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Getting started MOTIF

Download Git repository

Contents of archive (repository) description

Pre-requisition

Using Docker virtual environment

Using vagrant box

Requirements to install MOTIF on host, from scratch

R dependencies

Preparing target subjects

Configuration

Executing MOTIF for a mutant

Execute MOTIF for each phase (for a mutant)

Execution results

Executing MOTIF with the list of mutants

Check results for a list of mutants

General guideline of MOTIF

Preparing a target subject

Changing the configuration for a subject

Working with Singularity

Connecting to vagrant box (for Windows or Mac OS users)

Preparing singularity container

Configuration of the Singularity

Installing requirements (minimum requirements to execute `./motif.py list`)

Executing MOTIF

Working with HPC (High Performance Computing)

Mutation Testing on HPC (Sequential)

Mutation Testing on HPC (Parallel)

About

Releases

Packages

Languages

License

SNTSVV/MOTIF

Folders and files

Latest commit

History

Repository files navigation

Getting started MOTIF

Download Git repository

Contents of archive (repository) description

Pre-requisition

Using Docker virtual environment

Using vagrant box

Requirements to install MOTIF on host, from scratch

R dependencies

Preparing target subjects

Configuration

Executing MOTIF for a mutant

Execute MOTIF for each phase (for a mutant)

Execution results

Executing MOTIF with the list of mutants

Check results for a list of mutants

General guideline of MOTIF

Preparing a target subject

Changing the configuration for a subject

Working with Singularity

Connecting to vagrant box (for Windows or Mac OS users)

Preparing singularity container

Configuration of the Singularity

Installing requirements (minimum requirements to execute ./motif.py list)

Executing MOTIF

Working with HPC (High Performance Computing)

Mutation Testing on HPC (Sequential)

Mutation Testing on HPC (Parallel)

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Installing requirements (minimum requirements to execute `./motif.py list`)

Packages