Remote testing environment

The main goal of the project is to run IoT.js and JerryScript tests on low footprint devices such as STM32F4-Discovery reference board. Since the size of the volatile memory (SRAM) can be very limited, the testing happens remotely. This means that the main control flow (test selection, output processing) happens on the host side and just the test execution happens on the device side. The communication is established over serial port or SSH.

The following table shows the supported devices and applications:

	IoT.js	JerryScript
STM32F4-Discovery	🗸	🗸
Raspberry Pi 2	🗸	🗸
ARTIK 053	🗸	🗸
Raspberry Pi 3	🗸

In the first step, all the dependencies should be installed:
Note: minimum system requirement for Ubuntu: 16.04

# Assuming you are in the remote testrunner folder.
$ bash install-deps.sh

Set up STM32F4-Discovery

In case of the stm32f4-discovery devices the communication happens over the serial port. That is why a miniusb (flashing) and a microusb (serial communication) cables are needed.

Eliminate root permission and use permanent device ID

By default, unknown devices require root permission to use them. On UNIX systems, udev rule files help to add permissions and symbolic links to the devices. ST-Link already has a pre-defined rule file that should be copied to the udev folder. This helps to flash the device without root permission.

# Assuming you are in the remote testrunner folder.
$ sudo cp deps/stlink/etc/udev/rules.d/49-stlinkv2.rules /etc/udev/rules.d/

It is needed to create an other udev file for the serial communication. The rules of the file should allow the member of others (like us) to read/write the serial device without root permission. Furthermore, the board is restarted at every test, so the serial device id could change (from /dev/ttyACM0 to /dev/ttyACM1) during testing. This can be eliminated, if a symbolic link could be used to identify the device.

cat >> /etc/udev/rules.d/99-udev.rules << EOL
SUBSYSTEM=="tty", ATTRS{idVendor}=="0525", ATTRS{idProduct}=="a4a7", MODE="0666", SYMLINK+="STM32F4"
EOL

Sources:

• https://github.com/texane/stlink/blob/master/doc/compiling.md#permissions-with-udev
• http://hintshop.ludvig.co.nz/show/persistent-names-usb-serial-devices/

Set up Raspberry Pi 2

Raspberry devices should have a UNIX based operating system, like Raspbian Jessie. Since the communication happens over SSH, an ssh server should be installed on the device:

# Assuming you are on the device.
pi@raspberrypi $ sudo apt-get install openssh-server

After that, Raspberry devices can handle ssh connections. In order to avoid the authentication every time, you should create an ssh key (on your desktop) and share the public key with the device:

# Assuming you are on the host.
user@desktop $ ssh-keygen -t rsa
user@desktop $ ssh-copy-id pi@address

Since Raspberry devices have much more resources than microcontrollers, it is possible to use other programs to get more precise information from the tested application (iotjs, jerry). Such a program is the Freya tool of Valgrind, that monitors the memory management and provides information about the memory usage.

Set up ARTIK 053

In case of the ARTIK 053 devices, the communication happens over the serial port. You only need a microusb cable in order to test. Use udev rule files to define the appropriate permissions and symbolic link for the device (like in case of STM32F4-Discovery):

cat >> /etc/udev/rules.d/99-udev.rules << EOL
SUBSYSTEMS=="usb",ATTRS{idVendor}=="0403",ATTRS{idProduct}=="6010",SYMLINK+="ARTIK053",MODE="0666" RUN+="/sbin/modprobe ftdi_sio" RUN+="/bin/sh -c 'echo 0403 6010 > /sys/bus/usb-serial/drivers/ftdi_sio/new_id'"
EOL

Source:

• https://github.com/Samsung/TizenRT/blob/master/build/configs/artik053/README.md#add-usb-device-rules

Set up Raspberry Pi 3

In case of the RP3 devices, the communication happens over SSH, like in case of RP2. You need to check a few things before starting remote-test.

First, you must install rpm packages which are openssh, python, rsync on the remote target. To install them, you need sdb tool which is contained within tizen-studio. For more details, please refer to this link. After you have the sdb tool, install the packages with the commands below.

# The RPi3 board doesn't need to execute 'sdb connect` command.
user@desktop $ ~/tizen-studio/tools/sdb connect <your-device-ip>
user@desktop $ ./tools/install_tizen_packages.sh

Next, in order to avoid the authentication every time, you should create an ssh key (on your desktop) and share the public key with the device:

# Assuming you are on the host.
user@desktop $ ssh-keygen -t rsa
user@desktop $ ssh-copy-id root@address

Start testrunner

On the host side, it is enough to run the jstest module.

# Assuming you are in the remote testrunner folder.
$ python -m jstest

If you add the path of js-remote-test to the PYTHONPATH environment variable, you will be able to run js-remote-test from any other directory.

$ export PYTHONPATH=/path/to/js-remote-test:$PYTHONPATH

Testrunner parameters:

--app
  Defines the application to test {iotjs, jerryscript}.

--app-path
  Defines the path to the application project.

--buildtype
  Defines the buildtype for the projects {release, debug}. Just for debugging.

--device
  Defines the target device {stm32f4dis, rpi2, artik053, rpi3}.

--public
  Publish the test results to the public database.
  This option requires a username and a password pairs that the contributors maintain.
  These information should be set by environment variables:

  $ export FIREBASE_USER="your_user_email"
  $ export FIREBASE_PWD="your_user_password"

--timeout
  Defines a time (in seconds) when to restart the device.

--no-memstat
  Skip the memory measurements.

--no-build
  Do not build the projects.

--no-profile-build
  Do not build the different profiles.

--no-flash
  Do not flash the device.

--no-test
  Do not test.

--coverage
  Defines the server address for the jerry-debugger to calculate the JS source code.

--quiet
  Make the output less verbose.

--emulate
  Emulate the connection.

--testsuite
  Specify the path to user-owned tests.

SSH communication:

--username
  Defines the username for the Raspberry Pi target.

--password
  The password to login to the device.

--ip
  IP(v4) address of the device.

--port
  Defines the SSH port. (default: 22)

--remote-workdir
  Defines the working directory where the testing happens.

Serial communication:

--device-id
  Defines the serial device id (e.g. /dev/ttyACM0)

--baud
  Defines the baud rate (default: 115200)

Telnet communication:

--router
  Defines the router address.

--netmask
  Defines the netmask.

Examples to run tests

$ python -m jstest --device stm32f4dis --app iotjs --device-id /dev/STM32F4 --baud 115200
$ python -m jstest --device stm32f4dis --app jerryscript --device-id /dev/STM32F4 --baud 115200
$ python -m jstest --device rpi2 --app iotjs --ip a.b.c.d --username pi --remote-workdir /home/pi/testrunner
$ python -m jstest --device rpi2 --app jerryscript --ip a.b.c.d --username pi --remote-workdir /home/pi/testrunner
$ python -m jstest --device artik053 --app iotjs --device-id /dev/ARTIK053 --baud 115200
$ python -m jstest --device artik053 --app jerryscript --device-id /dev/ARTIK053 --baud 115200
$ python -m jstest --device rpi3 --app iotjs --ip a.b.c.d --username root --remote-workdir /root/testrunner

All the results are written into JSON files that are found in a results folder. Name of the output files are datetime with the following format:

%YY-%mm-%ddT%HH.%MM.%SSZ.json      e.g. 2017-04-15T10.02.33Z.json

Every JSON file contain information about the test results (status, output, memory usage), environments (used hashes, commit messages) and the main section sizes of the application (iotjs or jerryscript) binary.

Run iotjs tests with coverage measurement on Artik053

You are able to run coverage measurement on ARTIK053. A modified jerry-debugger is used to calculate the covered JS source lines, so it needs to specify a unique network address for the connection with the coverage option. ARTIK053 uses wifi for the communication, so it also needs to set the following environment variables:

export ARTIK_WIFI_NAME=your_wifi_name
export ARTIK_WIFI_PWD=your_wifi_password

To run tests with coverage:

$ python -m jstest --device artik053 --app iotjs --device-id /dev/ARTIK053 --baud 115200 --coverage=DEVICE_IP:DEVICE_PORT

Note: If coverage is enabled then the memory measurement data will not be processed. However, it is shown as an information.

Run js-remote-test from Docker

js-remote-test runs best on Ubuntu systems (especially Ubuntu 16.04). If you have a different distribution and/or you can't install required dependencies, you can use the iotjs/js_remote_test docker image.

Downloading the Docker image

$ docker pull iotjs/js_remote_test:0.6

Starting a Docker container

$ docker run -dit --user jsuser -w /home/jsuser --name jsremote iotjs/js_remote_test:0.6

Possible additional arguments of the docker run command:

• -v This argument can be used to mount your js-remote-test folder in the docker container. For example:

$ docker run -dit --user jsuser -w /home/jsuser --name jsremote -v /home/user/js-remote-test:/home/jsuser/js-remote-test iotjs/js_remote_test:0.6

Now you will be able to access your js-remote-test directory in docker at /home/jsuser/js-remote-test.

This can also be used to attach serial devices to your container. For example:

$ docker run -dit --user jsuser -w /home/jsuser --name jsremote -v /dev/ARTIK053:/dev/ARTIK053 iotjs/js_remote_test:0.6

This will make /dev/ARTIK053 visible in the container with the same name. Note: if you unplug the device while the container is running, it will not work after plugging back in, even though /dev/ARTIK053 still exists in the container.

• --env This argument can be used to set environment variables in the container. It can be especially useful to set the PYTHONPATH variable:

# Assuming the path to js-remote-test in the container is /home/jsuser/js-remote-test
$ docker run -dit --user jsuser -w /home/jsuser --name jsremote --env PYTHONPATH=/home/jsuser/js-remote-test:$PYTHONPATH iotjs/js_remote_test:0.6

Executing js-remote-test

You can get a bash prompt ("login" to the container), and then run js-remote-test as usual:

$ docker exec -it jsremote /bin/bash
# Assuming you are already "logged in" to the container:
$ cd path_to_js_remote_test
$ python -um jstest ...

Alternatively, you can also execute js-remote-test without logging in (NOTE: for this you need to set the PYTHONPATH variable as instructed above.)

$ docker exec -it jsremote /bin/bash -c python -um jstest ...