Regression Test System

vogl has a tracing/replaying/trimming regression and smoke test system that runs the following steps on a library of traces:

• Plays back either an apitrace or a vogl trace, captures its output using the libvogltrace SO, and records the backbuffer CRC's (or per-component checksums on traces with multisampling) to a text file.
• Plays back this trace and diffs the backbuffer CRC's vs. the CRC's seen during tracing.
• Finally, we trim the test trace, then play back the trimmed trace and compare the backbuffer CRC's vs. the original trace's CRC's. Trimming involves playing back the test trace up to a predetermined point, capturing the entire GL state vector to memory and serializing it out, so we get a lot of good coverage in this step. The system is located in the test directory of vogl's chroot repository, here. The script that runs the test is run_tests.sh. (This little script actually compiles and launches a small .C file that contains the entire test system.) The file tests.json configures which traces are tested and the parameters to the various test steps.

Currently, only our smallest traces (from the g-truc 3.x suite) are pushed up to vogl_chroot. We also have many GB's of game traces (please drop me a message if you would like these traces). It's pretty easy to add your own traces - I'll be documenting how on vogl's wiki this afternoon.

Prerequisites:

Right now, we primarily test on NVidia's closed source driver. (The tests also run on AMD but testing there lags vs. NVidia.) Some game traces require your desktop vertical resolution to be greater than 1080, so either enable virtual desktop in the nvidia-settings control panel (or buy a 2560x1600 monitor).

The tests directory is located in the vogl_chroot repo. Pull both repos and build vogl.

How to run it:

Run tests/run_tests.sh and wait. I recommend you don't try to use the system until it's done because as new replayer instances are launched your focus will change around.

At the end of the test you'll see a summary like this:

g-truc1 : gl-320-fbo-blit32.trace (#7 2/5) Return: 256
g-truc2 : gl-330-blend-rtt32.trace (#38 2/5) Return: 256
g-truc2 : gl-330-fbo-multisample-explicit-nv32.trace (#41 2/5) Return: 256
MetroLL : MetroLL.trace (#56 2/5) Return: 256

384 commands launched.
4 errors.

Time: 1425.58s
Wrote logfile /tmp/vogltests.2014_04_22-17_16_45.log

Command line params:

/home/user/dev/vogl_chroot/tests/x86_64/run_tests --help
  -d, --vogltracedir=DIR     libvogltrace32.so directory (defaults to
                             ../vogl_build/bin).
  -f, --filename[=FILE]      Test filename (defaults to tests.json).
  -j, --jobs=JOBS            Allow N test jobs to run at once.
  -l, --logfile=LOGFILE      Logfile name.
  -p, --pattern=PATTERN      Test name pattern.
  -t, --list                 List tests in file.
  -v, --verbose              Produce verbose output.
  -y, --dry-run              Don't execute commands.
  -?, --help                 Give this help message.

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

How to add more traces to the test:

Place an apitrace .trace file of the GL callstream you want to add into the tests/traces directory. (You can also use vogl traces, but we're trying to purposely avoid doing this. Test coverage is better is using apitrace, apitrace can play back on multiple threads, and its trace format is relatively stable vs. vogl's.)

Now edit tests.json and add a new object like this:

"darwinia": {
    "window_width": 800,
    "window_height": 600,
    "comparison_sum_threshold": 4096,
    "comparison_frames_to_skip": 0,
    "trim_frame_start": 100,
    "trim_frame_count": 100,
    "trace_files": [
        "traces/darwinia_trace.bin"
    ]
}

Most fields are self-explanatory. Set window_width/window_height to the replayer window dimensions needed by the trace. GL traces which utilize multiple windows (or resize the window during playback) aren't a good fit for the current system.

One of the most important fields to get right is comparison_sum_threshold. If it's is 0, CRC-64 is used for divergence checking, otherwise the replayer uses a simple sum of per-component differences. Some GPU's are not deterministic when multisampling is used, so you should definitely set this value to a non-zero value if your trace utilizes multisampling. Also, we've found a few non-multisampled traces that require per-component differences for unknown reasons (they diverge during replaying but only in very minor ways, like in the backbuffer LSB's around edges). apitrace has the same issue so we do not think this is a problem with vogl or the driver. The only way to determine if your trace should use CRC-64's vs. thresholding is to try it.