Skip to content
forked from bscotch/olympus

Testing framework for GMS 2.3.1+ projects with useful features.

License

Notifications You must be signed in to change notification settings

shichen85/olympus

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Olympus (GameMaker Studio 2 Testing Framework) Logo

Olympus

âš  This project is still in active development, and APIs are subject to change. âš 

Testing framework for GMS 2.3.1+ projects with useful features:

  • Record Keeping - Test results are recorded and json exportable
  • Crash Recovery - Resume progress after runner crashes
  • Async Support - Easily test async events
  • Tester Feedback - Easily compose instruction for testers and gather feedback
  • Flexible - Use any assertion library you want
  • Powerful Customization - Supports life cycle hooks, global options, and local options

Olympus is developed by Butterscotch Shenanigans ("Bscotch"). Check out our other project Ganary, which uses Olympus to run regression tests for GameMaker Runtime.

GameMaker Studio 2® is the property of Yoyo Games™. Butterscotch Shenanigans® and Olympus are not affiliated with Yoyo Games.

Quick start

Import the resources in the "Olympus" group, such as using Stitch:

stitch merge --source-github=bscotch/olympus --if-folder-matches=Olympus

Compose your test suite:

//Name your test suite
olympus_run("my suite name", function(){
  //Add a unit test
  olympus_add_test(    
    //Name your unit test
    "my unit test name",     
    //Define the test assertion logic 
    function(){
      var expected = "2";
      var actual = "1";
      if (actual != expected){
      throw({
        message: "Expected: " + expected + ". Actual: " + actual, 
        stacktrace: debug_get_callstack()
      });
    }     
  });  
});

Code example

Test record is written to file in the Save Area and a summary is shown in IDE output:

------------------------- 
passed: 0 
failed: 1 
skipped: 0 
crashed: 0 
Record written to file as Olympus_records/my_suite_name.olympus.json 
------------------------- 

Table of Contents

Full API Reference

Full API Reference is shown in the olympus_external_api script resource

Accessing Test Data with Hooks

Code example

Suite Summary

The test suite summary data is a GML struct with the following shape:

{
  "tallies": { //The tallies of unit tests results
    "skipped": 0,
    "crashed": 0,
    "total": 1,
    "passed": 0,
    "failed": 1
  },
  "name": "my_suite_name", //The suite name defined by `olympus_run(suite_name)`
  "tests": [ 
    //Array of unit test summaries. See [Unit Test Summary](#unit-test-summary)
  ]
}

You can access the most up to date version of this data through olympus_get_current_suite_summary(). You can also access this data at the start and the finish of a test suite by olympus_add_hook_before_suite_start() and olympus_add_hook_after_suite_finish().

On Suite Start

You can set a function be executed right before a test suite starts with olympus_add_hook_before_suite_start().

The entire suite summary is passed to the function, so you can do something like iterating through all the added tests and announcing their names:

olympus_add_hook_before_suite_start(function(suite_summary){
  show_debug_message("This suite contains the following tests:")
  var tests = suite_summary.tests;
  for (var i = 0; i < array_length(tests); i++){
    var this_test = tests[i];
    show_debug_message(this_test.name);
  }
})

On Suite Finish

You can set a function be executed after a test suite finishes with olympus_add_hook_after_suite_finish().

The entire suite summary is also passed to the function, so you can do something like annoucing the tallies of test results:

olympus_add_hook_after_suite_finish(function(suite_summary){
  show_debug_message("Test completed.")
  var tallies = suite_summary.tallies;
  show_debug_message("total: " + tallies.total);
  show_debug_message("skipped: " + tallies.skipped);
  show_debug_message("crashed: " + tallies.crashed);
  show_debug_message("passed: " + tallies.passed);
  show_debug_message("failed: " + tallies.failed);
})

Unit Test Summary

The unit test summary is the elements in the suite summary struct's tests array, which is also accessible by calling the olympus_get_current_test_summaries() function.

{
  "index": 0, //The unit test's index as the nth element of the suite's `tests` array 
  "name": "my unit test name", //The unit test name defined by `olympus_add_*(name)`
  "status": "failed", //The unit test result 
  "millis": 4, //The time span of the unit test in milliseconds
  "err": { //The error struct if unit the test did not pass
    "message": "Expected: 2. Actual: 1",
    "stacktrace": [
      "demo_olympus_quick_start:10",
      "_olympus_internal:485",
      "_olympus_async_test_controller_Step_0:18",
    ]
  }
}

You can access this data at the start and the finish of each unit test by olympus_add_hook_before_each_test_start() and olympus_add_hook_after_each_test_finish().

Once you get a hold of the unit test summary struct, you can use the convenience function olympus_get_test_status() to access the status variable and olympus_get_test_name() to access the name variable.

On Test Start

You can set a function be executed before each unit test starts with olympus_add_hook_before_each_test_start().

The unit test summary is passed to the function, so you can do something like announcing the name of the unit test:

olympus_add_hook_before_each_test_start(function(unit_summary){
  show_debug_message("Start testing: " + unit_summary.name)
})

On Test Finish

You can set a function be executed after each unit test finishes with olympus_add_hook_after_each_test_finish().

The unit test summary is also passed to the function, so you can do something like logging the error of the unit test if it did not pass:

olympus_add_hook_after_each_test_finish(function(unit_summary){
  if (unit_summary.status != olympus_test_status.passed){
    show_debug_message(unit_summary.err);
  }
})

Async Testing

Background

GML's async events are mediated through objects. Taking http_get() as an example, we need some sort of mediator objects (let's call it obj_http_mediator) whose Async HTTP Event gives us the access to async_load when http_get() finally resolves:

///Pt 1
///obj_http_mediator Create Event
http_handle = http_get("https://google.com")

///obj_http_mediator Async HTTP Event
if async_load[?"id"] == http_handle{
  show_debug_message("http status is: " + string(async_load[?"http_status"]) ) 
}

With GML 2.3.1+, we can store script functions in variables and execute the functions by "calling" the variables:

///Pt 2
handler_function = function(the_async_load){
  show_debug_message("http status is: " + string(the_async_load[?"http_status"]) )
}
handler_function(async_load); 

This language feature allows us to flexibly define what obj_http_mediator does with async_load. We start by storing the handler function into the instance variable handler_function:

///Pt 3
///Create Event
handler_function = function(){}
http_handle = http_get("https://google.com")

///Async HTTP Event
if async_load[?"id"] == http_handle{
  //Parse the async_load, such as reading async_load [? "result"]; 
  handler_function(async_load);
}

When we spawn obj_http_mediator, we can reassign the variable handler_function to a new function:

///Pt 4
with instance_create_depth(0,0,0,demo_obj_http_mediator){
  var new_handler_function = function(async_load_from_mediator){
    show_debug_message("http status is: "+string(async_load_from_mediator[?"http_status"])) 
  }  
  handler_function = new_handler_function;
}

Testing Async with Olympus

Async Testing code example

Once you set up an obj_http_mediator as shown above, you can test http_get() with Olympus by following these steps:

  1. Wrap your async mediator object spawning logic in a function, and make sure that this function returns the mediator instance ID:
///Pt 1
var mediator_spawning_logic = function(){
    return instance_create_depth(0,0,0,obj_http_mediator)
}
  1. Define your new_handler_function of how to handle the async_load. Note because Olympus packages the original async_load into the argument array, you have to retrieve it as the 0th element of the array:
///Pt 2
var new_handler_function = function(argument){
    var async_load_from_mediator = argument[0];
    var http_status = async_load_from_mediator[?"http_status"];
    if (http_status == 200){
        show_debug_message("Pinging Google succeeded.");
    }
    else{
        throw("Expected 200. Got: " + string(http_status));
    }    
}
  1. Let Olympus know the instance variable name of the handler function by constructing an options struct that has the variable name resolution_callback_name
///Pt 3
var options_to_register_handler_function_name = {
  resolution_callback_name: "handler_function"
}
  1. Pass all these to olympus_add_async_test():
///Pt 4
olympus_add_async_test("Test Pinging Google", mediator_spawning_logic, new_handler_function, options_to_register_handler_function_name);
  1. Wrap all of these inside the olympus_run() block:
///Pt 5
olympus_run("My Suite Name", function(){
  //Define the logic to spawn the async mediator object and return its instance ID
  var mediator_spawning_logic = function(){
      return instance_create_depth(0,0,0,obj_http_mediator)
  }

  //Define your new_handler_function
  var new_handler_function = function(response_array){
      var async_load_from_mediator = response_array[0];
      var http_status = async_load_from_mediator[?"http_status"];
      if (http_status == 200){
          show_debug_message("Pinging Google succeeded.");
      }
      else{
          throw("Expected 200. Got: " + string(http_status));
      }    
  }

  //Register the mediator object's instance variable name of the handler function
  var options_to_register_handler_function_name = {
    resolution_callback_name: "handler_function"
  }

  //Add the test as an async test to the suite
  olympus_add_async_test("Test Pinging Google", mediator_spawning_logic, new_handler_function, options_to_register_handler_function_name); 
});

Olympus will run all your added async tests sequentially, wait for each one to resolve before moving on to the next one.

Testing Async with User Feedback

Code example

For tasks such as confirming graphics/audio rendering, it may be difficult to verify with assertion logic. You can use olympus_add_async_test_with_user_feedback() to render the effect, serve a text propmt to the user, and let them decide whether the test passed or not.

The prompt uses the cross-platform supported get_string_async() method, which allows the user to pass or fail the test:

prompt

Advanced Use Cases

All the examples can be selected in the demo room creation code and run in the IDE with the demo config. You can only run one demo at a time as Olympus does not support concurrent suite running.

Async Handler Function Name

When adding async tests, Olympus needs to know the mediator object's instance variable name for the function that handles the async result. There are 3 ways to make the names known to Olympus:

Defining Through Test Options

As shown in step 5 of Testing Async with Olympus, we passed an options struct with the variable name resolution_callback_name to olympus_add_async_test to inform Olympus what the instance variable name is for the handler function:

var options_to_register_handler_function_name = {
  resolution_callback_name: "handler_function"
}

olympus_add_async_test(..., options_to_register_handler_function_name); 

Defining Through Suite Options

Code example

If all of your mediator objects use the same instance variable name for their async handler function, you can pass an options struct with the variable name global_resolution_callback_name to olympus_run to make that name known to Olympus:

var options_to_register_global_handler_function_name = {
  global_resolution_callback_name: "handler_function"
}

olympus_run(..., options_to_register_global_handler_function_name); 

global_resolution_callback_name is set to "callback" by default, so if your mediator objects already use that name, you do not need to override the default.

NOTE: Each test's own resolution_callback_name option will take precedence to the suite's global_resolution_callback_name option.

Using olympus_test_resolve

olympus_test_resolve is a syntactic sugar that saves the hassel of having to define the resolution_callback_name or global_resolution_callback_name options. Taking the earlier obj_http_mediator example in the Background section, instead of:

///Async HTTP Event
if async_load[?"id"] == http_handle{
  handler_function(async_load);
}

You can just have:

///Async HTTP Event
if async_load[?"id"] == http_handle{ 
  handler_function(async_load);
  //`handler_function` must not mutate the content of `async_load`
  olympus_test_resolve(async_load);
}

Behind the scenes, olympus_test_resolve calls a function whose name is already known to Olympus, so you don't have to define the resolution_callback_name or global_resolution_callback_name options.

NOTE: The best practice is to make a copy of async_load to be passed to olympus_test_resolve so that we don't have to worry about olympus_test_resolve and handler_function interfere with each other.

Recovering from Crashes

Code example

Because the runner has to exit after uncaught exception occurs, a suite of tests are not guaranteed to complete if a particular test unit throws an uncaught exception or silently crashes.

Olympus deals with this by keeping track of the last test unit status and writing the progress to file. Upon crash and reboot, this allows the runner to unstuck itself by identifying the last running unit as the crash cause and skipping it to complete the test suite.

To enable this behavior, create an options struct with the variable name resume_previous_record and set it to true, and pass it to olympus_run():

var options_to_enable_crash_recovery = {
  resume_previous_record: true
}

olympus_run(..., options_to_enable_crash_recovery); 

Passing Variables Between Unit Tests

Default Context

Code example

Sometimes we may want to pass shared variables between unit tests. This is doable as all the unit tests within olympus_run() have access to the same scope by default, so you can do something like this:

shared_variable_sum = 0;
olympus_run("shared variables test", function(){
  olympus_add_test("sum should be 1", function(){
    shared_variable_sum ++;
    show_debug_message(string(shared_variable_sum)); //1
  });
  
  olympus_add_test("sum should be 2", function(){
    shared_variable_sum ++;
    show_debug_message(string(shared_variable_sum)); //2
  })
})

Custom Context

Code example

Alternatively, you can explicitly define what variables the tests should have access to by passing the options struct with the variable olympus_suite_options_context that points to a struct:

not_explicitly_defined_variable = "goodbye";
olympus_run("shared variables from custom context test", function(){
  olympus_add_test("", function(){
    show_debug_message(explicitly_shared_variable); 
  show_debug_message(not_explicitly_defined_variable); //Variable struct.not_explicitly_defined_variable not set before reading it. 
  });
}, {
  olympus_suite_options_context: {
      explicitly_shared_variable : "hello"
  }
})

Setting Up Dependency Chains

Sometimes when an earlier unit test fails, we want to skip later unit tests. This can be done in 3 ways:

Bail

Code example

By passing an options struct as {bail_on_fail_or_crash: true} to olympus_run(), any unit test that fails or crashes will cause the rest of the unit tests to be skipped.

Defining Dependency by Names

Code example

By passing an options struct as {dependency_names: ["test_name1", "test_name2"]} to any of the olympus_add*() APIs, the unit test will be skipped if any of its dependencies did not pass.

Creating a Dependency Chain

Code example

Unit tests added between olympus_test_dependency_chain_begin() and olympus_test_dependency_chain_end() will be treated as sequentially dependent on each other, while tests outside of the chain are not affected.

Options

Global Options

You can construct an options struct and pass to olympus_run() that will affect the test suite's global behavior across all unit tests:

Name Type Default Description
[olympus_suite_options_resume_previous_record] boolean false Enabling this starts the suite from wherever the last run left off. Otherwise, the suite starts from the beginning.
[olympus_suite_options_skip_user_feedback_tests] boolean false Enabling this skips tests that requires user feedback.
[olympus_suite_options_suppress_debug_logging] boolean false Enabling this suppresses Olympus from logging to the IDE Output tab.
[olympus_suite_options_test_interval_milis] number 0 Adds a delay between each test. Useful if you want to allow an audio or a visual cue to be played between tests.
[olympus_suite_options_global_resolution_callback_name] string "callback" Name of the instance variable for the resolution callback for all the mediator objects
[olympus_suite_options_global_rejection_callback_name] string "reject" Name of the instance variable for the rejection callback for all the mediator objects
[olympus_suite_options_bail_on_fail_or_crash] boolean false Enabling this will skip the rest of the tests if an earlier test fails or crashes
[olympus_suite_options_context] struct The binding context for function_to_add_tests_and_hooks. The default uses the calling context.
[olympus_suite_options_global_timeout_milliseconds] number 60000 If any test is not able to resolve within this many milliseconds, the test will be failed.
[olympus_suite_options_allow_uncaught] boolean false By default, Olympus catches uncaught error and record it. Enabling this allows uncaught error to be thrown instead and will stop recording test summaries or resuming unfinished records.
[olympus_suite_options_ignore_if_completed] boolean false Enabling this will ignore re-running the suite if the suite has been completed previously.

Test Options

You can construct an options struct and pass to the olympus_add* APIs that will affect that specific unit test's behavior:

Name Type Default Description
[olympus_test_options_resolution_callback_name] string If you have not defined a global_resolution_callback_name or want to overwrite that, specify it here
[olympus_test_options_rejection_callback_name] string If you have not defined a global_rejection_callback_name or want to overwrite that, specify it here
[olympus_test_options_dependency_names] string | string[] Names of tests whose failure will cause this test to be skipped
[olympus_test_options_context] struct The binding context for function_to_execute_synchronous_logic or function_to_spawn_object. The default uses the calling context.
[olympus_test_options_resolution_context] struct The binding context for function_to_execute_at_resolution. The default uses the calling context.
[olympus_test_options_timeout_milliseconds] number 60000 If this test is not able to resolve within this many milliseconds, the test will be failed.

Caveat

  • Olympus uses exception_unhandled_handler() to log uncaught errors. If you also uses exception_unhandled_handler(), make sure to re-assign your error handler function after the Olympus test suites conclude.
  • All unit tests must have unique names to support the dependency chaining. If you named two tests with the same name, the runner should throw an error on boot.

Contributing

Please make sure to read the Contributing Guide before making a pull request.

About

Testing framework for GMS 2.3.1+ projects with useful features.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Game Maker Language 100.0%