From ec5dd120f9858e1d4113871a47098b50b67caa6e Mon Sep 17 00:00:00 2001 From: Hiroyasu Nishiyama Date: Mon, 13 Dec 2021 10:31:16 +0900 Subject: [PATCH] add implementation note of flow-testing --- designs/flow-testing/ImplementationNote.md | 170 +++++++++++++++++++++ designs/flow-testing/README.md | 42 +++-- 2 files changed, 197 insertions(+), 15 deletions(-) create mode 100644 designs/flow-testing/ImplementationNote.md diff --git a/designs/flow-testing/ImplementationNote.md b/designs/flow-testing/ImplementationNote.md new file mode 100644 index 0000000..b08d0b6 --- /dev/null +++ b/designs/flow-testing/ImplementationNote.md @@ -0,0 +1,170 @@ +# Implementation Note on Flow Testing Prototype + +## Summary + +This document describes the internal implementation of the prototype of flow-testing plugin for Node-RED as of December 2021. + +## Flow Testing Plugin + +### Initialization + +At the time of installation of the flow test plugin, the flow testing plugin registers the following: + +**On Runtime:** + +1. Event (`registry:plugin-added`) handler for monitoring plugin installation for test actions (see [below](#actions-extension)), + +2. Internal API endpoint (`/flow-tester/executeAction/:action`) for accepting instructions to execute test actions from the Node-RED editor, + +3. Internal API endpoint (`/flow-tester/testCase`) for getting information on currently defined test cases, + +4. Internal API endpoint (`/flow-tester/runTestCase/:suite/:test`) for accepting instructions to execute individual test case. + +**On Editor:** + +1. Message (`flow-test:log`) handler for displaying log message on flow testing sidebar, + +2. Message (`flow-test:notify`) handler for displaying log message on flow testing sidebar and showing notification message, + +3. Message (`flow-test:maxActions`) handler for notifying that actions execution has reached its limit, + +4. Message (`flow-test:clear-match-result`) handler for clearing expected number of checks, + +5. Message (`flow-test:click`) handler for clicking button of specified node (e.g. Inject node), + +6. Event (`registry:plugin-added`) handler for monitoring plugin installation for test actions (see [below](#actions-extension)), + +7. Sidebar for flow testing, + +8. Event (`nodes:added`, `nodes:remove`, and `nodes:change`) handler for monitoring addition/removal/change of flow test configuration (`flow-test-config` node), + +9. Event (`nodes:dialog-prepare`, `nodes:dialog-resize`, and `editor:save`) handler for preparing/resizing flow test tab of node settings panel and storing flow test configuration for the node (see [below](#flow-test-plugin-support-in-node-red-editor)), + +### Executing Test Actions + + Execution of test case is performed in the following order on editor side: + +1. Initialize test environment calling *init* action of runtime, + + - On editor side, + + - count number of checks in current test case, + + - this number is used for checking completion of the test case + + - On runtime side, + + - clear current actions list, + + - register a message router hook (`onReceive.flow-test`) for handling node `receive` event and executing corresponding actions, + + - register a message router hook (`preRoute.flow-test`) for handling node `stub` event and executing corresponding actions, + + - register a message router hook (`onSend.flow-test`) for handling node `send` event and executing corresponding actions. + +2. Register events and associated actions calling `registerActions` action of runtime, + + - On runtime side, + + - register events and associated actions information in current test in `actionMap` data. It maps *event* → *node id* → *array of action information*. The action information is a object consisting of action specific properties and following common properties: + + | name | type | description | + | ------- | ------ | ---------------- | + | index | number | index | + | suiteID | string | ID of test suite | + | testID | string | ID of test case | + +3. Execute `setup` events, + + - On runtime side, + + - process `setup` actions of test case. Test case specific actions are recorded in `actionMap` with a node id of `_global_`. + +4. Set timeout, + + - On editor side, + + - set timeout with timeout value specified for a test case, + + - if the timeout is reached or an error is occurred before completion of the test case, a log message is outputted and the test is cleaned up. + +5. Completion of the test execution + + - The action that validates the test (e.g. match) confirms the result and completes the test when all the checks are performed. + + - If number of executed checks reaches reaches number of checks in the test case counted by (1), test is completed and the result is reported. + +### Test Configuration + + Configuration of each test suite is stored as `flow-test-config` config node. It contains following properties: + +| name | type | description | +| ----------- | ------ | ----------------------------------- | +| timeout | number | timeout value in ms | +| max_actions | number | limit on number of executed actions | +| actions | object | array of action definitions | + +This configuration node is installed together with flow test plugin. Most part of settings UI of `flow-test-config` config node is generated by event handler of flow testing plugin. + +## Flow Test Plugin Support in Node-RED Editor + +Current flow tester implementation uses following additional events in this [repository]([GitHub - node-red-hitachi/node-red at flow-test-support](https://github.com/node-red-hitachi/node-red/tree/flow-test-support)) to Node-RED editor: + +- `nodes:dialog-prepare` - fired when preparing node setting panel with following option object as an argument: + + | name | type | description | + | ------- | ------ | ------------------------------------------- | + | node | object | target node | + | tabs | object | tab element of node settings panel | + | content | object | content part element of node settings panel | + + This is used for creating additional settings panel tab for flow testing for each node. + +- ` nodes:dialog-resize` - fired when node setting panel resized with following option object as an argument: + + | name | type | description | + | ---- | ------ | ----------------------------------------------------------------- | + | node | object | target node | + | size | object | object representing new size with `height` and `width` properties | + +- ` editor:save` - fired when node configuration is saved with target node as an argument. + +## Actions Extension + +Flow test actions can be added using plugin feature of Node-RED. Plugins should be initialized by `RED.plugins.registerPlugin` with option object on editor side and runtime side. + +On editor side option object consists of following properties: + +| name | type | description | +| -------- | -------- | -------------------------------------------------------- | +| type | string | constant string: "flow-tester-addon" | +| onadd | function | called on removal of plugin | +| onremove | function | called on removal of plugin | +| actions | function | function that returns array of action definition objects | + +Action definition object consists of following properties: + +| name | type | description | +| ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| name | string | internal name of action | +| label | string | menu label | +| onAddItem | function | called when adding this action to editableList with option object as an argument. The option object consists of:
- node: target node,
- container: container element of added item,
- selectRow: first row element of action added item,
- data: initial data | +| onSelect | function | called when this actionItem is selected with option object as an argument. The option object consists of:
- container: container element of added item,
- selectRow: first row element of action added item | +| onUnselect | function | called when this actionItem is unselected with option object as an argument. The option object consists of:
- container: container element of added item,
- selectRow: first row element of action added item | +| onSave | function | called when this actionItem is saved with its container element as an argument.  This function must return an object that represents action information.  It consists of following properties:
- value: value needed to execute action,
- performCheck: `true` if this action performs a check of test case | + +On runtime side option object consists of following properties: + +| name | type | description | +| -------- | -------- | ------------------------------------------------------------------ | +| type | string | constant string: "flow-tester-addon" | +| onadd | function | called on removal of plugin | +| onremove | function | called on removal of plugin | +| actions | function | function that returns array of action execution definition objects | + +Action definition object consists of following properties: + +| name | type | description | +| ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| name | string | internal name of action | +| execute | function | function to execute a test action. It returns Promise and takes an option object that consists of following properties:
- action: action definition object saved by onSave callback function,
- node: target node,
- msg: target message
If this function performs check, the returned Promise resolves when check is successful otherwise it rejects. | diff --git a/designs/flow-testing/README.md b/designs/flow-testing/README.md index 6238517..01dc958 100644 --- a/designs/flow-testing/README.md +++ b/designs/flow-testing/README.md @@ -219,11 +219,15 @@ Test events are defined for test case and test target nodes. List of actions are - setup: executed at start of the test case, + - success: executed after successful execution of the node, + - fail: executed after failed execution of the test case, - cleanup: executed at end of the test case, - timeout: executed after timeout reached for the test case. + + - exception: executed after occurrence of exception while executing the node, - Test case events - No events specific to a test case exists. @@ -236,16 +240,6 @@ Test events are defined for test case and test target nodes. List of actions are - stubbed execution: executed after *receive* event instead of message processing of the node, - send: executed after sending message from the node to output port, - - - success: executed after successful execution of the node, - - - fail: executed after failed execution of the node, - - - exception: executed after occurence of exception while executing the node, - - - cleanup: executed at end of the test case, - - - timeout: executed after timeout reached for the test case. ### Test Actions @@ -257,7 +251,9 @@ Test events are defined for test case and test target nodes. List of actions are - log - log message, - - function - write complex testing using JavaScript code. + - function - write complex testing using JavaScript code, + + - click - click button of specified node. - Test case actions @@ -271,8 +267,6 @@ Test events are defined for test case and test target nodes. List of actions are - send - send message to output port, - - pass - pass input message message to output port, - - catch - catch exception, - ... @@ -335,18 +329,36 @@ After starting Node-RED in the test mode, users can turn off/on the mode on the This flow testing needs to be run on the CLI for targeting to run automatically on Travis CI when receiving a pull request on GitHub. When running a command like `grunt test-flow`, Node-RED runs flow testing then outputs each result such as the existing Mocha tests. -[H.N.] Node-RED provides command line option for running test suitesfor CLI testing. +[H.N.] Node-RED provides command line option for running test suites for CLI testing. `-t test1,test2,...` runs test suites `test1`, `test2`, .... +## Extending Actions + +Flow testing supports addition of test actions using [plugin mechanism]([designs/plugins.md at master · node-red/designs · GitHub](https://github.com/node-red/designs/blob/master/designs/plugins.md)). It would be useful for adding actions: + +- testing files, + +- taking screenshot of Node-RED editor, + +- testing GUI interactions, + +- ... + ## Concerns - How to test dashboard nodes from flow testing? (it may be impossible) + → Use plugin support of actions for controlling GUI - We will add how to run on the CLI. (arguments, example, etc) +## Implementation Note + +Look over [here](ImplementationNote.md) for implementation notes. + ## History +- 2021-12-08 - Add extension mechanism and implementation note. Refine events definition. - 2021-05-28 - Updated proposal - 2020-07-17 - Add Concepts section - 2020-06-09 - Add requirements - 2020-04-13 - Updated proposal -- 2020-01-31 - Initial proposal### +- 2020-01-31 - Initial proposal