Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add implementation note of flow-testing prototype #65

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add implementation note of flow-testing prototype #65

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
170 changes: 170 additions & 0 deletions designs/flow-testing/ImplementationNote.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# Implementation Note on Flow Testing Prototype

## Summary

This document describes the internal implementation of the prototype of flow-testing plugin for Node-RED as of December 2021.

## Flow Testing Plugin

### Initialization

At the time of installation of the flow test plugin, the flow testing plugin registers the following:

**On Runtime:**

1. Event (`registry:plugin-added`) handler for monitoring plugin installation for test actions (see [below](#actions-extension)),

2. Internal API endpoint (`/flow-tester/executeAction/:action`) for accepting instructions to execute test actions from the Node-RED editor,

3. Internal API endpoint (`/flow-tester/testCase`) for getting information on currently defined test cases,

4. Internal API endpoint (`/flow-tester/runTestCase/:suite/:test`) for accepting instructions to execute individual test case.

**On Editor:**

1. Message (`flow-test:log`) handler for displaying log message on flow testing sidebar,

2. Message (`flow-test:notify`) handler for displaying log message on flow testing sidebar and showing notification message,

3. Message (`flow-test:maxActions`) handler for notifying that actions execution has reached its limit,

4. Message (`flow-test:clear-match-result`) handler for clearing expected number of checks,

5. Message (`flow-test:click`) handler for clicking button of specified node (e.g. Inject node),

6. Event (`registry:plugin-added`) handler for monitoring plugin installation for test actions (see [below](#actions-extension)),

7. Sidebar for flow testing,

8. Event (`nodes:added`, `nodes:remove`, and `nodes:change`) handler for monitoring addition/removal/change of flow test configuration (`flow-test-config` node),

9. Event (`nodes:dialog-prepare`, `nodes:dialog-resize`, and `editor:save`) handler for preparing/resizing flow test tab of node settings panel and storing flow test configuration for the node (see [below](#flow-test-plugin-support-in-node-red-editor)),

### Executing Test Actions

Execution of test case is performed in the following order on editor side:

1. Initialize test environment calling *init* action of runtime,

- On editor side,

- count number of checks in current test case,

- this number is used for checking completion of the test case

- On runtime side,

- clear current actions list,

- register a message router hook (`onReceive.flow-test`) for handling node `receive` event and executing corresponding actions,

- register a message router hook (`preRoute.flow-test`) for handling node `stub` event and executing corresponding actions,

- register a message router hook (`onSend.flow-test`) for handling node `send` event and executing corresponding actions.

2. Register events and associated actions calling `registerActions` action of runtime,

- On runtime side,

- register events and associated actions information in current test in `actionMap` data. It maps *event* → *node id* → *array of action information*. The action information is a object consisting of action specific properties and following common properties:

| name | type | description |
| ------- | ------ | ---------------- |
| index | number | index |
| suiteID | string | ID of test suite |
| testID | string | ID of test case |

3. Execute `setup` events,

- On runtime side,

- process `setup` actions of test case. Test case specific actions are recorded in `actionMap` with a node id of `_global_`.

4. Set timeout,

- On editor side,

- set timeout with timeout value specified for a test case,

- if the timeout is reached or an error is occurred before completion of the test case, a log message is outputted and the test is cleaned up.

5. Completion of the test execution

- The action that validates the test (e.g. match) confirms the result and completes the test when all the checks are performed.

- If number of executed checks reaches reaches number of checks in the test case counted by (1), test is completed and the result is reported.

### Test Configuration

Configuration of each test suite is stored as `flow-test-config` config node. It contains following properties:

| name | type | description |
| ----------- | ------ | ----------------------------------- |
| timeout | number | timeout value in ms |
| max_actions | number | limit on number of executed actions |
| actions | object | array of action definitions |

This configuration node is installed together with flow test plugin. Most part of settings UI of `flow-test-config` config node is generated by event handler of flow testing plugin.

## Flow Test Plugin Support in Node-RED Editor

Current flow tester implementation uses following additional events in this [repository]([GitHub - node-red-hitachi/node-red at flow-test-support](https://github.com/node-red-hitachi/node-red/tree/flow-test-support)) to Node-RED editor:

- `nodes:dialog-prepare` - fired when preparing node setting panel with following option object as an argument:

| name | type | description |
| ------- | ------ | ------------------------------------------- |
| node | object | target node |
| tabs | object | tab element of node settings panel |
| content | object | content part element of node settings panel |

This is used for creating additional settings panel tab for flow testing for each node.

- ` nodes:dialog-resize` - fired when node setting panel resized with following option object as an argument:

| name | type | description |
| ---- | ------ | ----------------------------------------------------------------- |
| node | object | target node |
| size | object | object representing new size with `height` and `width` properties |

- ` editor:save` - fired when node configuration is saved with target node as an argument.

## Actions Extension

Flow test actions can be added using plugin feature of Node-RED. Plugins should be initialized by `RED.plugins.registerPlugin` with option object on editor side and runtime side.

On editor side option object consists of following properties:

| name | type | description |
| -------- | -------- | -------------------------------------------------------- |
| type | string | constant string: "flow-tester-addon" |
| onadd | function | called on removal of plugin |
| onremove | function | called on removal of plugin |
| actions | function | function that returns array of action definition objects |

Action definition object consists of following properties:

| name | type | description |
| ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | string | internal name of action |
| label | string | menu label |
| onAddItem | function | called when adding this action to editableList with option object as an argument. The option object consists of:<br/>- node: target node,<br/>- container: container element of added item,<br/>- selectRow: first row element of action added item,<br/>- data: initial data |
| onSelect | function | called when this actionItem is selected with option object as an argument. The option object consists of:<br/>- container: container element of added item,<br/>- selectRow: first row element of action added item |
| onUnselect | function | called when this actionItem is unselected with option object as an argument. The option object consists of:<br/>- container: container element of added item,<br/>- selectRow: first row element of action added item |
| onSave | function | called when this actionItem is saved with its container element as an argument.  This function must return an object that represents action information.  It consists of following properties:<br/>- value: value needed to execute action,<br/>- performCheck: `true` if this action performs a check of test case |

On runtime side option object consists of following properties:

| name | type | description |
| -------- | -------- | ------------------------------------------------------------------ |
| type | string | constant string: "flow-tester-addon" |
| onadd | function | called on removal of plugin |
| onremove | function | called on removal of plugin |
| actions | function | function that returns array of action execution definition objects |

Action definition object consists of following properties:

| name | type | description |
| ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | string | internal name of action |
| execute | function | function to execute a test action. It returns Promise and takes an option object that consists of following properties:<br/>- action: action definition object saved by onSave callback function,<br/>- node: target node,<br/>- msg: target message<br/>If this function performs check, the returned Promise resolves when check is successful otherwise it rejects. |
42 changes: 27 additions & 15 deletions designs/flow-testing/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,11 +219,15 @@ Test events are defined for test case and test target nodes. List of actions are

- setup: executed at start of the test case,

- success: executed after successful execution of the node,

- fail: executed after failed execution of the test case,

- cleanup: executed at end of the test case,

- timeout: executed after timeout reached for the test case.

- exception: executed after occurrence of exception while executing the node,

- Test case events - No events specific to a test case exists.

Expand All @@ -236,16 +240,6 @@ Test events are defined for test case and test target nodes. List of actions are
- stubbed execution: executed after *receive* event instead of message processing of the node,

- send: executed after sending message from the node to output port,

- success: executed after successful execution of the node,

- fail: executed after failed execution of the node,

- exception: executed after occurence of exception while executing the node,

- cleanup: executed at end of the test case,

- timeout: executed after timeout reached for the test case.

### Test Actions

Expand All @@ -257,7 +251,9 @@ Test events are defined for test case and test target nodes. List of actions are

- log - log message,

- function - write complex testing using JavaScript code.
- function - write complex testing using JavaScript code,

- click - click button of specified node.

- Test case actions

Expand All @@ -271,8 +267,6 @@ Test events are defined for test case and test target nodes. List of actions are

- send - send message to output port,

- pass - pass input message message to output port,

- catch - catch exception,

- ...
Expand Down Expand Up @@ -335,18 +329,36 @@ After starting Node-RED in the test mode, users can turn off/on the mode on the
This flow testing needs to be run on the CLI for targeting to run automatically on Travis CI when receiving a pull request on GitHub.
When running a command like `grunt test-flow`, Node-RED runs flow testing then outputs each result such as the existing Mocha tests.

[H.N.] Node-RED provides command line option for running test suitesfor CLI testing.
[H.N.] Node-RED provides command line option for running test suites for CLI testing.
`-t test1,test2,...` runs test suites `test1`, `test2`, ....

## Extending Actions

Flow testing supports addition of test actions using [plugin mechanism]([designs/plugins.md at master · node-red/designs · GitHub](https://github.com/node-red/designs/blob/master/designs/plugins.md)). It would be useful for adding actions:

- testing files,

- taking screenshot of Node-RED editor,

- testing GUI interactions,

- ...

## Concerns

- How to test dashboard nodes from flow testing? (it may be impossible)
→ Use plugin support of actions for controlling GUI
- We will add how to run on the CLI. (arguments, example, etc)

## Implementation Note

Look over [here](ImplementationNote.md) for implementation notes.

## History

- 2021-12-08 - Add extension mechanism and implementation note. Refine events definition.
- 2021-05-28 - Updated proposal
- 2020-07-17 - Add Concepts section
- 2020-06-09 - Add requirements
- 2020-04-13 - Updated proposal
- 2020-01-31 - Initial proposal###
- 2020-01-31 - Initial proposal