Skip to content

Commit

Permalink
Merge branch 'issue_438' into 'master'
Browse files Browse the repository at this point in the history
[test] Add test harness for UI tests

Closes Bitcoin-ABC#438

See merge request bitcoin-cash-node/bitcoin-cash-node!1623
  • Loading branch information
ftrader committed Nov 24, 2022
2 parents 05dad79 + 1f11ca8 commit ac17d9e
Show file tree
Hide file tree
Showing 6 changed files with 222 additions and 5 deletions.
3 changes: 2 additions & 1 deletion .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -718,7 +718,8 @@ test-aarch64-functional:
- export NON_TESTS="example_test|test_runner|combine_logs|create_cache"
- export LONG_TESTS="abc-p2p-compactblocks|abc-p2p-fullblocktest|feature_block|feature_dbcrash|feature_pruning|mining_getblocktemplate_longpoll|p2p_timeouts|wallet_backup"
- export EXCLUDED_TESTS="getblocktemplate_errors|getblocktemplate-timing"
- (cd build; test/functional/test_runner.py `ls -1 ../test/functional/*.py | xargs -n 1 basename | egrep -v "($NON_TESTS|$LONG_TESTS|$EXCLUDED_TESTS)"`)
- export UI_TESTS=`ls -1 test/functional/*.py | xargs -n 1 basename | grep ^ui | tr '\n' '|' | sed 's/|$//'`
- (cd build; test/functional/test_runner.py `ls -1 ../test/functional/*.py | xargs -n 1 basename | egrep -v "($NON_TESTS|$LONG_TESTS|$EXCLUDED_TESTS|$UI_TESTS)"`)
dependencies:
- build-aarch64
artifacts:
Expand Down
78 changes: 77 additions & 1 deletion doc/functional-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,14 @@ The [/test/](/test/) directory contains integration tests that test bitcoind
and its utilities in their entirety. It does not contain unit tests, which
can be found in [/src/test](/src/test), [/src/wallet/test](/src/wallet/test), etc.

There are currently two sets of tests in the [/test/](/test/) directory:
There are currently three sets of tests in the [/test/](/test/) directory:

- [functional](/test/functional) which test the functionality of
bitcoind and bitcoin-qt by interacting with them through the RPC and P2P
interfaces.
- [ui](/test/functional/ui*) which test the UI functionality of
bitcoin-qt by instructing the user to follow a test plan to
reproduce the state of bitcoin-qt graphical user interface.
- [util](/test/util) which tests the bitcoin utilities, currently only
bitcoin-tx.

Expand Down Expand Up @@ -192,6 +195,79 @@ variable `PYTHONWARNINGS` as follow:

The warning message will now be printed to the `sys.stderr` output.

### UI tests

UI tests are a subset of functional tests and can be scripted in a similar way.

They are meant to be a helper during development and a tool to achieve and
verify a certain state of bitcon-qt GUI.

These tests are not included into CI (continuous integration) suite or tests
ran when invoking `ninja check-all`.

All UI test file names shall have `ui_` or `ui-` prefixed. For example:
`ui-example-test.py`

UI framework's `RunTestPlan` function takes test steps and an instance of a test class implementing `BitcoinTestFramework`

It expands the instance and embeds a `testPlan` attribute into it to connect the test plan and the test.

Upon initializaion the `main` method of the test class will be called for the test execution.

Test class shall use `self.testPlan.waitUntilMaxReachedStep` to declare test breakpoints
and wait for user input to advance the test.

Test plan window will become irresponsive upon test logic execution and user must allow the test steps
to be finished before advancing to the next step

To implement an example UI test one can start with te following bolerplate code:

```python
from test_framework.test_framework import BitcoinTestFramework
from test_framework.ui import RunTestPlan

class UITestFramework(BitcoinTestFramework):
def set_test_params(self):
self.num_nodes = 1
self.extra_args = [["-splash=0", "-ui"]]

def run_test(self):
# self.meaninfulWork()
self.testPlan.waitUntilMaxReachedStep(1)

# self.moreMeaningfulWork()
self.testPlan.waitUntilMaxReachedStep(2)

if __name__ == '__main__':
steps = [{
'description': "Be a good guy"
}, {
'description': "Have a nice day"
}]

framework = UITestFramework()
RunTestPlan(steps, framework)
```

#### Dependencies

Running UI tests requires pyqt5 package:

- On Unix, run `sudo apt-get install python3-pyqt5`
- On mac OS, run `brew install pyqt5`

#### Running the tests

UI tests are meant to be executed individually:

```
test/functional/test_runner.py ui-example-test
```

#### Troubleshooting and debugging test failures

Same as for function tests.

### Util tests

Util tests can be run locally by running `test/util/bitcoin-util-test.py`.
Expand Down
15 changes: 14 additions & 1 deletion test/functional/test_framework/test_framework.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/usr/bin/env python3
# Copyright (c) 2014-2019 The Bitcoin Core developers
# Copyright (c) 2022 The Bitcoin Cash Node developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Base class for RPC testing."""
Expand Down Expand Up @@ -155,6 +156,13 @@ def main(self):
self.config = config
self.options.bitcoind = os.getenv(
"BITCOIND", default=config["environment"]["BUILDDIR"] + '/src/bitcoind' + config["environment"]["EXEEXT"])
if sys.platform == "darwin":
self.options.bitcoinqt = os.getenv(
"BITCOINQT", default=config["environment"]["BUILDDIR"] + '/src/qt/BitcoinCashNode-Qt.app/Contents/MacOS/BitcoinCashNode-Qt' + config["environment"]["EXEEXT"])
else:
self.options.bitcoinqt = os.getenv(
"BITCOINQT", default=config["environment"]["BUILDDIR"] + '/src/qt/bitcoin-qt' + config["environment"]["EXEEXT"])

self.options.bitcoincli = os.getenv(
"BITCOINCLI", default=config["environment"]["BUILDDIR"] + '/src/bitcoin-cli' + config["environment"]["EXEEXT"])
self.options.emulator = config["environment"]["EMULATOR"] or None
Expand Down Expand Up @@ -317,7 +325,12 @@ def add_nodes(self, num_nodes, extra_args=None,
if extra_args is None:
extra_args = [[]] * num_nodes
if binary is None:
binary = [self.options.bitcoind] * num_nodes
binary = []
# treat -ui as a special flag to choose binary
for args in extra_args:
binary.append(self.options.bitcoinqt if "-ui" in args else self.options.bitcoind)
# drop -ui from args
extra_args = [[arg for arg in args if arg != "-ui"] for args in extra_args]
assert_equal(len(extra_confs), num_nodes)
assert_equal(len(extra_args), num_nodes)
assert_equal(len(binary), num_nodes)
Expand Down
95 changes: 95 additions & 0 deletions test/functional/test_framework/ui.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWidgets import QMainWindow, QTextEdit, QGridLayout, QWidget, QHBoxLayout
from PyQt5.QtWidgets import QPushButton
from PyQt5.QtCore import QSize, Qt

from .util import wait_until

class MainWindow(QMainWindow):
def __init__(self, app, steps):
self.app = app
self.steps = steps
self.currentStep = 0
self.maxReachedStep = 0

QMainWindow.__init__(self)

screenrect = app.primaryScreen().geometry()
self.move(screenrect.left(), screenrect.top())
self.setMinimumSize(QSize(300, 260))
self.resize(QSize(300, 260))
self.setWindowTitle("Bitcoin Cash Node - UI Test Plan")
self.setWindowFlags(QtCore.Qt.WindowStaysOnTopHint)


self.horizontalGroupBox = QWidget(self)
self.horizontalGroupBox.resize(self.size())

layout = QGridLayout()
layout.setRowStretch(1, 1)

self.stepDescription = QTextEdit()
self.stepDescription.setText("Step 1/6")
self.stepDescription.setReadOnly(True)
# self.stepDescription.setStyleSheet("background: red")
layout.addWidget(self.stepDescription, 0,0,0,3, Qt.AlignTop)


buttons = QWidget()
hLayout = QHBoxLayout()

self.backButton = QPushButton('Back', self)
self.backButton.setEnabled(False)
self.backButton.clicked.connect(self.back)
hLayout.addWidget(self.backButton)

self.advanceButton = QPushButton('Advance', self)
self.advanceButton.clicked.connect(self.advance)
hLayout.addWidget(self.advanceButton)

buttons.setLayout(hLayout)
layout.addWidget(buttons,1,0,1,3, Qt.AlignBottom)

self.horizontalGroupBox.setLayout(layout)

self.setCentralWidget(self.horizontalGroupBox)

self.updateUI()

def updateUI(self):
step = self.steps[self.currentStep]

self.stepDescription.setText(f"Step {self.currentStep+1}/{len(self.steps)}\n{step['description']}")
self.backButton.setEnabled(self.currentStep > 0)
self.advanceButton.setText("Finish" if self.currentStep+1 == len(self.steps) else "Advance")

def advance(self):
self.currentStep += 1
if (self.currentStep > self.maxReachedStep):
self.maxReachedStep = self.currentStep

if (self.currentStep >= len(self.steps)):
self.close()
else:
self.updateUI()

def back(self):
if (self.currentStep > 0):
self.currentStep -= 1
self.updateUI()

def waitUntilMaxReachedStep(self, step):
def process():
self.app.processEvents()
return self.maxReachedStep >= step
wait_until(lambda: process(), timeout=1e10)

def RunTestPlan(steps, framework):
app = QtWidgets.QApplication([''])
mainWindow = MainWindow(app, steps)
framework.testPlan = mainWindow
mainWindow.show()
framework.main()
app.exec()
return mainWindow

6 changes: 4 additions & 2 deletions test/functional/test_runner.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
# Copyright (c) 2014-2019 The Bitcoin Core developers
# Copyright (c) 2017-2020 The Bitcoin developers
# Copyright (c) 2022 The Bitcoin Cash Node developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Run regression test suite.
Expand Down Expand Up @@ -268,7 +269,8 @@ def main():
else:
# No individual tests have been specified.
# Run all tests that do not exceed
test_list = all_scripts
# Skip ui tests (those with names starting with "ui")
test_list = [script for script in all_scripts if script[:2] != "ui"]

if args.extended:
cutoff = sys.maxsize
Expand Down Expand Up @@ -645,7 +647,7 @@ def check_script_prefixes(all_scripts):
LEEWAY = 10

good_prefixes_re = re.compile(
"(bchn[_-])?(example|feature|interface|mempool|mining|p2p|rpc|wallet|tool)[_-]")
"(bchn[_-])?(example|feature|interface|mempool|mining|p2p|rpc|wallet|tool|ui)[_-]")
bad_script_names = [
script for script in all_scripts if good_prefixes_re.match(script) is None]

Expand Down
30 changes: 30 additions & 0 deletions test/functional/ui-example-test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/usr/bin/env python3
# Copyright (c) 2021-2022 The Bitcoin Cash Node developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
""" Example UI test plan boilerplate code """

from test_framework.test_framework import BitcoinTestFramework
from test_framework.ui import RunTestPlan

class UITestFramework(BitcoinTestFramework):
def set_test_params(self):
self.num_nodes = 1
self.extra_args = [["-splash=0", "-ui"]]

def run_test(self):
# self.meaninfulWork()
self.testPlan.waitUntilMaxReachedStep(1)

# self.moreMeaningfulWork()
self.testPlan.waitUntilMaxReachedStep(2)

if __name__ == '__main__':
steps = [{
'description': "Be a good guy"
}, {
'description': "Have a nice day"
}]

framework = UITestFramework()
RunTestPlan(steps, framework)

0 comments on commit ac17d9e

Please sign in to comment.