E2E tests setup and initial tests #1417

mnzaki · 2019-08-20T17:06:07Z

Resolves #1415

~~This builds on #1410 which must be merged first~~

NOTES

Android minSdkVersion bumped to 18
iOS not tested and may require more setup
No release build testing yet
yarn e2e:android to run the tests
yarn e2e:android -r to reuse last app installation and prevent detox from reinstalling
yarn e2e:android -r recovery to run the tests matching "recovery" (this is somewhat broken)

android/app/build.gradle

android/build.gradle

- It needs to be transpiled, so was excepted from the node_modules in transformIgnorePatterns - The global window variable had to be removed because it was breaking this code: https://github.com/typeorm/typeorm/blob/e4de0403e1f666639c0198e216409f997121c1e8/src/platform/BrowserPlatformTools.template#L20

- Android Notes - increases android's minSdkVersion to 18 - adds kotlin support - disables split ABI APKs for debug builds - Yarn Scripts - test:e2e:android to run on device - test:e2e:android:emu to run on emulator

So we don't have to hardcode devices/emulators in package.json This is somewhat hacky, there are dummy ones there or else detox will complain. But the device name keys are only generated last second before detox.init in e2e/init.js

until wix/Detox#445 is handled

DefinitelyTyped PR for missing testID props was merged

this includes some fixes we need, like android connectivity issue from wix/Detox#1601

failures due to misconfiguration (or random node module breakages) result in a whole bunch of unrelated/confusing failures in the tests, and there's no point of continuing tests if there's no detox anyway

react-devtools --depends-on--> electron -> electron-download -> fs-extra 0.3 for some reason detox loads fs-extra 0.3, though it depends on ^4.0.2 and that's why we can't have nice things

- find Text element by type - try to also find a Text child of the given testID - handle matching multiple children and read each recursively

Conflicts: src/ui/registration/components/entropy.tsx

saifahn

Android Emulator

I start up an emulator and then run:

yarn start

# separate terminal
yarn run e2e:android:emu

BUILD SUCCESSFUL

01_landing.spec.ts passes

03_recovery.spec.ts and 02_creation.spec.ts fail to run with ReferenceError: device is not defined.

The others fail as they have no tests.

Android Device

yarn start

yarn run e2e:android

BUILD SUCCESSFUL

then same as above. 01 passes but the other two say that device is not defined from the e2e/utils.ts file

I assume this isn't as intended?

I'm going to see if I can get this to run on iOS now.

saifahn · 2019-09-13T14:51:29Z

iOS Device

I ran detox build -c ios.sim.debug && detox test -c ios.sim.debug and after the dev bundle loaded onto the simulator, it gave the same as above. The 01_landing_spec passed but the others had the device ReferenceError.

There should also be some documentation somewhere regarding the pitfalls - e.g. needing the bundle server started and having the bundle fully loaded before running the tests.

mnzaki · 2019-09-13T14:55:19Z

There's also a conflict with the new entropy screen, because the infinite loop animation hangs detox :/

can't access devices unless detox.init is called

Conflicts: src/ui/recovery/components/inputSeedPhrase.tsx

saifahn · 2019-09-13T15:12:20Z

I can confirm the hanging on the entropy screen.

Next is:

Input Seed Phrase Screen
✓ should show a recoveryMsg (145ms)
✓ should show a seedWordFld (1853ms)
✓ should show word suggestions based on input prefix (8284ms)
✕ should add words to seed phrase on tap (2607ms)
✕ should show a restoreAccount button (35ms)
✕ should navigate home after a successful restore (10007ms)

'Multiple UI elements matched for given criteria.' as part of 'should add words to seed phrase on tap' test

At e2e/utils.ts:42

try {
   //console.error('expect with child')
   await expect(el).toBeVisible()
} catch (err) {

will need later for REPL

because it's an infinite loop animation using callbacks, always keeping the UI module busy, so detox need to be told to ignore that. Spoilers: it still doesn't work because the android disableSynchronization support is broken

this has a branch merged from leanmazzu/Detox.git with unreleased support for disableSynchronization now the e2e/01_new_user/02_creation test should be able to work around the hand animation screen Spoilers: it can't, the loop is never broken....

mnzaki · 2019-09-17T11:48:06Z

There's also a conflict with the new entropy screen, because the infinite loop animation hangs detox :/

I'm using a fork of detox with an unreleased branch to support proper disabling of synchronization, so we can avoid the infinite animation issue.

Another issue with the new entropy screen was the the animation loop would continue even after the animation component was unmounted, so even in the step after the entropy collection detox would hang (and presumably would hang forever if the hand animation is ever rendered)

These issues have been addressed in the last couple commits

Clearing the timeout is not useful, it will have already fired. The problem is with the recursive callbacks, which must be broken with a flag

mnzaki · 2019-09-17T13:00:32Z

I have moved the detox fork to the @jolocom organization and updated package.json.
Note that the fork contains a build of the branch https://github.com/jolocom/Detox/tree/jolocom on the tag detox-v9000.0.1-gitpkg using gitpkg.

The jolocom branch currently only has this PR merged in wix/Detox#1521

Exulansis · 2019-09-26T08:31:17Z

Tested as well, this looks good, outside of the merge conflicts I think this can be merged.

An extra issue should be created to add documentation on how to run this. I found the comments left by @saifahn really helpful.

Exulansis

LGTM, we should update the README with instructions for running these.

Conflicts: package.json yarn.lock

Conflicts: yarn.lock

VolkerSchiewe reviewed Aug 21, 2019

View reviewed changes

android/app/build.gradle Show resolved Hide resolved

android/build.gradle Show resolved Hide resolved

mnzaki added 16 commits September 11, 2019 13:23

fix: error because of missing type in src/config.ts

8a44324

chore(e2e): add and configure Detox e2e testing framework

68618a9

- Android Notes - increases android's minSdkVersion to 18 - adds kotlin support - disables split ABI APKs for debug builds - Yarn Scripts - test:e2e:android to run on device - test:e2e:android:emu to run on emulator

chore(e2e): auto-configure device/emulator names

fcb1c5f

So we don't have to hardcode devices/emulators in package.json This is somewhat hacky, there are dummy ones there or else detox will complain. But the device name keys are only generated last second before detox.init in e2e/init.js

chore(e2e): update jest config

aa6e4d5

chore(e2e): add utils.readVisibleText

02cacad

until wix/Detox#445 is handled

chore(e2e): add some comments

3d3e1cc

chore: update @types/react-native-material-ui

4c12bea

DefinitelyTyped PR for missing testID props was merged

chore: update detox and @types/detox

7827496

this includes some fixes we need, like android connectivity issue from wix/Detox#1601

chore(e2e): terminate e2e tests if detox.init fails

bc3cd3c

failures due to misconfiguration (or random node module breakages) result in a whole bunch of unrelated/confusing failures in the tests, and there's no point of continuing tests if there's no detox anyway

chore: add fs-extra as a direct dev dependency

55fa75c

react-devtools --depends-on--> electron -> electron-download -> fs-extra 0.3 for some reason detox loads fs-extra 0.3, though it depends on ^4.0.2 and that's why we can't have nice things

chore(e2e/utils): improve readVisibleText

51ac0de

- find Text element by type - try to also find a Text child of the given testID - handle matching multiple children and read each recursively

test(landing): e2e tests for landing screen

d268196

test(creation): e2e tests for identity creation

357fc71

test(recovery): seed phrase recovery tests

57b89e8

test(interactions): add test boilerplate

5a90cf1

mnzaki force-pushed the 1415/e2e_tests_setup branch from 50811db to 5a90cf1 Compare September 12, 2019 16:58

mnzaki changed the title ~~E2E tests setup~~ E2E tests setup and initial tests Sep 12, 2019

mnzaki added 2 commits September 13, 2019 14:37

Merge remote-tracking branch 'origin/develop' into 1415/e2e_tests_setup

93eedee

Conflicts: src/ui/registration/components/entropy.tsx

chore: update snapshots

6069821

mnzaki requested review from saifahn and VolkerSchiewe September 13, 2019 12:52

saifahn reviewed Sep 13, 2019

View reviewed changes

mnzaki added 2 commits September 13, 2019 16:57

chore(e2e/utils): fix detox device access pattern

aaa5b1d

can't access devices unless detox.init is called

Merge remote-tracking branch 'origin/develop' into 1415/e2e_tests_setup

cfd247b

Conflicts: src/ui/recovery/components/inputSeedPhrase.tsx

chore(e2e/utils): extract getDetoxConfig to utils

284cba7

will need later for REPL

mnzaki added 2 commits September 17, 2019 13:36

mnzaki mentioned this pull request Sep 17, 2019

1428/entropy screen changes #1446

Merged

fix(registration/handAnimation): stop looping after unmount

92984eb

Clearing the timeout is not useful, it will have already fired. The problem is with the recursive callbacks, which must be broken with a flag

mnzaki force-pushed the 1415/e2e_tests_setup branch from b8b437a to 92984eb Compare September 17, 2019 12:11

chore: use detox fork from jolocom

7587d59

chore(e2e): add dummy tests to suppress failure

395897d

VolkerSchiewe added the review label Sep 18, 2019

mnzaki added this to the Sprint N milestone Sep 22, 2019

Exulansis self-requested a review September 26, 2019 08:37

Exulansis approved these changes Sep 26, 2019

View reviewed changes

mnzaki added 3 commits September 26, 2019 10:40

Merge remote-tracking branch 'origin/develop' into 1415/e2e_tests_setup

2e5b07c

Conflicts: package.json yarn.lock

chore(README): add quick note about e2e

d7c444f

chore: add yarn e2e:ios:sim

1da2f46

mnzaki self-assigned this Sep 27, 2019

Merge remote-tracking branch 'origin/develop' into 1415/e2e_tests_setup

c739076

Conflicts: yarn.lock

mnzaki merged commit f2ce207 into develop Sep 27, 2019

Exulansis deleted the 1415/e2e_tests_setup branch October 10, 2019 10:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E tests setup and initial tests #1417

E2E tests setup and initial tests #1417

mnzaki commented Aug 20, 2019 •

edited

Loading

saifahn left a comment

saifahn commented Sep 13, 2019

mnzaki commented Sep 13, 2019

saifahn commented Sep 13, 2019 •

edited

Loading

mnzaki commented Sep 17, 2019 •

edited

Loading

mnzaki commented Sep 17, 2019

Exulansis commented Sep 26, 2019

Exulansis left a comment

E2E tests setup and initial tests #1417

E2E tests setup and initial tests #1417

Conversation

mnzaki commented Aug 20, 2019 • edited Loading

NOTES

saifahn left a comment

Choose a reason for hiding this comment

Android Emulator

Android Device

saifahn commented Sep 13, 2019

iOS Device

mnzaki commented Sep 13, 2019

saifahn commented Sep 13, 2019 • edited Loading

mnzaki commented Sep 17, 2019 • edited Loading

mnzaki commented Sep 17, 2019

Exulansis commented Sep 26, 2019

Exulansis left a comment

Choose a reason for hiding this comment

mnzaki commented Aug 20, 2019 •

edited

Loading

saifahn commented Sep 13, 2019 •

edited

Loading

mnzaki commented Sep 17, 2019 •

edited

Loading