Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup E2E testing - Appium for IOS/Android #1099

Open
wants to merge 17 commits into
base: service_rewrite_2023
Choose a base branch
from

Conversation

jiji14
Copy link
Contributor

@jiji14 jiji14 commented Nov 9, 2023

Setup E2E testing - Appium for IOS/Android

Related Issue

Considerations for Integration and End-to-End Testing Strategies

Testing Overview

I've configured WebdriverIO and Appium for E2E testing. WebdriverIO, an open-source testing utility for Node.js, seamlessly integrates with Appium, allowing us to write test scripts for both IOS and Android platforms. We have separate configuration files for iOS and Android, and we need to run tests for each platform separately. Test scripts will be written in WebdriverIO using Mocha. (Unfortunately, Jest was not in options, and Mocha was the best alternative due to its similarity.)

Check out this article for a comprehensive guide on Appium and WebdriverIO integration.

How to run test locally

Because we need to test different devices and platform versions, I made the script dynamic. When you run the test, you can pass the device name and platform version as follows:


  1. Run IOS or Android Emulator

  1. Run Appium
npx appium

  1. Run test
  • For IOS
npm run appium-ios -- --deviceName="{deviceName}" --platformVersion={platformVersion} 

If you encounter this issue,

error: Could not determine iOS SDK version

Try this,

sudo xcode-select --switch /Applications/Xcode.app

Resource: Stack Overflow


  • For Android
npm run appium-android -- --deviceName="{deviceName}" --platformVersion={platformVersion} 

Make sure apk file is in your local repository (e-mission-phone > apps > em-devapp-3.2.5.apk)


Test result

  1. IOS
    Screenshot 2023-11-09 at 12 04 45 PM
  2. Android
    Screenshot 2023-11-10 at 3 09 32 PM

To do

  1. Write Test Script for One Function
    Currently, the test checks if the Appium server can run successfully, connects to the emulator, and verifies the visibility of the app container.
  2. GitHub Actions Integration
    Explore setting up E2E tests on GitHub Actions. Investigate how Appium can access the emulator or real devices in the GitHub Actions environment.

@jiji14 jiji14 marked this pull request as draft November 9, 2023 05:01
@jiji14
Copy link
Contributor Author

jiji14 commented Nov 13, 2023

@JGreenlee
I ran e2e tests on IOS and Android in my local environment. For e2e tests, we need to 1. run emulator 2. run appium server 3. run test script. Can you please check if it runs in your local environment as well?

I also designed the script to be dynamic allowing us to test on various devices and platforms. What are your thoughts on this approach?

During this initial test phase, I focused on ensuring that the appium server connects to emulators and recognizes app container. Before proceeding to the test script, I would like to discuss more about test strategy.

WebdriverIO uses selector, and the selectors are differ between IOS and Android, so I am contemplating whether we should maintain separate test files for a single feature. For the sample test, I separated the selectors based on device in a single file, but I think it might become unwieldy as complexity increases. You can refer to the selectors documentation here

Also, I've found some resources on running tests with GitHub Actions. I was curious about how the Appium server connects to the emulator and found the answer in this blog Emulator will be setup during the GitHub Actions execution. I will try this later!

Please share any suggestions or ideas you may have regarding our e2e test 😄

@jiji14 jiji14 marked this pull request as ready for review November 13, 2023 22:31
Comment on lines +5 to +19
const getDeviceName = (platform) => {
const deviceName = process.argv.find((arg) => arg.includes('--deviceName'));
const defaultDeviceName = platform === 'iOS' ? 'iPhone 13' : 'Pixel 3a API 33';
return deviceName ? deviceName.split('=')[1] : defaultDeviceName;
};

/**
* get Platform Version from script
* @param platform iOS or Android
*/
const getPlatformVersion = (platform) => {
const platformVersion = process.argv.find((arg) => arg.includes('--platformVersion'));
const defaultPlatformVersion = platform === 'iOS' ? '15.0' : '13';
return platformVersion ? platformVersion.split('=')[1] : defaultPlatformVersion;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the way you have it split up currently is really good. The only suggestion I might make is for these functions to be split down even further.
Perhaps getDeviceName(platform) still exists, but instead of doing the logic for 'iOS' ? 'iPhone 13' : 'Pixel 3a API 33'; in here, we create an additional function to say getAndroidDeviceName and getiOSDeviceName, and then getDeviceName calls the appropriate one. This way we can modularize the iPhone/Android versions

Just a suggestion though! I am not too familiar with Appium or e2e testing strategies, and looking at what you've written so far looks great!!! 😊

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flexibility provided by deviceName and platformVersion seems adequate to me.
Clever handling of this, great job!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The one suggestion that I might have is to put these versions into a separate setup file, similar to all the other versions that we need in our build https://github.com/e-mission/e-mission-phone/blob/master/setup/export_shared_dep_versions.sh

so:

  1. we can upgrade without changing the code
  2. people can find all the versions in one place instead of wading through tests to find them.

@JGreenlee
Copy link
Collaborator

iOS

After opening emulator (successful) and running npx appium in another terminal (successful), appium-ios was not successful for me:

(base) jgreenle-34794s:e-mission-phone jgreenle$ npm run appium-ios -- --deviceName="iPhone 13" --platformVersion="15.5"
 
> [email protected] appium-ios
> npx wdio ./e2e-tests/config/ios.config.js -- --deviceName=iPhone 13 --platformVersion=15.5

platformVersion --platformVersion=15.5

Execution of 1 workers started at 2023-11-17T17:00:14.815Z

2023-11-17T17:00:14.851Z INFO @wdio/cli:launcher: Run onPrepare hook
2023-11-17T17:00:14.857Z INFO @wdio/appium-service: Will spawn Appium process: node /Users/jgreenle/openpath/e-mission-phone/node_modules/appium/index.js --base-path /
2023-11-17T17:00:24.263Z ERROR @wdio/appium-service: Appium exited before timeout (exit code: 1)
dbug
2023-11-17T17:00:24.265Z ERROR @wdio/cli:utils: A service failed in the 'onPrepare' hook
Error: Appium exited before timeout (exit code: 1)
dbug
    at ChildProcess.<anonymous> (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/appium-service/build/launcher.js:141:22)
    at Object.onceWrapper (node:events:627:26)
    at ChildProcess.emit (node:events:512:28)
    at ChildProcess.emit (node:domain:489:12)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)

Continue...
2023-11-17T17:00:24.269Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2023-11-17T17:00:24.271Z INFO @wdio/local-runner: Start worker 0-0 with arg: ./e2e-tests/config/ios.config.js,--,--deviceName=iPhone 13,--platformVersion=15.5
[0-0] 2023-11-17T17:00:26.180Z INFO @wdio/local-runner: Run worker command: run
[0-0] platformVersion --platformVersion=15.5
[0-0] RUNNING in iOS - file:///e2e-tests/specs/sample.js
[0-0] 2023-11-17T17:00:27.700Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-0] 2023-11-17T17:00:27.700Z INFO @wdio/utils: Connecting to existing driver at http://127.0.0.1:4723/
[0-0] 2023-11-17T17:00:27.905Z INFO webdriver: [POST] http://127.0.0.1:4723/session
[0-0] 2023-11-17T17:00:27.905Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: {
[0-0]       platformName: 'iOS',
[0-0]       'appium:deviceName': 'iPhone 13',
[0-0]       'appium:platformVersion': '15.5',
[0-0]       'appium:automationName': 'XCUITest',
[0-0]       'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]     },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: {
[0-0]     platformName: 'iOS',
[0-0]     'appium:deviceName': 'iPhone 13',
[0-0]     'appium:platformVersion': '15.5',
[0-0]     'appium:automationName': 'XCUITest',
[0-0]     'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]   }
[0-0] }
[0-0] 2023-11-17T17:01:33.965Z WARN webdriver: Request failed with status 500 due to An unknown server-side error occurred while processing the command. Original error: App with bundle identifier 'edu.berkeley.eecs.emission.devapp' unknown
[0-0] 2023-11-17T17:01:33.965Z INFO webdriver: Retrying 1/3
[0-0] 2023-11-17T17:01:33.965Z INFO webdriver: [POST] http://127.0.0.1:4723/session
[0-0] 2023-11-17T17:01:33.965Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: {
[0-0]       platformName: 'iOS',
[0-0]       'appium:deviceName': 'iPhone 13',
[0-0]       'appium:platformVersion': '15.5',
[0-0]       'appium:automationName': 'XCUITest',
[0-0]       'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]     },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: {
[0-0]     platformName: 'iOS',
[0-0]     'appium:deviceName': 'iPhone 13',
[0-0]     'appium:platformVersion': '15.5',
[0-0]     'appium:automationName': 'XCUITest',
[0-0]     'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]   }
[0-0] }
[0-0] 2023-11-17T17:01:58.264Z WARN webdriver: Request failed with status 500 due to An unknown server-side error occurred while processing the command. Original error: App with bundle identifier 'edu.berkeley.eecs.emission.devapp' unknown
[0-0] 2023-11-17T17:01:58.264Z INFO webdriver: Retrying 2/3
[0-0] 2023-11-17T17:01:58.264Z INFO webdriver: [POST] http://127.0.0.1:4723/session
[0-0] 2023-11-17T17:01:58.264Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: {
[0-0]       platformName: 'iOS',
[0-0]       'appium:deviceName': 'iPhone 13',
[0-0]       'appium:platformVersion': '15.5',
[0-0]       'appium:automationName': 'XCUITest',
[0-0]       'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]     },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: {
[0-0]     platformName: 'iOS',
[0-0]     'appium:deviceName': 'iPhone 13',
[0-0]     'appium:platformVersion': '15.5',
[0-0]     'appium:automationName': 'XCUITest',
[0-0]     'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]   }
[0-0] }
[0-0] 2023-11-17T17:02:26.699Z WARN webdriver: Request failed with status 500 due to An unknown server-side error occurred while processing the command. Original error: App with bundle identifier 'edu.berkeley.eecs.emission.devapp' unknown
[0-0] 2023-11-17T17:02:26.699Z INFO webdriver: Retrying 3/3
[0-0] 2023-11-17T17:02:26.699Z INFO webdriver: [POST] http://127.0.0.1:4723/session
[0-0] 2023-11-17T17:02:26.699Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: {
[0-0]       platformName: 'iOS',
[0-0]       'appium:deviceName': 'iPhone 13',
[0-0]       'appium:platformVersion': '15.5',
[0-0]       'appium:automationName': 'XCUITest',
[0-0]       'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]     },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: {
[0-0]     platformName: 'iOS',
[0-0]     'appium:deviceName': 'iPhone 13',
[0-0]     'appium:platformVersion': '15.5',
[0-0]     'appium:automationName': 'XCUITest',
[0-0]     'appium:app': 'edu.berkeley.eecs.emission.devapp'
[0-0]   }
[0-0] }
[0-0] 2023-11-17T17:02:58.854Z ERROR webdriver: Request failed with status 500 due to unknown error: An unknown server-side error occurred while processing the command. Original error: App with bundle identifier 'edu.berkeley.eecs.emission.devapp' unknown
[0-0] 2023-11-17T17:02:58.854Z ERROR webdriver: unknown error: An unknown server-side error occurred while processing the command. Original error: App with bundle identifier 'edu.berkeley.eecs.emission.devapp' unknown
[0-0]     at getErrorFromResponseBody (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriver/build/utils.js:195:12)
[0-0]     at NodeJSRequest._request (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriver/build/request/index.js:178:23)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:95:5)
[0-0]     at async startWebDriverSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriver/build/utils.js:64:20)
[0-0]     at async Function.newSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriver/build/index.js:19:45)
[0-0]     at async remote (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriverio/build/index.js:45:22)
[0-0]     at async Runner._startSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/runner/build/index.js:224:29)
[0-0]     at async Runner._initSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/runner/build/index.js:190:25)
[0-0]     at async Runner.run (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/runner/build/index.js:79:19)
[0-0] 2023-11-17T17:02:58.861Z ERROR @wdio/runner: Error: Failed to create session.
[0-0] An unknown server-side error occurred while processing the command. Original error: App with bundle identifier 'edu.berkeley.eecs.emission.devapp' unknown
[0-0]     at startWebDriverSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriver/build/utils.js:69:15)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:95:5)
[0-0]     at async Function.newSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriver/build/index.js:19:45)
[0-0]     at async remote (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/webdriverio/build/index.js:45:22)
[0-0]     at async Runner._startSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/runner/build/index.js:224:29)
[0-0]     at async Runner._initSession (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/runner/build/index.js:190:25)
[0-0]     at async Runner.run (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/runner/build/index.js:79:19)
[0-0] FAILED in iOS - file:///e2e-tests/specs/sample.js
2023-11-17T17:02:59.006Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2023-11-17T17:02:59.007Z INFO @wdio/cli:launcher: Run onComplete hook

Spec Files:	 0 passed, 1 failed, 1 total (100% completed) in 00:02:44  

2023-11-17T17:02:59.008Z INFO @wdio/local-runner: Shutting down spawned worker
2023-11-17T17:02:59.261Z INFO @wdio/local-runner: Waiting for 0 to shut down gracefully
2023-11-17T17:02:59.265Z INFO @wdio/local-runner: shutting down
(base) jgreenle-34794s:e-mission-phone jgreenle$ 

@JGreenlee
Copy link
Collaborator

Android

For Android, it was successful:

(base) jgreenle-34794s:e-mission-phone jgreenle$ npm run appium-android -- --deviceName="px4" --platformVersion="11"

> [email protected] appium-android
> npx wdio ./e2e-tests/config/android.config.js -- --deviceName=px4 --platformVersion=11

platformVersion --platformVersion=11

Execution of 1 workers started at 2023-11-17T17:08:59.024Z

2023-11-17T17:08:59.049Z INFO @wdio/cli:launcher: Run onPrepare hook
2023-11-17T17:08:59.052Z INFO @wdio/appium-service: Will spawn Appium process: node /Users/jgreenle/openpath/e-mission-phone/node_modules/appium/index.js --base-path /
2023-11-17T17:09:09.107Z ERROR @wdio/appium-service: Appium exited before timeout (exit code: 1)
dbug
2023-11-17T17:09:09.108Z ERROR @wdio/cli:utils: A service failed in the 'onPrepare' hook
Error: Appium exited before timeout (exit code: 1)
dbug
    at ChildProcess.<anonymous> (file:///Users/jgreenle/openpath/e-mission-phone/node_modules/@wdio/appium-service/build/launcher.js:141:22)
    at Object.onceWrapper (node:events:627:26)
    at ChildProcess.emit (node:events:512:28)
    at ChildProcess.emit (node:domain:489:12)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)

Continue...
2023-11-17T17:09:09.110Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2023-11-17T17:09:09.111Z INFO @wdio/local-runner: Start worker 0-0 with arg: ./e2e-tests/config/android.config.js,--,--deviceName=px4,--platformVersion=11
[0-0] 2023-11-17T17:09:10.185Z INFO @wdio/local-runner: Run worker command: run
[0-0] platformVersion --platformVersion=11
[0-0] RUNNING in Android - file:///e2e-tests/specs/sample.js
[0-0] 2023-11-17T17:09:11.026Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-0] 2023-11-17T17:09:11.026Z INFO @wdio/utils: Connecting to existing driver at http://127.0.0.1:4723/
[0-0] 2023-11-17T17:09:11.147Z INFO webdriver: [POST] http://127.0.0.1:4723/session
[0-0] 2023-11-17T17:09:11.147Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: {
[0-0]       platformName: 'Android',
[0-0]       'appium:deviceName': 'px4',
[0-0]       'appium:platformVersion': '11',
[0-0]       'appium:automationName': 'UiAutomator2',
[0-0]       'appium:app': '/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk'
[0-0]     },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: {
[0-0]     platformName: 'Android',
[0-0]     'appium:deviceName': 'px4',
[0-0]     'appium:platformVersion': '11',
[0-0]     'appium:automationName': 'UiAutomator2',
[0-0]     'appium:app': '/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk'
[0-0]   }
[0-0] }
[0-0] 2023-11-17T17:09:21.467Z INFO webdriver: COMMAND findElement("-android uiautomator", "new UiSelector().className("android.widget.FrameLayout")")
[0-0] 2023-11-17T17:09:21.469Z INFO webdriver: [POST] http://127.0.0.1:4723/session/1fc5fcc0-74f4-4ea4-97ed-94b74589daa6/element
[0-0] 2023-11-17T17:09:21.469Z INFO webdriver: DATA {
[0-0]   using: '-android uiautomator',
[0-0]   value: 'new UiSelector().className("android.widget.FrameLayout")'
[0-0] }
[0-0] 2023-11-17T17:09:21.557Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': '00000000-0000-000a-ffff-ffff7ffffffe',
[0-0]   ELEMENT: '00000000-0000-000a-ffff-ffff7ffffffe'
[0-0] }
[0-0] 2023-11-17T17:09:21.568Z INFO webdriver: COMMAND isElementDisplayed("00000000-0000-000a-ffff-ffff7ffffffe")
[0-0] 2023-11-17T17:09:21.569Z INFO webdriver: [GET] http://127.0.0.1:4723/session/1fc5fcc0-74f4-4ea4-97ed-94b74589daa6/element/00000000-0000-000a-ffff-ffff7ffffffe/displayed
[0-0] 2023-11-17T17:09:21.571Z INFO webdriver: COMMAND deleteSession()
[0-0] 2023-11-17T17:09:21.571Z INFO webdriver: [DELETE] http://127.0.0.1:4723/session/1fc5fcc0-74f4-4ea4-97ed-94b74589daa6
[0-0] 2023-11-17T17:09:21.915Z INFO webdriver: RESULT null
[0-0] PASSED in Android - file:///e2e-tests/specs/sample.js
2023-11-17T17:09:22.049Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2023-11-17T17:09:22.049Z INFO @wdio/cli:launcher: Run onComplete hook

 "spec" Reporter:
------------------------------------------------------------------
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0] Running: /Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk on Android
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0] Session ID: 1fc5fcc0-74f4-4ea4-97ed-94b74589daa6
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0]
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0] » /e2e-tests/specs/sample.js
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0] Connect test
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0]    ✓ should call app successfully
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0]
[/Users/jgreenle/openpath/e-mission-phone/apps/em-devapp-3.2.5.apk Android #0-0] 1 passing (476ms)


Spec Files:	 1 passed, 1 total (100% completed) in 00:00:23  

2023-11-17T17:09:22.050Z INFO @wdio/local-runner: Shutting down spawned worker
2023-11-17T17:09:22.303Z INFO @wdio/local-runner: Waiting for 0 to shut down gracefully
2023-11-17T17:09:22.303Z INFO @wdio/local-runner: shutting down

Copy link
Collaborator

@JGreenlee JGreenlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, I'm so glad you have set up the architecture for this!
I think we may want to consider targeting the built app instead of the devapp (we will await input from @shankari for this). But overall, it looks like the architecture is mostly there already!

WebdriverIO uses selector, and the selectors are differ between IOS and Android, so I am contemplating whether we should maintain separate test files for a single feature. For the sample test, I separated the selectors based on device in a single file, but I think it might become unwieldy as complexity increases. You can refer to the selectors documentation here

I see the inline conditional to distinguish between android and ios selectors

const selector = driver.isAndroid
      ? await $('android=new UiSelector().className("android.widget.FrameLayout")')
      : await $('UIATarget.localTarget().frontMostApp().mainWindow()');

Maybe this distinction can just be moved to a separate function used to target the Webview.
Once inside the Webview, there are not significant differences between Android and iOS layouts and I think we should be able to target things using common IDs.

'appium:deviceName': getDeviceName('Android'),
'appium:platformVersion': getPlatformVersion('Android'),
'appium:automationName': 'UiAutomator2',
'appium:app': join(process.cwd(), './apps/em-devapp-3.2.5.apk'),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should discuss whether we want to run e2e tests against the devapp (in the 'serve' configuration) or against the actual e-mission app (after having built it in the 'build' configuration)

I can foresee challenges with trying to run tests through the devapp, and I also think that if we are aiming to test the true behavior of the app, the test target should be as realistic as possible (ie the fully built app)

@shankari What do you think?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also negate the need to copy the devapp apk over to here.
When the app is built, the output apk appears somewhere in platforms/android/...

Comment on lines +5 to +19
const getDeviceName = (platform) => {
const deviceName = process.argv.find((arg) => arg.includes('--deviceName'));
const defaultDeviceName = platform === 'iOS' ? 'iPhone 13' : 'Pixel 3a API 33';
return deviceName ? deviceName.split('=')[1] : defaultDeviceName;
};

/**
* get Platform Version from script
* @param platform iOS or Android
*/
const getPlatformVersion = (platform) => {
const platformVersion = process.argv.find((arg) => arg.includes('--platformVersion'));
const defaultPlatformVersion = platform === 'iOS' ? '15.0' : '13';
return platformVersion ? platformVersion.split('=')[1] : defaultPlatformVersion;
};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flexibility provided by deviceName and platformVersion seems adequate to me.
Clever handling of this, great job!

'appium:deviceName': getDeviceName('iOS'),
'appium:platformVersion': getPlatformVersion('iOS'),
'appium:automationName': 'XCUITest',
'appium:app': 'edu.berkeley.eecs.emission.devapp',
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we switched from testing the devapp to testing the built 'emission' app, this line would change to edu.berkeley.eecs.emission.
We'd need to ensure that 'emission' was installed on the simulator before running tests - hopefully this can be done through commands.

Comment on lines +133 to +278
* Gets executed before test execution begins. At this point you can access to all global
* variables like `browser`. It is the perfect place to define custom commands.
* @param {Array.<Object>} capabilities list of capabilities details
* @param {Array.<String>} specs List of spec file paths that are to be run
* @param {object} browser instance of created browser/device session
*/
// before: function (capabilities, specs) {
// },
/**
* Runs before a WebdriverIO command gets executed.
* @param {string} commandName hook command name
* @param {Array} args arguments that command would receive
*/
// beforeCommand: function (commandName, args) {
// },
/**
* Hook that gets executed before the suite starts
* @param {object} suite suite details
*/
// beforeSuite: function (suite) {
// },
/**
* Function to be executed before a test (in Mocha/Jasmine) starts.
*/
// beforeTest: function (test, context) {
// },
/**
* Hook that gets executed _before_ a hook within the suite starts (e.g. runs before calling
* beforeEach in Mocha)
*/
// beforeHook: function (test, context, hookName) {
// },
/**
* Hook that gets executed _after_ a hook within the suite starts (e.g. runs after calling
* afterEach in Mocha)
*/
// afterHook: function (test, context, { error, result, duration, passed, retries }, hookName) {
// },
/**
* Function to be executed after a test (in Mocha/Jasmine only)
* @param {object} test test object
* @param {object} context scope object the test was executed with
* @param {Error} result.error error object in case the test fails, otherwise `undefined`
* @param {*} result.result return object of test function
* @param {number} result.duration duration of test
* @param {boolean} result.passed true if test has passed, otherwise false
* @param {object} result.retries information about spec related retries, e.g. `{ attempts: 0, limit: 0 }`
*/
// afterTest: function(test, context, { error, result, duration, passed, retries }) {
// },

/**
* Hook that gets executed after the suite has ended
* @param {object} suite suite details
*/
// afterSuite: function (suite) {
// },
/**
* Runs after a WebdriverIO command gets executed
* @param {string} commandName hook command name
* @param {Array} args arguments that command would receive
* @param {number} result 0 - command success, 1 - command error
* @param {object} error error object if any
*/
// afterCommand: function (commandName, args, result, error) {
// },
/**
* Gets executed after all tests are done. You still have access to all global variables from
* the test.
* @param {number} result 0 - test pass, 1 - test fail
* @param {Array.<Object>} capabilities list of capabilities details
* @param {Array.<String>} specs List of spec file paths that ran
*/
// after: function (result, capabilities, specs) {
// },
/**
* Gets executed right after terminating the webdriver session.
* @param {object} config wdio configuration object
* @param {Array.<Object>} capabilities list of capabilities details
* @param {Array.<String>} specs List of spec file paths that ran
*/
// afterSession: function (config, capabilities, specs) {
// },
/**
* Gets executed after all workers got shut down and the process is about to exit. An error
* thrown in the onComplete hook will result in the test run failing.
* @param {object} exitCode 0 - success, 1 - fail
* @param {object} config wdio configuration object
* @param {Array.<Object>} capabilities list of capabilities details
* @param {<Object>} results object containing test results
*/
// onComplete: function(exitCode, config, capabilities, results) {
// },
/**
* Gets executed when a refresh happens.
* @param {string} oldSessionId session ID of the old session
* @param {string} newSessionId session ID of the new session
*/
// onReload: function(oldSessionId, newSessionId) {
// }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that we aren't using hooks now, but it's good to keep this around since we may need them later.

describe('Connect test', () => {
it('should call app successfully', async () => {
const selector = driver.isAndroid
? await $('android=new UiSelector().className("android.widget.FrameLayout")')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain to me what $ does in this instance? Is this how we select elements? (sort of like jQuery?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, $ command is a short and handy way in order to fetch a single element on the page.
You can check more information here

Comment on lines +16 to +17
"appium-ios": "npx wdio ./e2e-tests/config/ios.config.js --",
"appium-android": "npx wdio ./e2e-tests/config/android.config.js --"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we switch to doing e2e tests against a built app instead of the devapp, these will instead be in package.cordovabuild.json.

Comment on lines +56 to +63
"@wdio/appium-service": "^8.22.1",
"@wdio/cli": "^8.22.1",
"@wdio/local-runner": "^8.22.1",
"@wdio/mocha-framework": "^8.22.0",
"@wdio/spec-reporter": "^8.21.0",
"appium": "^2.2.1",
"appium-xcuitest-driver": "^5.8.2",
"appium-uiautomator2-driver": "^2.34.1"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ (along with the dependencies)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could keep tests as e2e-tests/sample.spec.js instead of nesting them in a specs folder?
I've seen the testname.spec.js convention used a lot in other projects

@jiji14
Copy link
Contributor Author

jiji14 commented Nov 17, 2023

@JGreenlee
I identified the reason for the iOS test failure. It was unable to install the webDriverAgent app on the iOS simulator.

I am currently investigating the differences between the instances when the installation of webDriverAgent is successful and when it fails.

Additionally, as you pointed out, the devapp needs to be installed before the test. Consequently, I modified the logic for iOS testing to download devapp.app similar to the Android test. However, this code may change again once we finalize our target app.

@shankari
Copy link
Contributor

@JGreenlee @jiji14 thanks so much for doing this!
wrt the question for me on whether we should use the devapp or the built app, can we support both?
I see @JGreenlee's point about testing in the most realistic environment possible, which is definitely the final built app.
but if we want those of us working on the UI to run these tests, they will need to build the full app as well, which is more complicated than the devapp. I made the devapp at least partly to obviate the need for that change. I know that we are using mocks for most tests, but I believe that at least some of the UI tests are most meaningful when they can communicate with the native code.

On the other hand, before we release a new version, I would like to run the tests against the built version.
Is it possible to make that be configurable using a command line option or an environment variable?

For GitHub actions, we can run the devapp version as part of the UI build (osx-serve-install) and the built app tests as part of the native build (osx-build-android and osx-build-ios)

@JGreenlee
Copy link
Collaborator

I thought that we would only be running the e2e tests in GitHub actions, not manually. I'm anticipating that once we have a comprehensive suite of e2e tests, they could take a very long time to run (potentially 1+ hours?) and I don't think it will be practical to run them all locally.

I know that we are using mocks for most tests, but I believe that at least some of the UI tests are most meaningful when they can communicate with the native code.

The (Appium) e2e tests are separate from the (Jest) unit/component tests; I don't think there is any potential for overlap or communication between them.

@shankari
Copy link
Contributor

The (Appium) e2e tests are separate from the (Jest) unit/component tests; I don't think there is any potential for overlap or communication between them.

I understand that the appium test framework is different from Jest, but in terms of the functionality that we want to test, I think that there might indeed be overlap. For example, we would probably want to have an e2e test which loads data into the label screen, but runs through the full stack of UI -> native -> server to retrieve the data instead of looking only at mock results.

I thought that we would only be running the e2e tests in GitHub actions, not manually. I'm anticipating that once we have a comprehensive suite of e2e tests, they could take a very long time to run (potentially 1+ hours?) and I don't think it will be practical to run them all locally.

From experience on the server side, this is not a great idea. This will work for validation, but once the tests break, it is hard to debug tests that are run using GitHub actions, or to verify that the fixes have worked. It would be best if we could also run individual e2e tests locally.

@jiji14
Copy link
Contributor Author

jiji14 commented Nov 21, 2023

@JGreenlee

I resolved the IOS test issue by running the WebDriverAgent app once before executing Appium tests on the iOS simulator. (It took more than 30 minutes for the installation 😩 )

WebDriverAgent is an open-source tool developed by Facebook for automated testing of iOS apps. It acts as a WebDriver proxy, facilitating communication between the Appium server and the iOS application being tested.

Here is what happened to me. I got the WebDriverAgent app running when I ran the Appium test with the Appium GUI for my initial research. That's why the test passed when I submitted the PR. However, after I reset my emulator for debugging purposes, the WebDriverAgent got deleted. That's why it didn't pass on the second trial.

While there are ways to manually install WebDriverAgent, I discovered that it should be preinstalled when the test scripts start with appium version 2. And it does, but it takes too long! Here's the same GitHub issue: appium/appium#8161. You will find lots posts regarding this issue, but I haven't found any current solutions so far.

I think we can simply run the IOS test by manually installing WebDriverAgent or running the pretest(?) to get the app. Do you have any ideas regarding this matter?

@jiji14
Copy link
Contributor Author

jiji14 commented Nov 21, 2023

For the next step, Jack will handle the iOS setup, while I focus on developing the Android Appium test workflow in GitHub Actions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Issues being worked on
Development

Successfully merging this pull request may close these issues.

5 participants