Run EarlGrey on AWS

[Eruhenon]

Hey,

we're trying to run EarlGrey on AWS device farm. It fails with various errors. Since I can't find any information about anyone succeeding to run this (except for #154) I thought I'd ask here for advice :)

If anyone could guide me in the right direction, it would be very appreciated!

[tirodkar]

What form errors are you seeing @Eruhenon? Are you using a pre-built IPA to test or using one CocoaPods / Carthage etc?

[khandpur]

Adding @bootstraponline just in case he has something to share.

[bootstraponline]

We're also looking into running our EarlGrey tests on AWS. Step one was making the tests run on a local physical device. I'll comment once we have some tests running successfully on AWS.

[bootstraponline]

Amazon commented:

• https://forums.aws.amazon.com/thread.jspa?messageID=772838

I'm not looking into AWS anymore so the Amazon forums are probably the best place for support.

[githubist16]

Earlgrey works with XCTest and I have tested that it works. Use steps outlined here: https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest.html

[bootstraponline]

https://firebase.google.com/docs/test-lab/ios/command-line is what I'm using now.

[tirodkar]

@githubist16 & @bootstraponline is there any way we could add to our docs for support for the different CI platforms?

[bootstraponline]

Linking to https://firebase.google.com/docs/test-lab/ios/firebase-console for firebase is sufficient I think.

[githubist16]

Finally got this going again using EG 2.0 on AWS Device farm. These were the steps used (some of these steps need to be fleshed out. I will keep updating this):

1. Contact AWS team to add 1 or more private devices to your account:
   - https://docs.aws.amazon.com/devicefarm/latest/developerguide/working-with-private-devices.html

2. Add the private device(s) to your team's provisioning profile:
   - https://stackoverflow.com/questions/3578158/adding-devices-to-team-provisioning-profile
   - https://support.smartbear.com/testcomplete/docs/app-testing/mobile/ios/preparing/registering-devices.html
   - This can also be automated using tools such as https://github.com/nomad/cupertino
   - You can also use one of these two methods:
   - (Using this you can use xcode on your local machine to register the device to your profile) https://docs.aws.amazon.com/devicefarm/latest/developerguide/direct-device-access.html
   - (Not sure yet what to with this) https://docs.aws.amazon.com/devicefarm/latest/developerguide/amazon-vpc-endpoints.html

3. Run steps used to build EG 2.0 tests
   - You will need the .app file for your app from here. Do not use the standard method of archiving the app as that is not sufficient.
   - Still create a Payload folder with the .app included

4. Zip up Products folder (Feel free to zip up even more files ex. a shell script you might want to access, etc.)
   - ~/Library/Developer/Xcode/DerivedData//Build/Products
   - Products.zip should include *.xctestrun file and Debug-iphoneos/ directory
   - I called it Products.zip, it can be any name - Aragorn.zip

Products.zip content can look something like

Products
      | - *.xctestrun
      | - Debug-iphoneos/
      | - iOS_Cert.p12 file
      | - run_all_the_things.sh 

5. On AWS Device farm:
   - Create a new run
   - Under choose application: Upload Payload.ipa file created from step 3. If you got it wrong, Device farm will notify you
   - Under configure test: Select Appium Node.js
   - Select the Products.zip file
   - Ignore the warning: 'We could not find the tarball file in the root of your test package...

6. Edit the yaml spec file to look like the following:
   - Unwanted commands were commented out
   - Your steps will need to also import your iOS cert (*.p12) for your codesign identity (I did not show this below, but can cleanup mine and add scripts. I did show that cert file is part of the zip file)
   - Also note that you can access files within your zip and call whatever terminal command you want from the root. DEVICEFARM_TEST_PACKAGE_PATH represents the root of your zip

    version: 0.1
    
    # Phases are collection of commands that get executed on Device Farm.
    phases:
      # The install phase includes commands that install dependencies that your tests use.
      # Default dependencies for testing frameworks supported on Device Farm are already installed.
      install:
        commands:
          # By default, this testspec file uses Appium server version 1.9.1 which is a prerequisite for iOS 12
          # To use a different appium version, change 'APPIUM_VERSION' below to your desired version.
          # For convenience, we have preinstalled the following versions - 1.6.5, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.9.1
          
          
          # By default the node version installed is 11.4.0
          # you can switch to an alternate node version using below command.
          # - nvm install 10.13.0
          
          # Unpackage and install the node modules that you uploaded in the test phase.
          
         
         
      # The pre-test phase includes commands that setup your test environment.
      pre_test:
        commands:
          # We recommend starting appium server process in the background using the command below.
          # Appium server log will go to $DEVICEFARM_LOG_DIR directory.
          # The environment variables below will be auto-populated during run time.
          - echo "Start appium server"
          # The default WDA used is at DEVICEFARM_WDA_DERIVED_DATA_PATH_V1 (Supports versions iOS 12 and below), it is using commit f865d3. See (https://github.com/appium/appium-xcuitest-driver/tree/f865d32e78a5a8a15469bee30ed2f985d378575d)
          # If you need an older WDA version or need support for node modules, use the WDA at DEVICEFARM_WDA_DERIVED_DATA_PATH_V0. (This version does not suport iOS 12)
          
          
                    
    
      # The test phase includes commands that run your test suite execution.
      test:
        commands:
          # Go into the root folder containing your source code and node_modules
          - echo "Navigate to test source code"
          # Change the directory to node_modules folder as it has your test code and the dependency node modules.
          - cd $DEVICEFARM_TEST_PACKAGE_PATH/
          - ls $DEVICEFARM_TEST_PACKAGE_PATH
          - CUSTOM_XCUITESTRUN_FILE=$DEVICEFARM_TEST_PACKAGE_PATH/vEarlGreyTests_iphoneos12.1-arm64e.xctestrun
          - CUSTOM_DERIVED_DATA_PATH=$DEVICEFARM_TEST_PACKAGE_PATH
          - /usr/bin/xcodebuild test-without-building -destination id=$DEVICEFARM_DEVICE_UDID -xctestrun $CUSTOM_XCUITESTRUN_FILE -derivedDataPath $CUSTOM_DERIVED_DATA_PATH -only-testing:<YOURSCHEME>/<YOUR_TEST_CLASS>
    
          
          - echo "Start Appium Node test"
          # Enter the command below to start the tests . The comamnd should be similar to what you use to run the tests locally.
          # For e.g. assuming you run your tests locally using command "node YOUR_TEST_FILENAME.js.", enter the same command below:
          
    
      # The post test phase includes are commands that are run after your tests are executed.
      post_test:
        commands:
    
    # The artifacts phase lets you specify the location where your tests logs, device logs will be stored.
    # And also let you specify the location of your test logs and artifacts which you want to be collected by Device Farm.
    # These logs and artifacts will be available through ListArtifacts API in Device Farm.
    artifacts:
      # By default, Device Farm will collect your artifacts from following directories
    - $DEVICEFARM_LOG_DIR
   

7. Select your private device

8. Run test

[tirodkar]

That sounds awesome.

QQ -

Contact AWS team to add 1 or more private devices to your account

Could we get a link for how to do this? This would look great in a setup doc. Feel free to send it as a .md PR to us.