Continuous Integration

You can run Subliminal tests in continuous integration using the subliminal-test script, which takes care of building your application and running the tests on the appropriate simulator or device. To get started, choose a CI platform below.

CI Platforms

Self-hosted (Jenkins)

To use subliminal-test on CI servers like Jenkins, you must first:

1. Install Subliminal on the server:
   • If you added Subliminal to your project using Git submodules or manually, initialize your project's submodules (as applicable). Then, use Terminal to execute rake install DOCS=no from Subliminal's directory.
   • If you added Subliminal to your project using Cocoapods, use Terminal to execute pod install from the root directory of your project.
2. "Share" your "Integration Tests" scheme to make it available to your CI server: in Xcode, click Product > Schemes > Manage Schemes…, click the "Shared" checkbox next to the scheme, and check the resulting file into source control.
3. "Pre-authorize" Apple's instruments tool as described here.

You then can invoke subliminal-test using a script like the following:

#!/bin/bash

# Run the tests on the 3.5" Retina iPhone Simulator
DEVICE="iPhone Retina (3.5-inch)"

# Run the tests on iOS 7.0
VERSION=7.0

OUTPUT_DIR=reports
mkdir -p "$OUTPUT_DIR"

# Returns 0 on success, 1 on failure
# Log output and screenshots will be placed in $OUTPUT_DIR
"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal-test" \
    -project "$YOUR_PROJECT" \
    -sim_device "$DEVICE" \
    -sim_version "$VERSION" \
    -output "$OUTPUT_DIR"

You can process test logs into JUnit reports using the subliminal_uialog_to_junit script:

"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal_uialog_to_junit" \
    -i "$OUTPUT_DIR/Run\ Data/Automation\ Results.plist" \
    -o "$OUTPUT_DIR/junit.xml"

Travis CI

Travis CI is a CI service that runs OS X and is free for open-source projects. To use Travis:

1. Sign in and activate the GitHub webhook for your repository, as described here.

2. Make sure you've chosen to "Share" your "Integration Tests" scheme to make it available to Travis: in Xcode, click Product > Schemes > Manage Schemes…, click the "Shared" checkbox next to the scheme, and check the resulting file into source control.

3. Add a .travis.yml file to the root of your repository. Here's a minimal .travis.yml:

   language: objective-c
   
   env:
       global:
           - PROJECT_PATH="TestTravis.xcodeproj"
           - DEVICE="iPhone Retina (3.5-inch)"
           - VERSION="7.0"
   
   install:
       # Travis will automatically initialize git submodules & install Cocoapods.
       # Skip this step if using Cocoapods--the Podspec installs Subliminal automatically.
       - rake install --rakefile Integration\ Tests/Subliminal/Rakefile DOCS=no
   
   # Use Pods/Subliminal/.../subliminal-test if using Cocoapods
   script: >
       "Integration Tests/Subliminal/Supporting Files/CI/subliminal-test"
       -project "$PROJECT_PATH"
       -sim_device "$DEVICE"
       -sim_version "$VERSION"

   Subliminal runs integration tests against itself using Travis. Take a look at its configuration file for a more in-depth example.

   The only crucial elements are that your .travis.yml install Subliminal in the install step, unless it's using Cocoapods; and that it run subliminal-test in the build step.

4. Trigger a build by pushing to your repository.

FAQ

• What is "pre-authorization" and why is it necessary?

  Read this answer only if you will be self-hosting your CI environment. If you will be using Travis, you don't need to do anything.

  As a security feature, instruments asks the user for permission to take control of other processes. Nevertheless, permissions dialogs prevent scripts from running un-attended as is desirable for continuous integration.

  instruments can be "pre-authorized" to take control of your application by modifying a configuration file that determines the privileges granted to OS X applications. To pre-authorize instruments, first execute the following command(s) in Terminal:

  # If you are running OS X 10.8:
  sudo /usr/libexec/PlistBuddy -c "Set rights:system.privilege.taskport:allow-root true" /etc/authorization
  
  # If you are running OS X 10.9:
  security authorizationdb read system.privilege.taskport > /tmp/system.privilege.taskport.plist
  /usr/libexec/PlistBuddy -c "Set :allow-root true" /tmp/system.privilege.taskport.plist
  sudo security authorizationdb write system.privilege.taskport < /tmp/system.privilege.taskport.plist

  Then, try to run subliminal-test as you would normally (using a script like the example given in the self-hosted instructions above). You'll be prompted to enter your username and password once, but after that instruments will no longer require authorization.

  Obligatory warning: "Pre-authorization" will make your machine less secure. However, it is currently the only way to run UIAutomation scripts un-attended. Please dupe http://openradar.appspot.com/14135016 to encourage Apple to provide a true solution.