Continuous Integration

You can run Subliminal tests from the command line using the subliminal-test script, which takes care of building your application and running the tests on the appropriate simulator or device.

To use subliminal-test, first:

1. Install Subliminal on the test machine, typically by installing your dependencies and running the Rake task
2. "Share" the "Integration Tests" scheme to make it available to the CI server: in Xcode, click Product > Schemes > Manage Schemes…, click the "Shared" checkbox next to the scheme, and check the resulting file into source control.

3.  Enable GUI scripting: if the test machine is running Mac OS X 10.8 Mountain Lion, open System Preferences and check "Enable Access for Assistive Devices" in the Accessibility preference pane. If the test machine runs Mac OS X 10.9 Mavericks, open System Preferences and click on "Security & Privacy". Select "Accessibility" and drag Terminal.app from Applications/Utilities into the list. Do not forget to check the box.
   

A minimal test runner would then look something like this:

#!/bin/bash

# Run the tests on the 3.5" Retina iPhone Simulator
DEVICE="iPhone Retina (3.5-inch)"

# Run the tests on iOS 7.0
VERSION=7.0

# Allow `subliminal-test` to work around bugs in Apple's `instruments` tool 
# while running un-attended. See the FAQ for more information.
PASSWORD="password1234"

OUTPUT_DIR=reports
mkdir -p "$OUTPUT_DIR"

# Returns 0 on success, 1 on failure
# Log output and screenshots will be placed in $OUTPUT_DIR
"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal-test" \
	-project "$YOUR_PROJECT" \
	-sim_device "$DEVICE" \
	-sim_version "$VERSION" \
	-login_password "$PASSWORD" \
	-output "$OUTPUT_DIR"

CI Platforms

Travis CI

Travis CI is a free (for open-source) CI service that runs OS X. After creating an account on Travis and enabling it for your repo, Travis will look for a .travis.yml file to know how to run your tests. Here's a minimal .travis.yml:

language: objective-c

env:
    global:
        - PROJECT_PATH="TestTravis.xcodeproj"
        - DEVICE="iPhone Retina (3.5-inch)"
        - VERSION="7.0"
        - PASSWORD="unused" # Currently Travis works without a password but this may change.
        - OUTPUT_DIR="output"

before_install:
    # Travis will automatically install git submodules
    - rake install --rakefile Integration\ Tests/Subliminal/Rakefile DOCS=no

script: >
    "Integration Tests/Subliminal/Supporting Files/CI/subliminal-test"
    -project "$PROJECT_PATH"
    -sim_device "$DEVICE"
    -sim_version "$VERSION"
    -login_password "$PASSWORD"
    -output "$OUTPUT_DIR"

Based on experiment, Travis CI does not need the correct password for Travis to run Subliminal, nor does GUI scripting need to be enabled. However, Travis CI is in the process of upgrading to OS X Mavericks, which may invalidate these findings.

Subliminal runs integration tests against itself using Travis. Take a look at its configuration file for a more in-depth example.

Jenkins

For CI servers like Jenkins, you can process test logs into JUnit reports using the subliminal_uialog_to_junit script:

"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal_uialog_to_junit" \
	-i "$OUTPUT_DIR/Run\ Data/Automation\ Results.plist" \
	-o "$OUTPUT_DIR/junit.xml"

FAQ

• Why does the subliminal-test script require my login password?

  subliminal-test can work around several bugs in Apple's instruments tool, but only with superuser privileges. Providing your password lets the script run un-attended. The password is used:

  1. To authorize instruments to take control of your application if it asks for such permission when launched: http://openradar.appspot.com/radar?id=1544403.
  2. To temporarily modify the Xcode folder to force instruments to use the specified SDK to run the tests, whereas it would otherwise use an arbitrary SDK: http://openradar.appspot.com/radar?id=3107401.

  subliminal-test cannot itself be run with superuser privileges because instruments only works properly if it is run as the user.

  If a developer will be attending the tests as they execute (for instance on their local machine rather than on the build server), and so can enter their password as required, they may execute subliminal-test with the --live option.