Continuous Integration

You can run Subliminal tests from the command line using the subliminal-test script, which takes care of building your application and running the tests on the appropriate simulator or device.

To use subliminal-test, first:

1. Install Subliminal on the test machine. If you've already added Subliminal to your project, you only need to install Subliminal's dependencies by running rake install DOCS=no. If you're using Git, you may need to initialize your repository's submodules too.
2. "Share" the "Integration Tests" scheme to make it available to the CI server: in Xcode, click Product > Schemes > Manage Schemes…, click the "Shared" checkbox next to the scheme, and check the resulting file into source control.

3.  Enable GUI scripting: if the test machine is running Mac OS X 10.8 Mountain Lion, open System Preferences and check "Enable Access for Assistive Devices" in the Accessibility preference pane. If the test machine runs Mac OS X 10.9 Mavericks, open System Preferences and click on "Security & Privacy". Select "Accessibility" and drag Terminal.app from Applications/Utilities into the list. Do not forget to check the box.
   

A minimal test runner would then look something like this:

#!/bin/bash

# Run the tests on the 3.5" Retina iPhone Simulator
DEVICE="iPhone Retina (3.5-inch)"

# Run the tests on iOS 7.0
VERSION=7.0

# Allow `subliminal-test` to work around bugs in Apple's `instruments` tool 
# while running un-attended. See the FAQ at the bottom of this page (https://github.com/inkling/Subliminal/wiki/Continuous-Integration#faq) for more information.
PASSWORD="password1234"

OUTPUT_DIR=reports
mkdir -p "$OUTPUT_DIR"

# Returns 0 on success, 1 on failure
# Log output and screenshots will be placed in $OUTPUT_DIR
"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal-test" \
	-project "$YOUR_PROJECT" \
	-sim_device "$DEVICE" \
	-sim_version "$VERSION" \
	-login_password "$PASSWORD" \
	-output "$OUTPUT_DIR"

CI Platforms

Travis CI

Travis CI is a CI service that runs OS X and is free for open-source projects. After creating an account on Travis and enabling it for your repo, Travis will look for a .travis.yml file to know how to run your tests. Here's a minimal .travis.yml:

language: objective-c

env:
    global:
        - PROJECT_PATH="TestTravis.xcodeproj"
        - DEVICE="iPhone Retina (3.5-inch)"
        # Travis only supports testing on 7.0 until https://github.com/travis-ci/travis-ci/issues/2051 is resolved.
        # See the FAQ at the bottom of this page (https://github.com/inkling/Subliminal/wiki/Continuous-Integration#faq) for more information.
        - VERSION="7.0"

before_install:
    # Travis will automatically initialize git submodules & install Cocoapods.
    # (Skip this step if using Cocoapods--the Podspec installs the template automatically).
    - rake install --rakefile Integration\ Tests/Subliminal/Rakefile DOCS=no

# Use Pods/Subliminal/.../subliminal-test if using Cocoapods
script: >
    "Integration Tests/Subliminal/Supporting Files/CI/subliminal-test"
    -project "$PROJECT_PATH"
    -sim_device "$DEVICE"
    -sim_version "$VERSION"
    --live

Subliminal runs integration tests against itself using Travis. Take a look at its configuration file for a more in-depth example.

Jenkins

For CI servers like Jenkins, you can process test logs into JUnit reports using the subliminal_uialog_to_junit script:

"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal_uialog_to_junit" \
	-i "$OUTPUT_DIR/Run\ Data/Automation\ Results.plist" \
	-o "$OUTPUT_DIR/junit.xml"

FAQ

• Why does the subliminal-test script require my login password (except on Travis)?

  subliminal-test can work around several bugs in Apple's instruments tool, but only with superuser privileges. Providing your password lets the script run un-attended. The password is used:

  1. To authorize instruments to take control of your application if it asks for such permission when launched: http://openradar.appspot.com/radar?id=1544403.
  2. To temporarily modify the Xcode folder to force instruments to use the specified SDK to run the tests, whereas it would otherwise use an arbitrary SDK: http://openradar.appspot.com/radar?id=3107401.

  subliminal-test cannot itself be run with superuser privileges because instruments only works properly if it is run as the user.

  If a developer will be attending the tests as they execute (for instance on their local machine rather than on the build server), and so can enter their password as required, they may execute subliminal-test with the --live option.

  Travis is an exception to this requirement, because the Travis VMs have been successfully "pre-authorized" to run instruments (see this discussion). We require the user to specify the login password (or --live) because we have not been able to consistently pre-authorize other machines.

• Why can Travis only test on iOS 7.0 at present?

  Travis has not yet upgraded to Xcode 5.1. Prior to 5.1, the only way of forcing instruments to use older SDKs is to modify the Xcode folder, which requires the user's login password. That information is not publicly available for the Travis VMs, though.

  Once Travis upgrades, Subliminal will be able to select SDKs using a new feature of instruments.