Subliminal

An understated approach to iOS integration testing.

Code Quality Rank: L3

Programming language: Objective-C

License: Apache License 2.0

Tags: Testing     UI Testing    

Latest version: v1.1.0

README

Subliminal is a framework for writing iOS integration tests. Subliminal provides a familiar OCUnit/XCTest-like interface to Apple's UIAutomation framework, with tests written entirely in Objective-C. Subliminal also provides a powerful mechanism for your tests to manipulate your application directly.

[ Features • Getting Started • Requirements • Usage • Continuous Integration • Contributing • Contact • License ]

Features

Seamless Integration

Write your tests in Objective-C, and run them from Xcode. See rich-text logs and screenshots in Instruments. Use UIAutomation to simulate user interaction. Subliminal lets you use familiar tools, no dependencies required.

Full Control

By using UIAutomation, Subliminal can simulate almost any interaction--without resorting to private APIs. From navigating in-app purchase dialogs, to putting your app to sleep, Subliminal lets you simulate complex interaction like a user. And when you want to manipulate your app directly, Subliminal will help you do that too.

Scalable Tests

Define Objective-C methods to help set up and tear down tests. Leverage native support for continuous integration. Take confidence in Subliminal's complete documentation and full test coverage. Subliminal is the perfect foundation for your tests.

How to Get Started

• Download Subliminal and try out the included example app
• See an installation walkthrough and screencast
• Check out Subliminal's API documentation or the guides to using Subliminal on the Wiki
• Read a comparison of Subliminal to other integration test frameworks

Running the Example App

1. Clone the Subliminal repo: git clone https://github.com/inkling/Subliminal.git.
2. cd into the directory: cd Subliminal.
3. If you haven't already, set up Subliminal: rake install.
4. Open the Example project: open Example/SubliminalTest.xcodeproj.
5. Switch to the "Integration Tests" scheme. You may also see a scheme called "Subliminal Integration tests"--make sure you choose "Integration Tests."
6. Choose Product > Profile (⌘+I).
7. Under the User Templates, choose Subliminal.

Installing Subliminal

For an installation walkthrough, refer to Subliminal's wiki.

Requirements

Subliminal supports projects built using Xcode 5.1 and iOS 7.x SDKs, and deployment targets running iOS 6.1 through 7.1.

For iOS 5.1 support, use Subliminal 1.1.0 (found in the Releases section or on CocoaPods). To test in the iOS 5.1 Simulator, you will need to run OS X 10.8 and manually add the iOS 5.1 Simulator to Xcode 5.1, as described here.

Usage

Subliminal is designed to be instantly familiar to users of OCUnit/XCTest. In Subliminal, subclasses of SLTest define tests as methods beginning with test. At run-time, Subliminal discovers and runs these tests.

Tests manipulate the user interface and can even manipulate the application directly. Here's what a sample test case looks like:

@implementation STLoginTest

- (void)testLogInSucceedsWithUsernameAndPassword {
    SLTextField *usernameField = [SLTextField elementWithAccessibilityLabel:@"username field"];
    SLTextField *passwordField = [SLTextField elementWithAccessibilityLabel:@"password field" isSecure:YES];
    SLElement *submitButton = [SLElement elementWithAccessibilityLabel:@"Submit"];
    SLElement *loginSpinner = [SLElement elementWithAccessibilityLabel:@"Logging in..."];

    NSString *username = @"Jeff", *password = @"foo";
    [usernameField setText:username];
    [passwordField setText:password];

    [submitButton tap];

    // wait for the login spinner to disappear
    SLAssertTrueWithTimeout([loginSpinner isInvalidOrInvisible], 
                            3.0, @"Log-in was not successful.");

    NSString *successMessage = [NSString stringWithFormat:@"Hello, %@!", username];
    SLAssertTrue([[SLElement elementWithAccessibilityLabel:successMessage] isValid], 
                @"Log-in did not succeed.");

    // Check the internal state of the app.         
    SLAssertTrue(SLAskAppYesNo(isUserLoggedIn), @"User is not logged in.")
}

@end

For more information, see Subliminal's wiki.

Continuous Integration

Subliminal includes end-to-end CI support for building your project, running its tests on the appropriate simulator or device, and outputting results in a variety of formats.

For example scripts and guides to integrate with popular CI services like Travis and Jenkins, see Subliminal's wiki.

Comparison to Other Integration Test Frameworks

• How is Subliminal different from other integration test frameworks?

  Most other integration test frameworks fall into two categories: entirely Objective-C based, or entirely UIAutomation-based.

  Frameworks that are entirely Objective-C based, like KIF, Frank, etc., must hack the application's touch-handling system, using private APIs, to simulate user interaction. There is thus no guarantee that they accurately simulate a user's input. Moreover, these frameworks can only simulate interaction with the application, as opposed to interaction with the device, other processes like in-app purchase alerts, etc.

  Frameworks that are entirely based on Apple's UIAutomation framework require cumbersome workflows--writing tests in JavaScript, in Instruments--which do not make use of the developer's existing toolchain. Moreover, they offer the developer no means of manipulating the application directly--it is a complete black box to a UIAutomation-based test.

  Only Subliminal combines the convenience of writing tests in Objective-C with the power of UIAutomation.

• How is Subliminal different than UIAutomation?

  Besides the limitations of UIAutomation described above, it is extremely difficult to write UIAutomation tests. This is because UIAutomation requires that user interface elements be identified by their position within the "element hierarchy", like

  var cell = UIATarget.localTarget().frontMostApp().mainWindow().tableViews()[0].cells()["foo"];
  

  These references are not only difficult to read but are also difficult to write. To refer to any particular element, you have to describe its entire ancestry, while including only the views that UIAutomation deems necessary (images, yes; accessible elements, maybe; private UIWebView subviews, sure!).

  UIAutomation-based tests are not meant to be written, but to be "recorded" using Instruments. This forces dependence on Instruments, and makes the tests difficult to modify thereafter.

  Subliminal allows developers to identify elements by their properties, independent of their position in the element hierarchy:

  SLElement *fooCell = [SLElement elementWithAccessibilityLabel:@"foo"];
  

  Subliminal abstracts away the complexity of UIAutomation scripts to let developers focus on writing tests.

  Subliminal also fixes several bugs in UIAutomation and the instruments CLI tool, such as instruments' lack for true device support. And, last but not least, Subliminal rewrites instruments' output using human-friendly formatting and ANSI colors:

  

Contributing

Subliminal welcomes pull requests! Check out the contributing guidelines to learn how to set up Subliminal for development and how to make a successful pull request.

Credits

Created by Jeff Wear, made possible by Inkling, with help from:

• William Green
• John Detloff
• Aaron Golden
• Lukhnos Liu
• Aaron Haney

and Subliminal's growing list of contributors.

Contact

• If you need help, use Stack Overflow. (Tag 'subliminal'.)
• If you'd like to ask a general question, use Stack Overflow.
• If you've found a bug, open an issue.
• If you have a feature request, open an issue.
• If you'd like to contribute (awesome!), see the contributing guidelines to get started.

If you'd like to chat, we hold "office hours" on Gitter 3-4pm Pacific Time, Tuesdays and Thursdays.

Lastly, you can follow Subliminal on Twitter for news and tips (@subliminaltest).

Copyright and License

Copyright 2013-2014 Inkling Systems, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.