← Back to All Posts

Automated Testing of iOS Apps in CI/CD Pipelines (Part One)

This is a multipart series we are putting together to walk through automation of DevSecOps for mobile solutions. We are going to focus on iOS, but much of this is applicable to Android as well. Our goal is to leverage GitLab as the CI/CD engine and other services like AWS Device Farm, SonarQube, and NowSecure for testing. Finally, the app should pre-stage by self publishing to Apple's App Store for TestFlight publishing.

We want to see our CI/CD pipeline, at a minimum look like this:

For as many mobile solutions that exist out there, the write ups and documentation that exists to automate testing, specifically UI testing is substandard to say the least. This post will lay out some of the techniques we leverage to automate the testing of mobile apps (iOS specifically) to perform fully automated UI testing.

iOS Testing in AWS Device Farm

We leverage AWS Device Farm to implement testing—the capabilities of Device Farm are fantastic, the difficult bit is the practical application of documentation.

Again, everyone talks about automated testing, but who is actually doing it?

We'll dive into AWS Device Farm later.

Creating the test projects

To get started, in your application you'll want to add a new target to your app, a UI Testing Bundle. Provide a product name, select the target you will add it to and you are ready to start building test cases.

Adding the new Target

Configuring the new target

Writing test cases

When Xcode runs your test cases, it will relaunch your application every test case. This persists state, so it allows you to pick up where you left off.

In our (testing) development strategy for building out these test cases, we have developed a Singleton "helper" class that handles much of the authentication and/or registration necessary to log users in. But, this in itself introduces some complexities.

  1. What do you do for new users
  2. What do you do for existing users
  3. What do you do for users who may have left the app in the middle of the startup process
  4. What do you do for users who may not be allowed to use the app
  5. What do you do for users who are locked
  6. etc...

Configuration File Helper

First and foremost, we drive most of our testing by a configuration file. This configuration file provides some potential overrides as well as account information. We have a local-testing-config.json as well as the testing-config.json file to drive this configuration.

Obviously, locally we run the local-testing-config.json file. This has secrets in it and we prevent it from being checked into git (adding a local-testing-config.json to our .gitignore file).

In our CI/CD pipeline, we then pull the legitimate file from a secure location or, alternatively, pull the secrets from a secure location and inject them into the testing-config.json file. How you do that, is up to you. This should be done, obviously, before you build the solution.

Our sample testing-config.json file is as follows:

{
    "apiServiceKey": "",
    "accounts": {
        "alreadyConfigured": {
            "email": "demo1@monkton.us",
            "password": ""
        },
        "needsToSetupProfile": {
            "email": "demo2@monkton.us",
            "password": ""
        },
        "registerAccount": {
            "email": "demo3@monkton.us",
            "password": ""
        },
        "doneAccountSetup": {
            "email": "demo4@monkton.us",
            "password": ""
        },
        "accountLocked": {
            "email": "demo5@monkton.us",
            "password": ""
        }
    }
}

As one can see, we provide unique names to each of the accounts, allowing us to grab the account and perform test cases with it. Your naming strategy should suit your desired testing goals.

This file gets injected into your testing application during build time.

Singleton Authentication Helper

While we aren't going to go into all the code necessary for building out a test framework, we'll touch on a bit of it here. Our singleton helper class is called AppAuthenticationTests, the implementation is here:


import Foundation
import XCTest

/* We may prompt or ask for permissions so ones to accept */
enum AppTestUIPermission {
    case push
    case location
    case microphone
    case camera
    case calendar
    case contacts
    case bluetooth
    case motion
    case photos
    case reminders
    case pii
    case welcomeBanner
}

/* Indicates the login success or failure */
enum AppTestLoginExpectation {
    case success
    case error
    case locked
}

/* Helper class */
class AppAuthenticationTests {

    /* Nothing */
    fileprivate init() {}

    /**
     Declares the App authentication test provider
     */
    public static var `default`: AppAuthenticationTests = AppAuthenticationTests()
    
    /**
     Launches the application itself.
     
     - Parameter reset: indicates if the app data should be reset
     - Parameter completion: A callback once the app is launched
     */
    func launch(reset: Bool = false, completion: ( (XCUIApplication) -> Void )?) throws {
        
        let app = XCUIApplication()
        if reset {
            app.launchArguments = ["UI_TEST_RESET"]            
        }
        app.launch()

        completion?(app)
    }
    
    /**
     Finds an account authenticate with
     
     - Parameter accountIdentifier: the identifier to look for
     */
    func accountFromConfig(accountIdentifier: String) -> AppTestAccount? {
        
        // Validate accounts exist
        guard let serviceKeys = ForTestingAppConfiguration.default["accounts"] as? [String:Any] else {
            return nil
        }
        
        // Validae this account exists
        guard let accountDetails = serviceKeys[accountIdentifier] as? [String:String] else {
            return nil
            
        }
        
        // The account to return
        let foundAccount = AppTestAccount()
        
        foundAccount.email = accountDetails["email"]! as String
        foundAccount.password = accountDetails["password"]! as String
        
        return foundAccount
        
    }
    
    /**
     Logs the user into the app itself
     
     - Parameter accountIdentifier: the identifier to look for
     - Parameter loginExpectation: the expected login result
     - Parameter test: the test we are operating in
     - Parameter permissions: permissions we will accept
     - Parameter completion: A callback once authenication completes
     */
    func login(accountIdentifier: String, loginExpectation: AppTestLoginExpectation, test: XCTestCase, permissions: [AppTestUIPermission] = [], completion: ( (XCUIApplication, AppTestAccount?) -> Void )?) throws {
        
        // We want to wait for the account provider to finish, we will
        // leverage a semaphore here
        let semaphore = DispatchSemaphore(value: 0)

        // The account we will grab
        var accountOuter: AppTestAccount? = nil

        // Grab the account
        AppTestingCredentialProvider.default.account(accountIdentifier: accountIdentifier) {
            (account) in

            accountOuter = account

            // Done
            semaphore.signal()

        }

        // wait for the semaphore to be signaled
        semaphore.wait()

        // The account should have a value always
        if accountOuter == nil {
            XCTAssertFalse(true)
        }

        // Launch the account, pass the flag into the app
        // The app launch should look for teh argument and reset the
        // app if it contians this argument
        let app = XCUIApplication()
        app.launchArguments = ["UI_TEST_RESET"]
        app.launch()

        // Wait for the button (We use static identifiers for accessability identifiers)
        _ = app.buttons[AppTestIdentifiers.WelcomeScreen.loginButton].waitForExistence(timeout: 10000)
        
        test.screenshot(withFileName: "welcome-screen.png")
        
        // Tap the login button
        app.buttons[AppTestIdentifiers.WelcomeScreen.loginButton].tap()
        
        
        // Wait for the email field
        _ = app.textFields[AppTestIdentifiers.UsernamePasswordLoginScreen.emailTextBox].waitForExistence(timeout: 10000)
        
        // Wait for the password field
        _ = app.secureTextFields[AppTestIdentifiers.UsernamePasswordLoginScreen.passwordTextBox].waitForExistence(timeout: 10000)
        
        test.screenshot(withFileName: "login-screen.png")
        
        // Entry
        let emailtextTextField = app.textFields[AppTestIdentifiers.UsernamePasswordLoginScreen.emailTextBox]
        emailtextTextField.tap()
        emailtextTextField.typeText(accountOuter!.email!)
        
        // Entry
        let passwordtextSecureTextField = app.secureTextFields[AppTestIdentifiers.UsernamePasswordLoginScreen.passwordTextBox]
        passwordtextSecureTextField.tap()
        passwordtextSecureTextField.typeText(accountOuter!.password!)
        
        // Login now
        app.buttons[AppTestIdentifiers.UsernamePasswordLoginScreen.loginButton].tap()
        
        if loginExpectation == .success {
            // This is good, do nothing
        }
        else if loginExpectation == .error {
//            Could not login
            
            test.addUIInterruptionMonitor(withDescription: "Could not login") {
                (alert) -> Bool in
                
                if alert.buttons["OK"].exists {
                    alert.buttons["Allow"].tap()
                }
                
                return true
                
            }
            
        }
        
        // Wait for tab bar
        _ = app.tabBars["tabBar"].waitForExistence(timeout: 10000)
        
        // Permissions
        _ = XCUIApplication().staticTexts.matching(NSPredicate(format: "label CONTAINS 'App Permissions'")).element(boundBy: 0).waitForExistence(timeout: 10000)
        
        // Dismiss permissions popup
        XCUIApplication().navigationBars.children(matching: .button).firstMatch.tap()

        // Done now
        completion?(app, accountOuter)

    }
    
}

Now, once this is in place we have an easy means to authenticate our user in a repeatable manner. Note, Xcode has no means to order tests other than alphabetical order. So, we adopt a pattern of test_Account_A_Step_1, test_Account_B_Step_1, etc to facilitate that.

/**
    Log the user into the app itself
*/
func test_Account_A_Step_1() throws {
    
    // Authenticate the user with the app
    try AppAuthenticationTests.default.login(accountIdentifier: "alreadyConfigured", loginExpectation: .success, test: self, permissions: [ .push, .location, .contacts, .camera ]) {
        (app, account) in
        
        self.account = account
        
        _ = app.tabBars["tabBar"].waitForExistence(timeout: 10000)
        
        // Tap settings (3rd element, not easy to set and access accessability identifiers for tabs)
        app.tabBars["tabBar"].buttons.element(boundBy: 2).tap()
        
        _ = XCUIApplication().staticTexts.matching(NSPredicate(format: "label CONTAINS 'Settings'")).element(boundBy: 0).waitForExistence(timeout: 10000)

    }
    
}

Building for testing

To build the test cases, you'll tap Command + Shift + U within Xcode. This will build the tests and provide you compilation errors. To run the unit tests, tap Command + U and it will run them on the selected device or simulator.

Automating this process is not as simple. To automate, we shall build from the command line. Be warned, this is an exercise in frustration to get it right.

To get started, we will be using xcodebuild's build-for-testing option. This will build the app as well as the test harness. This combination can then be uploaded to AWS Device Farm for automated testing.

Here we provide a sample bash command to run the build. Each of these attributes are self explanatory:

xcodebuild build-for-testing -workspace ${WORKSPACE_PATH} \
    PLATFORM_NAME=iphoneos \
    -destination "generic/platform=iOS" \
    -scheme ${PROJECT_SCHEME} \
    -derivedDataPath $DERIVED_DATA \
    -sdk iphoneos \
    -configuration ${CONFIGURATION} \
    archive -archivePath $ARCHIVE_PATH \
    PROVISIONING_PROFILE_SPECIFIER="$SETTING_PROVISIONING_PROFILE_UDID" \
    PROVISIONING_PROFILE="$SETTING_PROVISIONING_PROFILE_UDID" \
    CODE_SIGN_STYLE="Manual"

Of note, we override the PROVISIONING_PROFILE_SPECIFIER and PROVISIONING_PROFILE, passing the UDID from the provisioning profiles in manually. We do this, because in our CI/CD pipeline, we do not enable automated code signing. We manually set the profiles and code sign to avoid having to "login" on the build server.

Next, you will want to export the archives and resign them:

xcodebuild -exportArchive \
    -archivePath $ARCHIVE_PATH \
    -exportOptionsPlist $OPTIONS_LIST \
    -exportPath $BUILD_DIR/exported

if [ "$RESIGN_ENTITLEMENTS" = "true" ]; then
    # Resign for upload 
    unzip "$BUILD_DIR/exported/${IPA_NAME}" -d "$BUILD_DIR/exported/tmp" 
    #> /dev/null 2>&1
    codesign --entitlements "$CODE_SIGNING_ENTITLEMENTS" --preserve-metadata=entitlements -f -s "$CODE_SIGNING_PROFILE" "${APP_LOCATION}"
    codesign -d --entitlements - "${APP_LOCATION}"
    cd "$BUILD_DIR/exported/tmp"
    rm -rf ../myapp.ipa
    zip -r ../myapp.ipa . 
    #> /dev/null 2>&1
    cd "$THE_PWD"
fi

CODE_SIGNING_PROFILE represents the common name of our Developer/Production Apple signing certificates. Limitations in Xcode and the build system prevent us from just letting Xcode sign them with xcodebuild, so we leverage codesign here to perform the manually.

After this is complete, you will have your myapp.ipa built as well as your myappUITests-Runner.app runner for AWS Device Farm.

Caveats and issues

One issue we ran into was the conflicting automated signing process. We have to disable this for the build server, but want to keep in place for the developers. To do this, we call sed:

sed -i '' 's/CODE_SIGNING_ALLOWED = YES/CODE_SIGNING_ALLOWED = NO/g' ${XCPROJECT_PATH}/project.pbxproj
sed -i '' 's/CODE_SIGNING_REQUIRED = YES/CODE_SIGNING_REQUIRED = NO/g' ${XCPROJECT_PATH}/project.pbxproj

This then allows us to manually perform the code signing ourselves.

AWS Device Farm

We'll walk through in another blog how to deploy this to AWS Device Farm, then wait for testing to complete with a pass/fail status.

Posted by Harold Smith III on Thursday, 04 March 2021
© Monkton, Inc. 2024