Xcode Server + Ansible: Scalable and Programmable iOS CI/CD Infrastructure

One of the complaints about Xcode Server is it's not scalable to be used at a large-scale or enterprise level. However, if you have the skills to understand and manage an infrastructure for iOS deployment, then scalability can be easily achieved for small, medium, or large iOS teams using a mixture of iOS and DevOps toolkits. We have achieved it in the past in my previous role at Photobox Group using the combination of tools Ansible + Fastlane + TeamCity. You can read my blog post on the Moonpig Engineering blog here to learn how we achieved this.

In this post, we will see how learning a little bit of DevOps tools like Ansible and Apple's CI solution Xcode Server can help to manage large-scale iOS CI/CD infrastructure without any pain.

Background

Continuous Integration practices for iOS development aren't great as compared to the web world because of some unique challenges specific to iOS development. It's essential to understand the entire iOS ecosystem to setup and manage CI/CD infrastructure for the team. Some Senior/Lead iOS developers or iOS Tech Architects/Lead are great at the understanding whole stack of Apple developer tools, technologies, and infrastructure and implementing it in their projects, but most of the iOS developers in the industry at the moment are just iOS developers. They are good at writing Swift or Objective-C to build UI for applications inside Xcode; however, when it comes to understanding the infrastructure, they suck. They don't properly understand the underlying tools and technologies used by Xcode under the hood and that's why they became totally incapable of setting up or managing CI/CD practices for iOS apps. As an effect of this, they always have to depend on local Xcode or someone from the DevOps team to manage the infrastructure stack for them, or CI/CD is handled by highly expensive third-party services. It means developers are not full-stack and do not understand their own tools and technologies. Unfortunately, the world of iOS development is full of these kinds of amateurs and half-arsed developers. Continuous Integration won't be successful without skilled engineers on the team. In my previous post, I wrote about the top five reasons for CI failures. Those are wrong selection of CI server, CI amature engineers, lack of Infrastructure as Code, poor quality build servers, and lack of support from the team.

iOS Infrastructure Automation Challenges

On top of iOS engineers' inability to understand the entire iOS ecosystem, there are some traditional challenges in automation of iOS ecosystem. Some of them are imposed by Apple's proprietary software, and others are due to lack of tooling to automate iOS and macOS infrastructure. Some of the common challenges are

• Lack of a configuration management tool that can be used to automate the infrastructure. Chef, Ansible, and Puppet are not designed to work with macOS servers.
• Lack of an official package management system to provision and automate software installation. We have to use third-party solutions like HomeBrew
• Lack of an official dependency management tools. We have to use Carthage or Cocoapods to manage dependencies. Swift Package Manager is not ready to use with iOS.
• Lack of an official build tool. We have to use tools like Fastlane to automate iOS development tasks.

As a result of these challenges, nobody wanted to spend the time to explore a clean and automated solution to set up iOS infrastructure for smooth Continuous Delivery, which results in the following situations:

• Manual setup of CI servers, which takes days for engineers to get things working on all build servers.
• Flaky build automation, which produces inconsistent results.
• The software configuration on local and build machines is different.
• Continuous Integration is managed by a third-party provider and they need to be contacted them in case of build errors.
• The team loses trust in CI and start using the local machine for testing and release.

These symptoms are obstacles to the best application development practices. We can solve some of these issues by knowing a little bit about configuration management tools and proper continuous integration tools. Now, we will see how the combination of Xcode Server and Ansible can help to set up large-scale continuous integration and manage it effectively.

Xcode Server

Xcode Server is a native Continuous Integration service developed by Apple. Xcode Server is one of the easiest ways to get started with Continuous Integration for iOS applications. It just needs Xcode installation and permission to access the Apple developer portal on macOS servers. If you use other solutions like Jenkins or TeamCity, you will need to install other dependencies like Java and other related software which have nothing to do with iOS development. Apple still has old documentation for Xcode Server at the time of writing this post, but I wrote the setup instructions in my previous blog post here. In order to set up the Continuous Integration server, we just need to install the following on the build machines:

• Xcode
• Xcode Command Line tools
• macOS default settings
• Ruby and Homebrew (optional)

We can manually install that software on each machine, but it will time-consuming if we have multiple macOS machines. We need to make sure that Xcode and other tool versions are consistent on all the servers. Now, we need to have configuration management to manage servers and software versions on those servers.

Provisioning Xcode Server With Ansible

There are various configuration management tools on the market, like Chef, Puppet, and Salt, but Ansible is very easy to learn and use. Although Ansible is designed for Linux, there are enough modules that can be used with macOS as well. This is not an Ansible tutorial, but you need to read a bit of documentation to get started with Ansible here.

In order to provision Xcode Server, we need the following things:

• Xcode.xip file downloaded from Apple Developer Portal
• Ansible installed on a remote macOS machine; Ansible will then ssh into a different macOS and install all available software.

Additionally, we can install other tools like Ruby and Homebrew using Ansible, if needed.

Provisioning Xcode Installation

In order to install Xcode, we need to have the .xip file downloaded from Apple Developer Portal and put it on the Ansible remote machine to copy over to ssh machines. There are various other things to be installed as part of the Xcode installation:

• Archive XIP file and move Xcode app to /Applications
• Accept Licence agreement
• Install Xcode Command Line tools and other mobile development packages
• Enable developer tools and install additional components
• Set Xcode Developer directory

We can achieve all these tasks using Ansible.

---
- name: Check if Xcode is already installed
  stat: path=~/Library/Developer/Xcode/
  register: xcode_dir

- name: test whether xcode was uploaded earlier
  stat: path=/Users/{{ ansible_ssh_user }}/{{ xcode_src }}
  register: xcode_dl_path

- name: upload xcode
  copy: src={{ xcode_src }} dest=/Users/{{ ansible_ssh_user }}/
  when: xcode_dl_path.stat.exists == False

- name: check that the xcode archive is valid
  command: "pkgutil --check-signature ~/{{ xcode_src }} | grep \"Status: signed Apple Software\""
  when: xcode_dir.stat.exists == False

- name: Install Xcode from XIP file Location
  command: bash -c 'open -FWga "Archive Utility" --args /Users/{{ ansible_ssh_user }}/{{ xcode_src }}'
  when: xcode_dir.stat.exists == False

- name: Move Xcode To Application
  command: bash -c 'mv ~/Xcode*.app /Applications/'
  when: xcode_dir.stat.exists == False


- name: accept license agreement
  command: bash -c "sudo /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -license accept"

- name: install mobile device package
  command: bash -c " sudo installer -pkg /Applications/Xcode.app/Contents/Resources/Packages/MobileDevice.pkg -target /"

- name: install mobile device development package
  command: bash -c "sudo installer -pkg /Applications/Xcode.app/Contents/Resources/Packages/MobileDeviceDevelopment.pkg -target /"

- name: install Xcode System Resources package
  command: bash -c "sudo installer -pkg /Applications/Xcode.app/Contents/Resources/Packages/XcodeSystemResources.pkg -target /"

- name: install additional xcode components
  command: bash -c "sudo installer -pkg /Applications/Xcode.app/Contents/Resources/Packages/XcodeSystemResources.pkg -target /"

- name: check on the status of developer mode
  command: DevToolsSecurity -status
  become: yes
  become_method: sudo
  register: dev_tools_security_status


- name: enable developer mode
  command: DevToolsSecurity -enable
  become: yes
  become_method: sudo
  when:
    - "'disabled' in dev_tools_security_status.stdout"
    - xcode_dir.stat.exists == False

- name: Set Xcode Developer Directory
  command: bash -c "sudo xcode-select -s /Applications/Xcode.app/Contents/Developer/"

Note that the code above is the example of an Ansible task; we have to set up some variables to configure users and directories.

Provisioning Xcode Server

Now we can install Xcode as part of the task mentioned above. The next thing is to control the Xcode Server using Ansible. Xcode Server has command line tools, xcscontrol, to manage the Xcode Server. We can start, stop, reset, and restart Xcode Server using this utility. We can write an Ansible task like this:

---

  - name:  Start Xcode Server with given user
    command: bash -c 'sudo xcrun xcscontrol --initialize --build-service-user {{xcode_server_user}}'

  - name:  COnfigure Xcode Server Certificate
    command: bash -c 'sudo xcrun xcscontrol --configure-ota-certificate'

  - name:  Configure Xcode Server timeout
    command: bash -c 'sudo xcrun xcscontrol --configure-integration-timeout 3600'

  - name:  Check Xcode Server Health
    command: bash -c 'sudo xcrun xcscontrol --health'

With this simple configuration, we can easily manage Xcode Servers on multiple macOS servers.

Xcode Server: Ansible Role

I have created an Ansible role for Xcode Server and published to Ansible Galaxy. You can find this role here.

This role is very configurable and can be used for setting up multiple macOS servers as per your needs. You can find detailed information in the README file.

You can see all the variables used for this role and turn them ON/OFF or set them as per your needs. You can optionally set up Ruby and Homebrew if needed.

vars:
    clean_xcode: yes
    clean_rvm: no
    clean_homebrew: no

    configure_xcode: yes
    configure_xcodeserver: yes
    configure_macos_defaults: yes
    configure_ruby_rvm: no
    configure_homebrew: no

    xcode_src: Xcode_9.1.xip

    xcode_server_user: shashi
    ansible_ssh_user: shashi

How to Use an Xcode Server Ansible Role

Imagine, you have a fresh Mac with fresh macOS installed. You can set up all your Xcode Server for CI by creating a Playbook for this role. You can set up config variables as per your needs. Assuming you have installed Ansible, we can download the role by running this command:

$ ansible-galaxy install Shashikant86.XcodeServer

Now we have to create our own playbook for this role by setting variables. We can use the default variables here. Now make a new directory called xcode_server and create a files directory and xcs_playbook.yml file.

$ mkdir xcode_server
$ mkdir XcodeServer/files
$ touch xcs_playbook.yml

Now place your Xcode_9.x.xip file inside the files directory and insert the following into the playbook file:

---
- hosts: localhost
  connection: local

  vars:
    clean_xcode: yes
    clean_rvm: no
    clean_homebrew: no

    configure_xcode: yes
    configure_xcodeserver: yes
    configure_macos_defaults: yes
    configure_ruby_rvm: no
    configure_homebrew: no

    xcode_src: Xcode_9.1.xip

    xcode_server_user: shashi
    ansible_ssh_user: shashi

  roles:
    - Shashikant86.XcodeServer

Change ansible_ssh_user and xcode_server_user to your username and Feel free to set variables as per your need. You can also specify your inventory to enable this playbook to run on multiple hosts. Now execute this playbook using the following command:

$ ansible-playbook xcs_playbook.yml

Watch your Mac Mini servers getting set up with Xcode Server for iOS Continuous Integration.

When to Use Xcode Server and Ansible

There are situations when we want to run Ansible playbook on servers; some common situations are:

• New Xcode Versions to be installed
• Resetting the Rebuild CI environment at regular intervals
• Cleaning up the environment to fix code signing issues
• Resetting the server to save disk space

Ideally, we should run the Ansible playbook regularly to check if anything has changed.

Benefits

Some of the benefits of using configuration management tools with Xcode Server are

• Consistent setup all across the servers
• Infrastructure as a code
• Destroy and build CI servers whenever needed
• Save hours of engineers manually setting up CI server machines
• Eliminate the cost of expensive third-party CI services
• No need to have dedicated DevOps resources to manage iOS CI/CD infrastructure
• Understanding of the iOS ecosystem and the confidence to deal with any issues
• No need to use third-party build automation tools like Fastlane as Xcode Server can handle almost everything needed for CI/CD

You might see several other benefits and increased confidence and trust in Continuous Integration.

Conclusion

Using the combination of DevOps tools and Apple's native CI/CD tools, we can achieve scalable and programmable infrastructure for our iOS apps. The Xcode Server and Ansible combination can work well together to achieve Continuous Delivery of iOS apps without any pain. Are you using any configuration management tools for iOS CI/CD infrastructure? What are your experiences with Xcode Server? Please let me know what you think about this approach- any ideas or suggestions would be great.