Running Android Tests in Docker

As part of the project I'm currently engaged on, my team is writing automated tests for an application which has a web interface, and also two mobile apps, one for Android, and one for iOS. As part of the project, we've built a test automation pipeline which runs our tests against our application to ensure changes we're making don't impact other tests. Yes, we're testing our tests. One of the challenges we ran into was ensuring we could verify our Android and iOS tests still worked in a timely fashion. We tried multiple options including running tests on SauceLabs but this consumed resources too quickly to be effective due to the multiple threads of the pipeline (we decided to save SauceLabs bandwidth for our actual testing). The solution we eventually found worked best was to create our own emulators inside of a Docker container to test on.

The Setup

If you're not familiar with Docker, check out some of our many excellent posts on it. It also works great for testing. Essentially, instead of spinning up a web application inside a container, we wanted to spin up an Android emulator, running our app. We would then connect to the container, the same way we would a tethered physical device, a local emulator, or something in the cloud. This made it incredibly cheap (and fast) to parallelize our testing. All we needed to do was launch multiple Docker containers, and then connect using ADB to each one.

The Dockerfile

Unfortunately, building the Docker container wasn't straightforward. We needed a Docker container with Android SDK installed, an Android VM created, and Appium. Because I had gotten everything running just fine locally (Ubuntu), I decided to start with a similar base image (maven:3.5.2-jdk-8). We then installed the ADK, set up the emulator, and installed Appium. Finally, we coped over the APK and test suite. We soon discovered that the base image didn't have kvm set up, so we needed to enable this. Not an easy task, which unfortunately required a manual step. As a result, I decided to split this into two Docker files, a new base one with kvm setup, and one with all of the installs.

DockerfileKVM

This Docker container was relatively straightforward, however, not quite simple. I built it once, uploaded it to my local Docker repository, and then built on top of it for the Docker Android container. All I did was install kvm, and manually copy over the proper lib modules. The Dockerfile looked like:

FROM maven:3.5.2-jdk-8
#debian based

RUN apt-get update -qqy \
    && apt-get -qqy install libglu1 qemu-kvm libvirt-dev virtinst bridge-utils msr-tools kmod \
    && wget -q http://security.ubuntu.com/ubuntu/pool/main/c/cpu-checker/cpu-checker_0.7-0ubuntu7_amd64.deb \
    && dpkg -i cpu-checker_0.7-0ubuntu7_amd64.deb \
    && apt-get install -f \
    && kvm-ok

From there, I built and tagged the Docker file. I then ran it, logged in (-it), and copied over my machine's lib modules (/lib/modules). Please note, your mileage might vary based on your machine's kernel and version. Then I pushed this image into my local Docker repository.

DockerfileAndroid

This Docker container had a lot more steps, but ultimately wasn't too much more complex. I set up my Android SK, loaded up my emulator, installed Appium, and copied over our APK and tests. The Dockerfile looked like this:

FROM kvm:maven-3.5.2-jdk-8#tag we gave to DockerfileKVM
# debian based

ENV UDIDS=""

#=====================
# Install android sdk
#=====================
ARG ANDROID_SDK_VERSION=4333796
ENV ANDROID_SDK_VERSION=$ANDROID_SDK_VERSION
ARG ANDROID_PLATFORM="android-25"
ARG BUILD_TOOLS="26.0.0"
ENV ANDROID_PLATFORM=$ANDROID_PLATFORM
ENV BUILD_TOOLS=$BUILD_TOOLS

# install adk
RUN mkdir -p /opt/adk \
    && wget -q https://dl.google.com/android/repository/sdk-tools-linux-${ANDROID_SDK_VERSION}.zip \
    && unzip sdk-tools-linux-${ANDROID_SDK_VERSION}.zip -d /opt/adk \
    && rm sdk-tools-linux-${ANDROID_SDK_VERSION}.zip \
    && wget -q https://dl.google.com/android/repository/platform-tools-latest-linux.zip \
    && unzip platform-tools-latest-linux.zip -d /opt/adk \
    && rm platform-tools-latest-linux.zip \
    && yes | /opt/adk/tools/bin/sdkmanager --licenses \
    && /opt/adk/tools/bin/sdkmanager "emulator" "build-tools;${BUILD_TOOLS}" "platforms;${ANDROID_PLATFORM}" "system-images;${ANDROID_PLATFORM};google_apis;armeabi-v7a" \
    && echo no | /opt/adk/tools/bin/avdmanager create avd -n "Android" -k "system-images;${ANDROID_PLATFORM};google_apis;armeabi-v7a" \
    && mkdir -p ${HOME}/.android/ \
    && ln -s /root/.android/avd ${HOME}/.android/avd \
    && ln -s /opt/adk/tools/emulator /usr/bin \
    && ln -s /opt/adk/platform-tools/adb /usr/bin
ENV ANDROID_HOME /opt/adk

#====================================
# Install latest nodejs, npm, appium
#====================================
ARG NODE_VERSION=v8.11.3
ENV NODE_VERSION=$NODE_VERSION
ARG APPIUM_VERSION=1.9.1
ENV APPIUM_VERSION=$APPIUM_VERSION

# install appium
RUN wget -q https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64.tar.xz \
    && tar -xJf node-${NODE_VERSION}-linux-x64.tar.xz -C /opt/ \
    && ln -s /opt/node-${NODE_VERSION}-linux-x64/bin/npm /usr/bin/ \
    && ln -s /opt/node-${NODE_VERSION}-linux-x64/bin/node /usr/bin/ \
    && ln -s /opt/node-${NODE_VERSION}-linux-x64/bin/npx /usr/bin/ \
    && npm install -g appium@${APPIUM_VERSION} --allow-root --unsafe-perm=true \
    && ln -s /opt/node-${NODE_VERSION}-linux-x64/bin/appium /usr/bin/

EXPOSE [4723,2251,5555]
CMD ["docker-entrypoint.sh"]

What you'll notice is there was one last piece to this puzzle, the docker-entrypoint, which we used to launch the emulator, and get everything connected with Appium. This luckily was pretty easy, once you knew what to do.

docker-entrypoint.sh

#!/bin/bash

# launch the emulator
exec /opt/adk/tools/emulator -avd Android -no-audio -no-window &

# setup appium
while [ -z $udid ]; do
    udid=`adb devices | grep emulator | cut -f 1`
done
exec appium -p 4723 -bp 2251 --default-capabilities '{"udid":"'${udid}'"}' &

And that was it, all we needed to do, was launch our tests.

Test Execution

Without getting into the specifics of the project, we could do that either locally, or through another Docker container, but the simplest way was definitely locally, using Maven. We could do this by simply specifying the Docker IP, the same way you would if you have the emulator running locally, or the device tethered. Swapping ports is simple enough as well.

Final Thoughts

We did run into one last issue, which was trying to run this all in AWS. We've been trying to keep our pipelines as fast moving as possible, and a large portion of that means dynamically provisioned machines in AWS when we need them, it greatly increased our throughput. Unfortunately, AWS machines don't support nested KVM (yes, I'm aware of using metal, but we wanted to avoid that cost increase). Unfortunately, that meant a large portion of this couldn't be used in this fashion. Stay tuned for the next blog post, in which I get into the work around to solve this issue.

Find this useful? Got stuck? As always, please leave some comments below.