Last time we got Vagrant configured to run a single VM with three docker containers: a Selenium Grid hub, a Chrome node, and a Firefox node. This is a good start, but I wanted to configure a Selendroid node to round out the browser selection. That’s when things got a little… messy.
So upon investigation into how the Docker images I was already using were constructed, I discovered a few key points:
- Docker images are defined by Dockerfiles in the same way Vagrant VMs are defined by Vagrantfiles. The format and syntax are totally different, but both are more-or-less human-readable flatfiles that explain how to set up a system. So far so good.
- Dockerfiles are nestable, and in fact, are often nested. The ones from Selenium HQ have a clear hierarchy. This pleased me, because I figured it gave me a nice stable base to work on: my file, like the Chrome and Firefox files, would inherit from the node-base image, but with tweaks specific to Selendroid.
So there’s the top of my Dockerfile:
FROM selenium/node-base:2.53.0 MAINTAINER Bgreen <[redacted]> USER root
I cracked open the Chrome dockerfile and the Dockerfile reference guide and got reading. It looked pretty straightforward at first: just write a bash script, but stick “RUN” in front of it. Spoiler alert: as I started working on my own script, I learned that this was entirely the wrong way to go about writing a dockerfile. Docker has a lot of useful commands other than “RUN”, and it wasn’t long before I was breaking apart my scripts learning how to put the dockerfile together property.
Looking at the Selendroid grid instructions and the Selendroid getting started guide, there were three major steps:
- Install the JDK
- Install the Android SDK
- Install Selendroid and start it in grid mode
Step 1 appeared to be done already, by virtue of the Node-Base dockerfile. This was a dangerous and ultimately wrong assumption, but it was one I worked with for over a day before I realized my mistake. It turns out, the JRE was installed in the base image… under the name openJDK. Nice.
Java installation:
#=============== # JAVA #=============== RUN apt-get update && apt-get install -y openjdk-8-jdk RUN ls -l /usr/lib/jvm/ ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
Now it was time to install the Android SDK. And here I ran into the first massive bunch of problems. I spent several hours fighting with the system, making small tweaks, before I realized I’d accidentally installed Android Studio instead of Android SDK and had to start over.
I originally was going to do a wget followed by a tar, but then I learned about Docker’s ADD command. The ADD command takes a file that is located in the same directory structure as the Dockerfile and moves it into the directory structure inside the container. If the file is a tarball, it will untar the file into a folder as it does, removing the need to write an explicit tar command — a major plus, as tar commands are always annoying to write. I chose to download the tar into the file structure to avoid the network hit and used the ENV command to set ANDROID_HOME the same way I set JAVA_HOME:
#===============
# Android SDK
#===============
ADD android-sdk_r24.4.1-linux.tgz /opt/selenium/
ENV ANDROID_HOME=/opt/selenium/android-sdk-linux
ENV PATH=${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools
However, upon installation, there is no such folder as ANDROID_HOME/platform-tools. This is because it is only created once you fire up the android sdk tool and begin downloading sdks to develop with. So I figured I’d do RUN android update sdk --no-ui. Then I learned you have to accept the license agreement. So I updated my code to the very idomatic RUN yes | android update sdk --no-ui . And well… the results were mildly amusing, but not what I hoped for:
Do you accept the license 'google-gdk-license-35dc2951' [y/n]: Unknown response 'y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y
Dear maintainers of linux software: Don’t break the ‘yes’ command! Sincerely, Bay.
Thanks to Stack Overflow, I found this:
#===============
# Android SDK
#===============
ADD android-sdk_r24.4.1-linux.tgz /opt/selenium/
ENV ANDROID_HOME=/opt/selenium/android-sdk-linux
ENV PATH=${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools
#The following downloads the platform-tools folder
RUN ( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) \
| android update sdk --no-ui --all\
--filter tool,platform-tools,android-23,build-tools-23.0.3
Which got us to our next section: Installing Selendroid. It seemd pretty simple:
#=============== # Selendroid #=============== ADD selendroid-standalone-0.17.0-with-dependencies.jar /opt/selenium/selendroid.jar ADD selendroid-grid-plugin-0.17.0.jar /opt/selenium/selendroid-grid.jar RUN java -jar /opt/selenium/selendroid.jar RUN java -Dfile.encoding=UTF-8 -cp "/opt/selenium/selendroid-grid.jar:/opt/selenium/selendroid.jar" org.openqa.grid.selenium.GridLauncher -capabilityMatcher io.selendroid.grid.SelendroidCapabilityMatcher -role hub -host 127.0.0.1 -port 4444
But it didn’t work. And this, dear reader, is where I was stuck for hours, tearing my hair out in frustration. There were three errors. The first, it seems, is a red herring: there’s nothing actually wrong here (so why is it marked “SEVERE”? Bad usability, Selendroid!)
android: SEVERE: Error executing command: /opt/selenium/android-sdk-linux/bu
ild-tools/23.0.3/aapt remove /tmp/android-driver7255065332626262791.apk META-INF
/NDKEYSTO.RSA
android: org.apache.commons.exec.ExecuteException: Process exited with an er
ror: 1 (Exit value: 1)
android: at org.apache.commons.exec.DefaultExecutor.executeInternal(Defau
ltExecutor.java:377)
android: at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecut
or.java:160)
android: at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecut
or.java:147)
android: at io.selendroid.standalone.io.ShellCommand.exec(ShellCommand.ja
va:49)
android: at io.selendroid.standalone.android.impl.DefaultAndroidApp.delet
eFileFromWithinApk(DefaultAndroidApp.java:112)
android: at io.selendroid.standalone.builder.SelendroidServerBuilder.dele
teFileFromAppSilently(SelendroidServerBuilder.java:133)
android: at io.selendroid.standalone.builder.SelendroidServerBuilder.resi
gnApp(SelendroidServerBuilder.java:148)
android: at io.selendroid.standalone.server.model.SelendroidStandaloneDri
ver.initApplicationsUnderTest(SelendroidStandaloneDriver.java:172)
android: at io.selendroid.standalone.server.model.SelendroidStandaloneDri
ver.<init>(SelendroidStandaloneDriver.java:94)
android: at io.selendroid.standalone.server.SelendroidStandaloneServer.in
itializeSelendroidServer(SelendroidStandaloneServer.java:63)
android: at io.selendroid.standalone.server.SelendroidStandaloneServer.<i
nit>(SelendroidStandaloneServer.java:52)
android: at io.selendroid.standalone.SelendroidLauncher.launchServer(Sele
ndroidLauncher.java:65)
android: at io.selendroid.standalone.SelendroidLauncher.main(SelendroidLa
uncher.java:117)
The second drove me nuts because it outright lied to me. The file it complained about not having was right there, with executable permissions, owned by root (which I was operating as)!
INFO: Executing shell command: /opt/selenium/android-sdk-linux/build-tools/23.0.
3/aapt remove /tmp/android-driver2951817352746346830.apk META-INF/MANIFEST.MF
←[0m←[91mJul 06, 2016 8:22:29 AM io.selendroid.standalone.io.ShellCommand exec
SEVERE: Error executing command: /opt/selenium/android-sdk-linux/build-tools/23.
0.3/aapt remove /tmp/android-driver2951817352746346830.apk META-INF/MANIFEST.MF
java.io.IOException: Cannot run program "/opt/selenium/android-sdk-linux/build-t
ools/23.0.3/aapt" (in directory "."): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
It turns out that the “No such file or directory” was coming from inside the executable named, not referring to that executable. I needed to install some dependencies I’d missed, which I did using RUN apt-get update && apt-get install -y lib32stdc++6 lib32z1.
Now I had a different problem involving the keytool:
Jul 07, 2016 6:17:04 AM io.selendroid.standalone.io.ShellCommand exec
INFO: Executing shell command: /usr/lib/jvm/java-8-openjdk-amd64/bin/keytool -genkey -v -keystore /home/seluser/.android/debug.keystore -storepass android -alias androiddebugkey -keypass android -dname CN=Android Debug,O=Android,C=US -storetype JKS -sigalg MD5withRSA -keyalg RSA -validity 9999
Jul 07, 2016 6:17:06 AM io.selendroid.standalone.io.ShellCommand exec
SEVERE: Error executing command: /usr/lib/jvm/java-8-openjdk-amd64/bin/keytool -genkey -v -keystore /home/seluser/.android/debug.keystore -storepass android -alias androiddebugkey -keypass android -dname CN=Android Debug,O=Android,C=US -storetype JKS -sigalg MD5withRSA -keyalg RSA -validity 9999
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:377)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)
at io.selendroid.standalone.io.ShellCommand.exec(ShellCommand.java:49)
at io.selendroid.standalone.builder.SelendroidServerBuilder.signTestServer(SelendroidServerBuilder.java:277)
at io.selendroid.standalone.builder.SelendroidServerBuilder.resignApp(SelendroidServerBuilder.java:154)
at io.selendroid.standalone.server.model.SelendroidStandaloneDriver.initApplicationsUnderTest(SelendroidStandaloneDriver.java:172)
at io.selendroid.standalone.server.model.SelendroidStandaloneDriver.<init>(SelendroidStandaloneDriver.java:94)
at io.selendroid.standalone.server.SelendroidStandaloneServer.initializeSelendroidServer(SelendroidStandaloneServer.java:63)
at io.selendroid.standalone.server.SelendroidStandaloneServer.<init>(SelendroidStandaloneServer.java:52)
at io.selendroid.standalone.SelendroidLauncher.launchServer(SelendroidLauncher.java:65)
at io.selendroid.standalone.SelendroidLauncher.main(SelendroidLauncher.java:117)
This simply claims that the process has exited with a failure; neither the stack trace nor the error message are useful. When I tried to execute that command myself, it complained about an invalid command-line option, which to me indicated that there needed to be quotes around the OU. I didn’t have access to change that, though I could generate my own keystore. However, I also had the third error to deal with:
SEVERE: Error building server: io.selendroid.standalone.exceptions.ShellCommandE xception: Error executing shell command: /usr/lib/jvm/java-8-openjdk-amd64/bin/j arsigner -sigalg MD5withRSA -digestalg SHA1 -signedjar /tmp/resigned-android-dri ver694668080026603748.apk -storepass android -keystore /root/.android/debug.keys tore /tmp/android-driver694668080026603748.apk androiddebugkey ←[0m←[91mException in thread "main" ←[0m←[91mjava.lang.RuntimeException: io.sele ndroid.standalone.exceptions.ShellCommandException: Error executing shell comman d: /usr/lib/jvm/java-8-openjdk-amd64/bin/jarsigner -sigalg MD5withRSA -digestalg SHA1 -signedjar /tmp/resigned-android-driver694668080026603748.apk -storepass a ndroid -keystore /root/.android/debug.keystore /tmp/android-driver69466808002660 3748.apk androiddebugkey←[0m←[91m ←[0m←[91m at io.selendroid.standalone.server.model.SelendroidStandaloneDri ver.initApplicationsUnderTest(SelendroidStandaloneDriver.java:175)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.server.model.SelendroidStandaloneDri ver.<init>(SelendroidStandaloneDriver.java:94)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.server.SelendroidStandaloneServer.in itializeSelendroidServer(SelendroidStandaloneServer.java:63)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.server.SelendroidStandaloneServer.<i nit>(SelendroidStandaloneServer.java:52)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.SelendroidLauncher.launchServer(Sele ndroidLauncher.java:65)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.SelendroidLauncher.main(SelendroidLa uncher.java:117)←[0m←[91m ←[0m←[91mCaused by: io.selendroid.standalone.exceptions.ShellCommandException: E rror executing shell command: /usr/lib/jvm/java-8-openjdk-amd64/bin/jarsigner -s igalg MD5withRSA -digestalg SHA1 -signedjar /tmp/resigned-android-driver69466808 0026603748.apk -storepass android -keystore /root/.android/debug.keystore /tmp/a ndroid-driver694668080026603748.apk androiddebugkey←[0m←[91m ←[0m←[91m at io.selendroid.standalone.io.ShellCommand.exec(ShellCommand.ja va:56)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.builder.SelendroidServerBuilder.sign TestServer(SelendroidServerBuilder.java:296)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.builder.SelendroidServerBuilder.resi gnApp(SelendroidServerBuilder.java:154)←[0m←[91m ←[0m←[91m at io.selendroid.standalone.server.model.SelendroidStandaloneDri ver.initApplicationsUnderTest(SelendroidStandaloneDriver.java:172)←[0m←[91m ←[0m←[91m ... 5 more←[0m←[91m ←[0m←[91mCaused by: io.selendroid.standalone.exceptions.ShellCommandException: ← [0m←[91m
This executable was missing altogether, and rightly so: I couldn’t find it on the filesystem. And that was when I realized my “JDK” was a JRE. Installing the proper JDK, shown above, took care of both those errors. Lessons learned.
(As a side note: one thing I really like about Docker is the caching strategy. It only seemed to re-install the Android SDKs if I changed that step or an earlier one, preferring the cached version when I was working on the later steps — something that saved me a ton of time and frusturation.)
So now we have a working (sort of) Dockerfile! Two problems left:
- There are no emulators available for Selendroid. Oops!
- The dockerfile starts Selenium Server and then hangs forever, because it doesn’t return from that. TBD
Now, normally you’d fire up the Android Studio GUI and create yourself an AVD file for Selendroid to use, but I’m doing it all the hard way, via the command line in my Docker file. The first thing I have to do is download an ABI to make an AVD out of:
#The following downloads the platform-tools folder and the ABI
RUN ( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) \
| android update sdk --no-ui --all\
--filter tool,platform-tools,android-23,sys-img-x86-android-23,build-tools-23.0.3
And then, I create an ABI out of it. Note that we are asked one question I couldn’t get rid of using command-line flags, so I used the “echo” command to send a newline and accept the default option (no hardware profile):
#Create AVD. Echo sends a newline and nothing else here, for accepting the default to the question asked. RUN echo | android create avd --name Default --target android-23 --abi x86
Now before it hangs forever, it clearly states:
android: INFO: Shell command output android: --> android: Available Android Virtual Devices: android: Name: Default android: Path: /root/.android/avd/Default.avd android: Target: Android 6.0 (API level 23) android: Tag/ABI: default/x86 android: Skin: WVGA800 android: <-- android:
Progress made!
It was then that I began to really dig into the nitty gritty about how the base image started selenium server. It seems that Selenium HQ chose to use a shell script to wrangle a series of environment variables; since they know the product better than I do, I went down the same path and created my own version of this script, modified for Selendroid:
#!/bin/bash
source /opt/bin/functions.sh
java ${JAVA_OPTS} -jar /opt/selenium/selendroid.jar -keystore /home/seluser/debug.keystore &
NODE_PID=$!
curl -H "Content-Type: application/json" -X POST --data /opt/selenium/config.json http://$HUB_PORT_4444_TCP_ADDR:$HUB_PORT_4444_TCP_PORT/grid/register
trap shutdown SIGTERM SIGINT
wait $NODE_PID
You can see how much shorter it is; Selendroid is weird in that it doesn’t seem to take most of the config options required, and requires me to manually curl the config to the hub node.
A note: be sure to save this with unix line endings. You’ll get a very strange error reading [8] System error: no such file or directory if you do not, and that’s awful to try and figure out because it’s so generic. I also got really comfortable SSHing into the underlying VM to run “docker rm -f” at this point, because the container was building fine but erroring out, so now it had name conflicts when I tried to rebuild.
At this point, the container built, but did not run successfully. This means our debugging strategy changes from inspecting the vagrant output carefully to reading the container’s logs using “docker logs selenium-selendroid”. I now found the resurgence of several of our older problems, which was incredibly frustrating; I was sure that those had been resolved already. It was all stupid little fixes, like generating the keystore in a known location as root and then passing it into selendroid (an approach I had tried earlier but found unnecessary, and one that is already accounted for in the above shell script) or making sure to generate the AVD as the same user that would run selendroid so it ended up in the right location.
At this point we have a working Selendroid docker container! But….. it doesn’t register with the hub correctly. Also, the hub’s web console isn’t accessible, making debugging troubling. At this point, I’m taking a breather, because it’s been 3 days and I’m frustrated. We’ll return in part 3 to make this fully functional. Feel free to comment if you have tips and tricks!
Our current dockerfile:
FROM selenium/node-base:2.53.0
MAINTAINER Bgreen <[redacted email]>
USER root
#===============
# JAVA
#===============
RUN apt-get update && apt-get install -y openjdk-8-jdk
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
#Keystore generation is broken somehow?
RUN /usr/lib/jvm/java-8-openjdk-amd64/bin/keytool -genkey -v -keystore /home/seluser/debug.keystore \
-storepass android -alias androiddebugkey -keypass android \
-dname CN="Android Debug,O=Android,C=US" -storetype JKS -sigalg MD5withRSA \
-keyalg RSA -validity 9999
RUN chown seluser /home/seluser/debug.keystore
#===============
# Android SDK
#===============
ADD android-sdk_r24.4.1-linux.tgz /opt/selenium/
ENV ANDROID_HOME=/opt/selenium/android-sdk-linux
ENV PATH=${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools
#The following downloads the platform-tools folder and the ABI
RUN ( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) \
| android update sdk --no-ui --all\
--filter tool,platform-tools,android-23,sys-img-x86-android-23,build-tools-23.0.3
#========================
# Selenium Configuration
#========================
COPY config.json /opt/selenium/config.json
#========================
# Extra libraries
#========================
RUN apt-get update && apt-get install -y lib32stdc++6 lib32z1
RUN apt-get install -y curl
#===============
# Selendroid
#===============
ADD selendroid-standalone-0.17.0-with-dependencies.jar /opt/selenium/selendroid.jar
ADD selendroid-grid-plugin-0.17.0.jar /opt/selenium/selendroid-grid.jar
COPY startSelendroid.sh /opt/bin/
RUN chmod +x /opt/bin/startSelendroid.sh
#===============
# Start the grid
#===============
USER seluser
#Create AVD. Echo sends a newline and nothing else here, for accepting the default to the question asked.
RUN echo | android create avd --name Default --target android-23 --abi x86
CMD ["/opt/bin/startSelendroid.sh"]