Bay's Blog

Selenium Grid on Docker and Vagrant: Part 1

2016-07-01 10 min read Longer Tales SQA

I’ve been putting together a quick proof-of-concept here at work about how we could use Docker to run a Selenium Grid. I’m not sure we’ll go that route, but I was curious how it could be done.

One of the main advantages of doing this sort of rough proof in Vagrant is that it becomes very portable. At the end of the day, I have a mini testing cloud I can run my tests against — and any member of my team can check out a few files and have their own mini testing cloud. It’s pretty neat, and it means that even if we decide against implementing this on a larger scale, I get some value out of it in years to come.

I’ll assume you’re passingly familiar with vagrantalready, and have at least read the getting started docs. I was an absolute newbie to Docker when I started, so this discussion will assume no prior Docker knowledge. If you do know Docker, feel free to tell me how wrong I am in the comments section 🙂

I went down the path of using a Docker Provisioner for an hour or so before I realized that was the wrong path: I want to use the Docker Provider. The way to think of this is like a series of super tiny VMs which have to live on a giant VM in much the same way lily pads decorate the top of a pond. Docker as a provider can manage the whole set of lily pads and knows nothing about the pond; Docker as a provisioner can add a lily pad to your existing pond ecosystem without making as many waves.

So we have a secret VM, and a series of explicit Docker containers. Now, this was a proof of concept, but I actually care what OS that secret VM uses; if it’s not compatible with RHEL 6, then I won’t be able to make a good case for it in the end. Lots of shiny new toys only work on Ubuntu, after all.

Vagrant by default picks the tiniest OS it can find, just enough to support the containers on top. Usually that’s a good decision, but as we just discussed I want that secret VM to be CentOS 6 instead. This is where things get a little difficult: to specify your own VM to use, you give Vagrant another Vagrantfile.

Because Vagrantfiles need to be called “Vagrantfile”, you have to create a subfolder; mine is “dockerHost/Vagrantfile” for lack of better terminology. I also wanted to limit the amount of RAM Virtualbox would eat up, and enable networking (this will become important later). Try to think through what you’ll need, because every time you need to destroy and recreate this box, it’s going to suck and feel like it takes forever.

My dockerHost vagrantfile:

Vagrant.configure("2") do |config|
    # Every Vagrant development environment requires a box. You can search for
    # boxes at https://atlas.hashicorp.com/search.
    config.vm.box = "bento/centos-6.7"


    # Create a forwarded port mapping which allows access to a specific port
    # within the machine from a port on the host machine. In the example below,
    # accessing "localhost:8080" will access port 80 on the guest machine.
    config.vm.network "forwarded_port", guest: 80, host: 8088

    # Create a private network, which allows host-only access to the machine
    # using a specific IP.
    config.vm.network "private_network", ip: "192.168.33.10"


    # Provider-specific configuration so you can fine-tune various
    # backing providers for Vagrant. These expose provider-specific options.
    config.vm.provider "virtualbox" do |vb|
        # Customize the amount of memory on the VM:
        vb.memory = "1024"

        # enable network features
        vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
        vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
    end

    # Docker provisioner will install docker when given no options
    # This prepares the box to be a base image for the docker-related script
    config.vm.provision "docker"

    # The following line terminates all ssh connections. Therefore
    # Vagrant will be forced to reconnect.
    # That's a workaround to have the docker command in the PATH
    config.vm.provision "shell", inline:
        "ps aux | grep 'sshd:' | awk '{print $2}' | xargs kill"

    # Below are to fix an issue with docker provisioning
    config.vm.provision "shell", inline: "sudo chmod 777 /var/lib/docker "

end

A few things to point out:

  • I needed to enable network features, but not right away; that’s a later addition, for things we won’t get to until part 2.
  • Docker as a provisioner comes back into the mix in a rather unintuitive way. When given no images to load or dockerfiles to build, the provisioner simply installs Docker and exits. This makes a very easy, platform-agnostic way to install Docker. I was halfway through a shell script to do the provisioning when I learned this, and frankly, I just didn’t want to bother learning how to install Docker. On the other hand, this step takes forEVER to run, so you don’t want to recreate the VM often.
  • Docker isn’t available as a command until the ssh has been kicked out and reconnected. This is probably a Vagrant bug. I found the workaround listed above and stopped looking, because I didn’t want to spend more time on this than necessary.
  • The last line isn’t needed until part 2 of this series, but if you plan to build your own docker images, you probably want it.

When this is run with “vagrant up”, it creates a VM that has Docker installed. You probably want to test this before moving on, but once you do, you won’t need to explicitly start this again.

So let’s go to our upper-level Vagrantfile. I looked around and very quickly found some Docker images I want to use out of the box: https://github.com/SeleniumHQ/docker-selenium. The first one to get running is the hub node, the central node for our grid. We configure Docker like any other provider:

Vagrant.configure("2") do |config|
    # The most common configuration options are documented and commented below.
    # For a complete reference, please see the online documentation at
    # https://docs.vagrantup.com.

    # Skip checking for an updated Vagrant box
    config.vm.box_check_update = false

    # Always use Vagrant's default insecure key
    config.ssh.insert_key = false

    # Disable synced folders (prevents an NFS error on "vagrant up")
    config.vm.synced_folder ".", "/vagrant", disabled: true

    # Configure the Docker provider for Vagrant
    config.vm.provider "docker" do |docker|

        # Define the location of the Vagrantfile for the host VM
        # Comment out this line to use default host VM
        docker.vagrant_vagrantfile = "dockerHost/Vagrantfile"

        # Specify the Docker image to use
        docker.image = "selenium/hub"

        # Specify a friendly name for the Docker container
        docker.name = 'selenium-hub'
    end

Here we can see:

  • I’ll confess I stole that synced-folders workaround from another tutorial. It’s probably cargo-culting, since I never ran into that issue myself, but on the other hand, I’m not using shared folders here, and neither should you be. If you need to use shared-folders, use them in the lower level. If you need to move files into your container, that should be done using Docker’s native utilities for file system manipulation, which will be covered in part 2.
  • The vagrantfile for the host VM is the vagrantfile we built above, the centOS one.
  • The image to use is just the name of the image. Much like vagrant boxes, this will search the central repository and find the right container image to use, so don’t worry about this unless it fails.
  • The friendly name is used in the log output, so make it something you’ll recognize.

Once that launches successfully, the hard part is done: we now have a container on top of a custom VM. Now we just add nodes, which are also provided from the same source. Of course, now we’re moving from a single-machine setup to a multi-machine setup, so we use the multi-machine namespace tools Vagrant provides. We also should probably open port 4444 so that we can actually connect to the grid from our proper host machine.

# Parallelism will damage the links 
ENV['VAGRANT_NO_PARALLEL'] = 'yes'

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
    # The most common configuration options are documented and commented below.
    # For a complete reference, please see the online documentation at
    # https://docs.vagrantup.com.

    # Skip checking for an updated Vagrant box
    config.vm.box_check_update = false

    # Always use Vagrant's default insecure key
    config.ssh.insert_key = false

    # Disable synced folders (prevents an NFS error on "vagrant up")
    config.vm.synced_folder ".", "/vagrant", disabled: true

    config.vm.define "hub" do |hub|
        # Configure the Docker provider for Vagrant
        hub.vm.provider "docker" do |docker|

            # Define the location of the Vagrantfile for the host VM
            # Comment out this line to use default host VM
            docker.vagrant_vagrantfile = "dockerHost/Vagrantfile"

            # Specify the Docker image to use
            docker.image = "selenium/hub"

            # Specify port mappings
            # If omitted, no ports are mapped!
            docker.ports = ['4444:4444']

            # Specify a friendly name for the Docker container
            docker.name = 'selenium-hub'
        end
    end

    #We can parallel now
    ENV['VAGRANT_NO_PARALLEL'] = 'no'
    config.vm.define "chrome" do |chrome|
        # Configure the Docker provider for Vagrant
        chrome.vm.provider "docker" do |docker|

            # Define the location of the Vagrantfile for the host VM
            # Comment out this line to use default host VM that is
            # based on boot2docker
            docker.vagrant_vagrantfile = "dockerHost/Vagrantfile"

            # Specify the Docker image to use
            docker.image = "selenium/node-chrome:2.53.0"

            # Specify a friendly name for the Docker container
            docker.name = 'selenium-chrome'

            docker.link('selenium-hub:hub')
        end
    end

    config.vm.define "firefox" do |firefox|
        # Configure the Docker provider for Vagrant
        firefox.vm.provider "docker" do |docker|

            # Define the location of the Vagrantfile for the host VM
            # Comment out this line to use default host VM that is
            # based on boot2docker
            docker.vagrant_vagrantfile = "dockerHost/Vagrantfile"

            # Specify the Docker image to use
            docker.image = "selenium/node-firefox"

            # Specify a friendly name for the Docker container
            docker.name = 'selenium-firefox'

            docker.link('selenium-hub:hub')
        end
    end
end

Some things to note:

  • We use docker.link to link the nodes to the hub. This is a very Dockery thing, so i’m not entirely sure of the implications yet, but essentially, this pokes a bit of a hole in the container walls so that the processes in one container can see another container. This link creates our little network of grid nodes, allowing the nodes to register in the grid
  • We can’t create the hub and the nodes in parallel, because the nodes need to link to the hub and the hub may not be started yet when they try to register. I tried to turn parallel back on after the hub was created but I don’t think it actually works. Oh well. Maybe move the hub to its own machine that’s always up and only control the nodes with Docker?
  • You can pin to a specific version of the container, as I did for chrome, or you can leave it at the latest, as I did for firefox. There’s no reason I did them both differently except that I was testing out options to become more comfortable with the setup.

If you only need to test Chrome and Firefox, you can easily see how you can set up a small grid network this way. If you use vagrant heavily already with a cloud or private-cloud setup, you can just plug and play, replacing the virtualbox stuff with your provider of choice.

What about testing IE? Well, I started to put something together with the modernie VMs, as separate VMs that would need to be launched alongside the Docker provider and plug back into the hub, but ultimately I abandoned that course of action. We wouldn’t use vagrant for that task in a real setup, we’d just have a permanent VM set up to multithread requests for testing IE.

Instead, what interested me more was android testing with Selendroid. There was no docker image for selendroid however… yet. Docker as a provider also lets you build your own custom image, so that’s what I set out to do. Unfortunately, that doesn’t work yet. To Be Continued!

devops docker vagrant
© 2018 - 2023 by Lednerb
Bilberry Hugo Theme