Everywhere I go, I see talk about BDD and Cucumber. Cucumber is the promised messiah, the single technology that bridges the world of the business and the world of the programmer, allowing your BA to write executable test cases so you don’t have to spend time automating once you have a solid framework. It’s the future, the new order, and it’s here, now, ready for prime time. Who wouldn’t want to learn it, right?
That’s what I thought before I took a class on Ruby that happened to use it and got a good look under the hood. And what I saw wasn’t magical unicorn sparkles at all. It was a tangled nest of Regex. Lots and lots of Regex.
If that hasn’t already scared you away, dear reader, strap in, because you’re in for a wild ride.
So I have a test automation framework for our public-facing site, Right Turn, which allows users to search for and purchase tires for their vehicle. We have tests, but the maintenance is killing us, so i’m exploring all kinds of alternate options. Among other things, one thing I wanted to do was expose the framework to a Gherkin-style test harness so that the BA could put in test scripts for the ephemeral one-off bug fixes, get them tested, and remove them when she’s sure they’re not going to regress.
That’s one thing I really want to stress: I already have a framework that’s capable of driving my page using traditional WebDriver pageObjects. This initiative doesn’t remove or lesson that requirement at all, so there goes the overhead of “I don’t have to write as much code” out of the gate. You might be able to start doing this without a proper framework, but I doubt you’d be able to finish without it.
So I start puttering about in Java, pulling in Cucumber JVM as a Maven dependency. What will I need? Well, every test is going to want to specify a vehicle I imagine, so I’ll toss in a step like that:
@Given("I have searched for tires for a ([\\w\\s]+)")
public void I_have_found_a_product(String searchCriteria) {
}
This regex is simple, just grab everything after the intro bit. Now what? Well, I need to turn that string into Vehicle object. So I skim over my existing Vehicle object for a parser… and I don’t have one. I’ll need to write it. Okay, how hard could it be to turn “2009 Kia Spectra” into Vehicle(2009, “Kia”, “Spectra”)?
//Sample: 2009 Kia Spectra
pattern = Pattern.compile("(\\d{4}) (\\w+) (\\w+)");
m = pattern.matcher(vehicleString);
if (m.matches()) {
return new Vehicle(m.group(1), m.group(2), m.group(3), null, null);
}
Those nulls? Those are Trim and Option. We’ll ignore them for now. Year Make Model, done, check.
Until I want to use a “2005 Land Rover LR3”. Well crap. I know the right way to do this, but it’s not fun.
private final static String Makes = "Acura|Audi|BMW|Bentley|Buick|Cadillac|Chevrolet|Chrysler|Dodge|Ford|GMC|Honda|Hummer|Hyundai|Infiniti|Jaguar|Jeep|Kia|Land Rover|Lexus|Lincoln|MINI|Maserati|Maybach|Mazda|Mercedes-Benz|Mercury|Mitsubishi|Nissan|Pontiac|Porsche|Rolls Royce|Saab|Saturn|Scion|Smart|Subaru|Suzuki|Tesla|Toyota|Volkswagen|Volvo";
public static Vehicle parseString(String vehicleString) {
Pattern pattern;
Matcher m;
//Sample: 1999 Kia Spectra
pattern = Pattern.compile("(\\d{4}) (" + Makes + ") ([\\w\\s]+)");
m = pattern.matcher(vehicleString);
if (m.matches()) {
return new Vehicle(m.group(1), m.group(2), m.group(3), null, null);
}
throw new IllegalArgumentException("Cannot parse vehicle string: " + vehicleString);
}
}
Okay. So then I’ll need a product. So I go to the site, putting in my test vehicle (RIP my poor little Kia, but she makes a better test vehicle than she did a transportation option anyway), and… I can’t go on in the purchase funnel. Why? You can’t deduce the tire size from the YMM, you need Trim on this vehicle.
Okay. Sure. Whatever. Let’s do this.
private final static String Makes = "Acura|Audi|BMW|Bentley|Buick|Cadillac|Chevrolet|Chrysler|Dodge|Ford|GMC|Honda|Hummer|Hyundai|Infiniti|Jaguar|Jeep|Kia|Land Rover|Lexus|Lincoln|MINI|Maserati|Maybach|Mazda|Mercedes-Benz|Mercury|Mitsubishi|Nissan|Pontiac|Porsche|Rolls Royce|Saab|Saturn|Scion|Smart|Subaru|Suzuki|Tesla|Toyota|Volkswagen|Volvo";
public static Vehicle parseString(String vehicleString) {
Pattern pattern;
Matcher m;
//Sample: 2009 Kia Spectra EX
pattern = Pattern.compile("(\\d{4}) (" + Makes + ") ([A-Za-z0-9]+) ([\\w]+)");
m = pattern.matcher(vehicleString);
if (m.matches()) {
return new Vehicle(m.group(1), m.group(2), m.group(3), m.group(4), null);
}
//Sample: 1999 Kia Spectra
pattern = Pattern.compile("(\\d{4}) (" + Makes + ") ([\\w\\s]+)");
m = pattern.matcher(vehicleString);
if (m.matches()) {
return new Vehicle(m.group(1), m.group(2), m.group(3), null, null);
}
throw new IllegalArgumentException("Cannot parse vehicle string: " + vehicleString);
}
I’m starting to feel vaguely nauseated, but I’ve got a vehicle parser that can handle the one test case I’m trying to put together for a demo. It’s held together with twine and duct tape, but it parses reliably. I can tell because, of course, I wrote unit tests:
Anyway, the Kia doesn’t have options, so we’ll just move on for now. (This is hard for me: it’s wrong and I know it but I want to get to a working demo this week, so I have to force myself to leave it alone).
Right, so I’ve got a vehicle. Now I need to navigate through the purchase funnel. This is where the framework saves me:
LocationPage locationPage = (LocationPage) new LocationPage(driver).navigateTo();
locationPage.enterZipCode("44321");
VehiclePage vehiclePage = (VehiclePage) locationPage.clickNext();
vehiclePage.selectVehicle(vehicle);
//Skip vehicle page and tire coach
vehiclePage.clickNext().clickNext().clickNext();
All that logic was pre-existing, lifted right out of one of our existing tests. Except, you may have noticed one tiny problem: where did that Driver come from?
For now, I just construct one the old-fashioned way:
Webdriver driver = new FireFoxDriver();
driver.get("demo.rightturn.com");
Great, it works, we navigate. Now what?
One test case I heard the BA complaining about regressing often had to do with our product comparison feature: when you added product A, then product B, then product C, then hit “compare”, it should list them in the order A, B, C on the comparison page, but it kept doing them in the order that the API happened to return them, which was arbitrary. It had regressed a few times from simple mistakes, and she never remembered to test it, so she’d benefit from a test that could verify it quickly.
So I figure, okay, we’ll need to add a product to the compare widget:
@When("I add ([\\w\\s]+) to the compare widget")
public void I_add_to_compare_widget(String product) {
WebDriver driver = new FirefoxDriver();
ProductPage productPage = new ProductPage(driver);
Product p = getProductByName(product, productPage);
productPage.addProductToCompare(p);
}
And click compare:
@When("I click compare")
public void I_click_compare() {
WebDriver driver = new FirefoxDriver();
ProductPage productPage = new ProductPage(driver);
productPage.clickCompare();
}
And verify the position:
@Then("([\\w\\s]+) should be in the (left|right|middle) container")
public void item_in_compare_bucket(String productName, String position) {
WebDriver driver = new FirefoxDriver();
ComparePage productComparePage = new ComparePage(driver);
int slotNum = 0;
if (position.equalsIgnoreCase("left")) {
slotNum = ComparePage.LEFT_SLOT;
}
if (position.equalsIgnoreCase("right")) {
slotNum = ComparePage.RIGHT_SLOT;
}
if (position.equalsIgnoreCase("middle")) {
slotNum = ComparePage.CENTER_SLOT;
}
Product actual = productComparePage.getProductInSlot(slotNum);
assertEquals(actual.getName(), productName);
}
I’m sure by now you’re screaming at me; the mistake is a newbie one, but it’s glaring and obvious once you know what to look for. You see, each of those drivers will drive separate instances of the browser; it’ll open three windows, and be very confused when it’s not on the right page at all.
What I need now is one of the harder problems with Cucumber: shared state. Somehow, I have to persist the driver between steps, but not between tests that happen to run in parallel (and we do a lot of parallelization of our webdriver tests, as they’re slow and clunky).
For now, I’ll pray that the parallelization engine properly constructs a new instance of my step class for each test, and make it a class variable:
public class sampleStepDefs {
WebDriver driver;
And while I’m at it, I’ll move driver construction to a method, and swap it out for our remoteWebDriver boilerplate code (hardcoded to localhost for now, as I don’t want to get into configuration just yet):
private WebDriver getDriver() throws IOException {
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("firefox");
capabilities.setCapability(CapabilityType.SUPPORTS_LOCATION_CONTEXT, true);
capabilities.setCapability("autoAcceptAlerts", true);
WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capabilities);
driver.get("http://demo.rightturn.com");
return driver;
}
Oh, and of course, it throws a malformedURLException in case the hardcoded URL that’s worked every other time somehow stops working. Which means I need to catch that or bubble it up:
@Given("I have searched for tires for a ([\\w\\s]+)")
public void I_have_found_a_product(String searchCriteria) throws IOException {
if (driver == null) {
driver = getDriver();
}
Vehicle vehicle = Parser.parseVehicleString(searchCriteria);
LocationPage locationPage = (LocationPage) new LocationPage(driver).navigateTo();
locationPage.enterZipCode("44321");
VehiclePage vehiclePage = (VehiclePage) locationPage.clickNext();
vehiclePage.selectVehicle(vehicle);
//Skip vehicle page and tire coach
vehiclePage.clickNext().clickNext().clickNext();
}
I also now have a distinction between which steps are allowed to be Givens (and thus construct a WebDriver) and which are only Whens and Thens (which do not). I’m imposing arbitrary rules above and beyond the domain language, and it’s awful; I’ll have to put some thought around a better way to enact this. But now my brain is firmly gathering wool, chasing every little optimization, and I still don’t have a working demo just yet.
This writeup will gloss over the half hour I spent with an online regex tester perfecting the regexes you saw above; do not, however, let that fool you: you’ll need to be good at regex to make this work. Essentially, the more flexible you want to be for your users, the more you need to get into natural language processing, which is a skill I would never expect an automation engineer to possess. Don’t we have enough domain skills we have to pick up without adding entire fields of study to our toolbox? So regex it is, and forcing our users to bend to fit our molds, which goes against everything we know about usability but what can we do about it, really?
All that work for this:
Feature: Comparison
Scenario: Add to compare page
Given I have searched for tires for a 2009 Kia Spectra LX
When I add Assurance Fuel Max to the compare widget
And I add Precision Sport to the compare widget
And I add AVID Ascend to the compare widget
And I click compare
Then Assurance Fuel Max should be in the left container
And Precision Sport should be in the middle container
And AVID Ascend should be in the right container
Is it worth it? Is this something our business users or BAs can even produce? There’s a hidden rigidity behind the deceptively fluid language, a whole world of rules they have to learn and memorize. But if we can offload some of the cognitive load to them, doesn’t that let us solve more of the hard problems? I don’t have answers here. I just want to be clear about what we’re doing: involving BAs in the process of test automation, not handwaving the entire process away with a magic wand.