Welcome back to Teatime! This is a weekly feature in which we sip tea and discuss some topic related to quality. Feel free to bring your tea and join in with questions in the comments section.
Tea of the week: It’s been a stressful (and cold!) week as I pre-write this, so I’m chilling out with some Toasted Caramel Rooibos from Sub Rosa Teas.
Today’s topic: What kind of testing should I do?
For the first QA-content-filled teatime, I wanted to start at the beginning and touch briefly on what kinds of testing there are, and when to use each of them. This is not intended to be an exhaustive list, more of a gentle overview while we sip tea and meditate on our own projects and the testing each one needs.
Things to consider
When you’re putting together a test plan, and you’re considering performing various types of tests, you should consider the following:
- What is the purpose of this kind of test?
- In what situations is this test most suited?
- What differentiates this test type from others?
- Why are we performing this test?
Testing without purpose is not so much testing as playing; if there’s no business need, and no reason to perform the test, all you’re doing is wasting your time. Furthermore, if you already have your needs covered, adding more types of tests won’t help anything, by definition: your needs are already covered, and further testing is just wasting time.
The Testing Quadrants
I came across a version of this diagram in the book Continuous Delivery by Jez Humble and David Farley, and I really liked it, so I’ve recreated my own version here. There are two axes represented above, both equally important. On the horizontal axis, testing can either support development efforts, which is to say, it can provide input into the ongoing effort of building the software to correct the course by small degrees; or, it can critique a finished product, producing a feedback loop for development of future enhancements and/or future products. On the vertical axis, tests can be developer-facing, giving feedback to the development team on the project, or they can be user-facing, giving feedback to the BA from a user’s perspective of the software.
Therefore, in this model, acceptance tests are user-oriented tests; do not write acceptance tests for things that are invisible to the user! On the other hand, they support development, so we want to run them as early as possible to tighten the feedback loop so the devs can course-correct. Which argues for an Agile approach 🙂
Unit tests are similar to Acceptance Tests in that they support development and so should be written and executed as early as possible, but they are developer-facing, so they should absolutely test for things that only developers can see, like method signatures and code organization.
Exploratory tests are like Acceptance Tests in that they are user-facing, and thus should only focus on things the end user can see; however, they are a critique, intended to find things when the product is in a stable, “release candidate” state. They serve to prove to the business that what we built meets their expectations, not to aid developers in building it.
And finally, Nonfunctional acceptance tests allow us to critique the product from a development standpoint: now that we know it works (because of Unit tests), we need to see if it’s performant, secure, et cetera.
What about regression testing?
You may have noticed that regression testing isn’t in any of the four quadrants. It doesn’t really fit into this graph; it sort of lies orthogonal to the graph, or envelops the entire thing. Regression testing is simply the act of verifying that requirements which were previously tested to work have not been broken since the test was run. Without qualifiers, we typically mean Functional Regression Testing, which is simply running old acceptance or unit tests over again to ensure that the functionality was not broken. You can also, however, perform nonfunctional regression testing, say, to verify that software that was previously fast has not gotten slower, or software that previous installed on Windows still does after being enhanced to run on Linux. Exploratory regression testing would be exploratory testing of areas that have not changed in the most recent version.
So what kinds should I do?
All of them 🙂
Honestly, only you can answer the question of what types of testing are right for your business needs, your product, your development team, your release cycle, et cetera. Hopefully, however, you have some ideas now that you’ve paused to consider the types available and what goals they fulfill. So, you tell me: what kinds of testing will you do?