Welcome back to Teatime! This is a weekly feature in which we sip tea and discuss some topic related to quality. Feel free to bring your tea and join in with questions in the comments section.
Tea of the week: I’m feeling herbal today, chilling out with a refreshing cup of mint tea. I grow my own mint, but you can find good varieties at your local grocery store in all probability. I like mine with honey and lemon.
Today’s topic: The Three Ways of Devops
DevOps is one of my favorite topics to touch on. I firmly feel that the path to better quality software lies through the DevOps philosophy, even more so than through good QA. After all, finding bugs in finished software can only do so much; it’s better to foster a mindset in developers that prevents them from occurring in the first place.
What is DevOps?
DevOps is essentially a cultural phenomenon that aims to build a working relationship and harmony between development, QA, and operations. You can’t buy devops; you can’t hire devops. You have to shift your culture to follow the tao of devops, one step at a time. Anyone trying to sell you instant devops is lying, and you should avoid them like the plague. In that way, it’s a lot like Agile 🙂
DevOps is a logical extension of Agile, in fact. It was developed from the theories laid out in The Phoenix Project, a book I suggested reading a few weeks back, and made famous by a talk at the Velocity conference called “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” by John Allspaw and Paul Hammond.
You can see in the below graphic how it sort of extends the agile cycle into an infinity symbol:
For teams that are already agile, the jump is pretty straightforward: the definition of “done” now means, not just tested, but released, working in production. It adds a mindset that devs should absolutely care about how their code is doing in production, that the feedback loop should go all the way to the end user and back to the dev.
The First Way
Let me explain each way with a graphic designed by Gene Kim, one of the authors of The Phoenix Project:
The first way is the left-to-right flow of development to release. It promotes systems thinking, trying to get people in each stage of the way (us QA folks are about halfway along that line) to consider not just their own piece in the assembly line, but the line as a whole, maximizing for global efficiency rather than local. What good is testing quickly if the devs can’t get you another release until two weeks after you finish testing? What good is finding and fixing a ton of bugs if the user won’t see the fixes for two years? What good is a million passing tests if the application crashes after three hours in prod?
The principles necessary here to enact the first way are:
- Small batch sizes. Don’t release huge projects once a year, release small chunks of functionality regularly.
- Continuous build. If it doesn’t build, it won’t go to prod.
- Continuous integration. If your code builds but not when your neighbor’s code is added, it’ll blow up in production if it’s only ever tested in isolation.
- Continuous deployment. Get the code on a server ASAP so it can be tested
- Continuous testing. Unit tests, integration tests, functional tests; we want to know ASAP when something’s broken.
- Limit the amount of work in progress; too many balls in the air means you never know what’s going and what’s not.
The biggest rule of the first way: never pass defects downstream. Fix them where you find them.
The Second Way
The first way is the left-to-right flow of software the second way concerns the right-to-left flow of information back down to development:
Information needs to flow back from QA to Dev, back from staging to dev, back from deployment to dev, back from production to dev. The goal is to prevent problems from occurring over again when we already solved that issue. We need to foster faster detection of problems, and faster recovery from them when they occur. As QA professionals, we know you have to create quality at the source; you can’t test it in. In order to do that, we need to embed knowledge where it can do the most good: at the source.
In order to achieve the second way, you need to:
- Stop the production line when tests fail. Getting that crucial information back to development is key, and fixing problems before moving on to new development is crucial to ensure quality.
- Elevate the improvement of work over the work itself. If there’s a tool you could spend 2 hours building that would save 10 hours per week, why wouldn’t you spend the two hours and build it?
- Fast automated test suites. Overnight is too slow; we need information ASAP, before the dev’s attention span wanders.
- Shared goals and shared pain between dev and ops. When ops has a goal to keep the system up, dev should be given the same goal; when the system goes down, devs should help fix it. Their knowledge can be highly useful in an incident.
- Pervasive production information. Devs should know at all times what their code is doing in the wild, and be keeping an eye on things like performance and security.
The Third Way
We have a cycle, it’s very agile, we’re testing all the things; what more could we possibly learn? The Third Way, it turns out, is fostering a culture of experimentation and repetition in order to prepare for the unexpected and continue to innovate.
We have two competing concerns here in the third way, the yin and yang of innovation: how do we take risks without hurting our hard-won stability? The key is repetitive practice. It’s the same principle behind fire drills: when you do something regularly, it become second nature, and you’re ready to handle when it happens for real. That leaves you free to experiment and take risks, falling back on your well-practiced habits to save you when something goes wrong.
Key practices are:
- Promote risk taking over mindless order taking
- Create an environment of high trust over low trust
- Spend 20% of every cycle on nonfunctional requirements, such as performance and security concerns
- Constantly reinforce the idea that improvements are encouraged, even celebrated. Bring in cake when someone achieves something. Make it well known that they’ve done something good.
- Run recovery drills regularly, making sure everyone knows how to recover from common issues
- Warm up with code kata (or test kata) daily
Where are you in your journey toward DevOps? Have you begun? What can you take from this back to your daily life?