If there’s one thing about quality assurance (QA) that is painful, it’s inheriting a project that has depended solely on manual testing for years. The scenario practically guarantees at least one of the following:
- One or more people regularly invested numerous hours performing tedious, repetitive tasks — and they didn’t need to do that work manually.
- One or more people tried to manually test the app as regularly as they could, but the app was never fully tested on a routine basis (meaning there are almost certainly things that have been broken for a while and no one has noticed).
But perhaps the most agonizing thought about manual testing is this: the moment that a newer version of the site or app is released, none of that earlier testing effort guarantees anything. A manual tester can spend an entire day testing a new “security update” only for a new security update to get released the next day — a Sisyphean task if ever there was one.
A Manual Testing Disclaimer
Manual testing is not a bad thing. In fact, it is absolutely a necessity to test manually when the circumstances call for it. When writing a test automation, our process always starts with manually performing a test while writing the automation in parallel. But manual testing should feel like exploration or experimentation; when it feels routine, that’s a clear sign to consider automation.
Knowing when manual testing is a necessity versus when it becomes a burden is the important distinction.
One counterintuitive benefit of automated testing is that it allows more time to perform manual testing where it actually matters. For example, the majority of accessibility violations cannot be detected by any automated tests. They require real humans experiencing real interactions. Automating the repetitive testing means people have more bandwidth to catch problems that can only be detected manually.
Examining the Cost Curve
The allure of manual testing is the misleading “initial” cost. Setting up automated testing requires more time and specialized knowledge than manual testing. Ultimately, that initial cost more than pays for itself, but it can also scare people away.
What most people don’t realize is that manual testing only continues to cost more and more as a site or application grows. Adding more features means more things need to be tested. Even just adding more pages to a website means more pages that need to be checked for broken parts. As unpleasant as that sounds, the alternative is even worse — as the app grows, there is more and more that just never gets tested (and that’s even more costly in the long term).
In the time it takes to manually test a handful of routine maintenance updates, we can build an automated test suite to perform the same checks. The majority of the cost happens at the start of the project, which is building an infrastructure to run the automations. Once the infrastructure is built and the tests are automated, the cost stays relatively static. The automations get a few minutes of maintenance every release compared to a few hours manual testing each release.
As applications add new features, we simply add new automations to cover what that feature does. It’s a small spike, which is comparable to the effort it would take to test that same feature manually.
The difference is that the effort invested into test automation has reusable value. Once the automation is built, the cost of testing goes back down to the baseline maintenance cost. The opposite is true with manual testing. Adding a new feature means increasing the time it takes to manually test that feature now and for every future release.
In other words, the cost of manual testing can only continue to increase. If the time spent manually testing ever decreases, it is because parts of the app have been cut from the testing scope because of time, budget, or both.
Shortening the Feedback Loop
Humans are not meant to perform monotonous tasks. As a task becomes more and more repetitive, we start to pay less and less attention. Repetitive manual testing is exactly the kind of task that puts the human brain on autopilot. Personally, when I am manually testing, I need a checklist. Without one, it’s way too easy to lose track of my progress.
Machines on the other hand, are built for repetitive tasks. A script that is written to test a website feature will never get bored and make a human mistake. Automations are extremely fast compared to a human doing manual testing. They also don’t rely on a person’s busy schedule as a bottleneck for testing feedback.
That speed plus autonomy is what makes it possible for developers to get a fast feedback loop when making code changes. A developer can run a test automation locally and uncover defects before they’ve even committed any code changes. That short feedback loop means problems can be detected and fixed before anything is even deployed to a test environment.
Automation as Risk Management
The biggest unseen risk of depending on manual testing alone is developers will fear making significant changes. Good engineering relies on engineers refactoring code as the app grows. Refactoring is essentially changing how the program does something without changing the final outcome of what it does.
As apps scale, some code paths become ineffective and need to be replaced with more sophisticated solutions. Without refactoring, a codebase can become littered with duplicated code and ineffective code paths, which are more opportunities for bugs to appear. To oversimplify the topic with a comparison: refactoring is similar to a carpenter who keeps their materials well organized and their tools well maintained. An efficient workshop means jobs can be completed more skillfully than if the space were cluttered with junk.
Thorough test coverage — which is only realistic long-term via automation — allows developers to confidently make optimizations and improvements without worrying that something will break and go unnoticed. When an automated test suite does its job, a developer trusts that when something does break, the tests will catch it.
Coming Up
In our first piece in our QA series, we explore how proactive testing throughout the development cycle creates faster, more confident releases. Next up in our QA series, we’ll explore the types of testing we employ, so stay tuned.
Want to explore how to automate testing for your next release? Reach out to us today.