Test automation framework - do I need it at all?

While there're lots of information published on the internet about how one should develop/buy a Test Automation solution, there's an elephant in the room - which is the fundamental question "Do I need to do it at all?".

That is a good question. While I think that there’re some legitimate contexts where product/project/team can benefit from having a sophisticated test automation solution/framework, my general rule of thumb is “If you need a framework - you do it wrong”.

• There’re at least several decent papers showing the benefit of unit-testing ([1],[2], [3]), while I could not find any research proving that system level tests could bring better results. Please let me know if you can prove me wrong.
• There’s at least one half-scientific paper showing the benefits of unit tests in comparison to integration/system level tests [4]
• There’s a good paper bringing a notion of running/writing and maintaining test vs. the benefit of having/running it at all [5]. If we know that, by definition, system level tests are more complex in maintenance, expensive in running and inspection, why do we insist on having so many of them?
• Inability to write small and reliable unit tests is almost universally considered an indication of poor product architecture.

However, there’re legitimate contexts where you might want/need to invest in a complex test automation solution. My personal opinion is that it is when both of the following is true:

• Product under test does not have enough reliable unit-test coverage;
• The cost of writing/maintaining test automation solution is lower than the cost of refactoring/investing in product architecture and unit-test coverage for the expected planning horizon.

Or where this is true:

• The quality is critical to the extent where we don't really care about how expensive QA would be.

All things considered, at the time of writing I think we have only one working hypothesis, which is:

While all type of automated verification has its own benefits and areas of applicability, unit testing seem to be the most efficient way of automatic verification.

[1] https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html

[2] https://collaboration.csc.ncsu.edu/laurie/Papers/Unit_testing_cameraReady.pdf

[3] https://www.linkedin.com/pulse/unit-testing-software-quality-empirical-research-results-avteniev/

[4] https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam

[5] https://www.microsoft.com/en-us/research/wp-content/uploads/2015/05/The-Art-of-Testing-Less-without-Sacrificing-Quality.pdf

If you want to know more on the topic - feel free to have a look at my Test Automation hub: https://senpay.github.io/

Follow @aqaguy

P.S. Used image is a public domain and taken from https://www.needpix.com/photo/19152/question-asking-questioning-wonder-why-ask-faq-sign-symbol