Test automation framework - do I need it at all?


While there're lots of information published on the internet about how one should develop/buy a Test Automation solution, there's an elephant in the room - which is the fundamental question "Do I need to do it at all?".



That is a good question. While I think that there’re some legitimate contexts where product/project/team can benefit from having a sophisticated test automation solution/framework, my general rule of thumb is “If you need a framework - you do it wrong”.
  • There’re at least several decent papers showing the benefit of unit-testing ([1],[2], [3]), while I could not find any research proving that system level tests could bring better results. Please let me know if you can prove me wrong.
  • There’s at least one half-scientific paper showing the benefits of unit tests in comparison to integration/system level tests [4]
  • There’s a good paper bringing a notion of running/writing and maintaining test vs. the benefit of having/running it at all [5]. If we know that, by definition, system level tests are more complex in maintenance, expensive in running and inspection, why do we insist on having so many of them?
  • Inability to write small and reliable unit tests is almost universally considered an indication of poor product architecture.
However, there’re legitimate contexts where you might want/need to invest in a complex test automation solution. My personal opinion is that it is when both of the following is true:

  • Product under test does not have enough reliable unit-test coverage;
  • The cost of writing/maintaining test automation solution is lower than the cost of refactoring/investing in product architecture and unit-test coverage for the expected planning horizon.
Or where this is true:
  • The quality is critical to the extent where we don't really care about how expensive QA would be.
All things considered, at the time of writing I think we have only one working hypothesis, which is:
While all type of automated verification has its own benefits and areas of applicability, unit testing seem to be the most efficient way of automatic verification.


If you want to know more on the topic - feel free to have a look at my Test Automation hub: https://senpay.github.io/




P.S. Used image is a public domain and taken from https://www.needpix.com/photo/19152/question-asking-questioning-wonder-why-ask-faq-sign-symbol


Comments

  1. Techsaga Corporations is an Operational automation strategy makers company. The benefits of automation create expectations for customers, stakeholders, and employees; the effectiveness of automation allows us to maintain them. As IT environments increase in complexity and the amount of data, devices, systems, and platforms expand.

    ReplyDelete
  2. Thank you for sharing this post. A few useful links for job seekers in the software testing domain,
    QA Job Board
    Software Testing Blog
    QA Job Search App

    ReplyDelete

Post a Comment

Popular posts from this blog

Test automation framework architecture. Part 2 - Layered architecture

Test automation framework architecture. Part 2.1 - Layered architecture example