An effective test automation

-
So what it means to have effective test automation? Let's say I can automate 10 000 tests a day. Impressive number, isn't it? Does it mean that my test automation effective? Hardly we can tell without knowing other things. Turns out that test automation effectiveness has little to do with how many tests you can automate a day. To judge effectiveness we need to think about what value test automation adds.

That leads me to a thought that test automation may be really the wrong term. Test automation is an activity, not a product. Even worse, the product that test automation creates (the value it produces) may be created by other means, too.

What value test automation creates? Faster execution of 70 000 UI test scenarios? Don't think so. Nice test result report? Well, maybe, but not necessarily. My favourite one, "elimination of human error in testing". I will not even bother to comment.

So the value it creates? My best guess nowadays would be that testing is an activity within the feature development process, and its' goal is to ensure that feature implementation adds more value than risks. Test automation is one of the tools to make testing activity faster, so you move quick.

If we take more concrete examples, I think test automation is effective if:
  1. It is properly distributed between levels (unit, integration, UI), so
  2. It takes a reasonable amount of time to run, to
  3. Provide reasonably accurate quality feedback about the product

Properly distributed between levels (unit, integration, UI)

Well, pretty much everyone knows about test pyramid.  The nice thing that comes as part of the parcel is that when you do add tests on all levels you also unintentionally create better designed and maintainable software. And it is also probably the only one scalable approach to assure that test run

Takes a reasonable amount of time

Let's say less than 30 minutes. So it is possible to do run after each commit or merge to master, so tests are being run often. It is crucial to be able to run as often as possible so problems don't get accumulated and at any moment we have

Reasonably accurate quality feedback

Meaning that if we have failed tests it indicates product problem and if we don't have failed test it yet possible that product has some bugs for some peculiar scenarios, but is reasonably catastrophe-level bugs free and can be deployed on the spot at least to the Beta environment.

This definition probably overly biased due to the environment I worked and am working in, and also likely to change and evolve over time. I would be glad if somebody can give me alternative definitions.

Comments

Post a Comment

Popular posts from this blog

Test automation framework - do I need it at all?

-
Image
While there're lots of information published on the internet about how one should develop/buy a Test Automation solution, there's an elephant in the room - which is the fundamental question "Do I need to do it at all?". That is a good question. While I think that there’re some legitimate contexts where product/project/team can benefit from having a sophisticated test automation solution/framework, my general rule of thumb is “If you need a framework - you do it wrong” . There’re at least several decent papers showing the benefit of unit-testing ([1],[2], [3]), while I could not find any research proving that system level tests could bring better results. Please let me know if you can prove me wrong. There’s at least one half-scientific paper showing the benefits of unit tests in comparison to integration/system level tests [4] There’s a good paper bringing a notion of running/writing and maintaining test vs. the benefit of having/running it at...
Read more

Efficient test automation - Secret #1 | Test Club

-
Image
It seems everyone talks about efficient test automation, but not so many people have actually seen it! To cover this I decided to make several videos regarding the secrets of efficient test automation. Today we will talk about the first (and, arguably, the main) secret - realistic expectations. Very often, test automation projects or initiatives are started with the following unrealistic expectations: "Test automation will replace 'manual' testing" "Test automation will provide fast results and benefits" "Test automation efficiency depends solely on test automation engineers" Unfortunately, all those expectations are not realistic! Watch the video to know more!
Read more

An alternative to ubiquitous UI-level checking - Subcutaneous tests

-
Image
Let's assume a hypothetical situation - you were assigned to a project to help with the "test automation" initiative. You have a huge "test plan" as an input, containing hundreds (if not thousands) of "test cases" and you need to do something about it, quick. Problem statement Your first urge (if it is a Web application) may be to write some UI-level automated checks using tools like, you know, Selenium. In fact, there's a huge demand for such "Selenium test automators" in the industry these days. But please, please, don't do this. There're lots of things which make UI-level automated checking the least desirable approach: UI-level automated checks will be slow. There's just no way to avoid it. You can parallel them or do some other tweaks to speed them up somehow, but they still will be slow. UI-level automated checks will be flaky. Partly - because they're slow. Partly because Web browser and UI interface w...
Read more