Test needs model - when test and test automation meets



So, you decide that your project needs test automation, because manual testing is too long or too costly to address your needs. How would you approach that? In essence, there’re two opposite approaches to test automation. Let’s call them “test first” and “automate first”

Test first approach 

In this realm, you have luxury of having somebody actually designing a test case for you, so you can automate it. Quite often, this is a place where many “test automation” engineers would like to be. These means they are actually being developers, because they have requirements (which is a test case) and a product (which is aautomated test case). They also have a clear goal (automate that) and they are not responsible for quality of the product under test. In contranst, they are responsible only for quality of their automated tests.
All things considered, this approach is actually quite sustainable from quality control perspective. Provided automated tests are accurate, stable (which is test automation engineer’s work) and chosen wisely, quality will be controlled as good as provided test cases are and as stable as test automation solution is.

On the other hand, it, basically, means having one developer (“test automation engineer”) working on non-production code at least part of the time. This also means she is going to work on something not very exciting, and therefore less likely to attract most experienced specialist into the role. To make things worse, those specialists may decide to do all automation on UI level only, thus falling into IceCream trap.



It is also may lead to increased costs, both because you have to hire two people for one thing, and because those people may work with different tempo, so sometimes time waste occurs.

You can address this by not having dedicated test automation engineer role and ask developer to perform test automation, but guess what? You just created an opportunity for people to tell you that they were working on more important feature implementation stuff so they didn't have a chance to automate this and that. And you probably also going to believe this.

 Automate first approach


In this case, you may decide that you cannot afford having manual testing phase on a project, and you should have 100% automated testing. For instance, you may choose get rid of "manual" testers and making “lucky” test automation engineer responsible for all test-related activities.

In this case, you make your poor automation guy responsible for both providing solid test strategy and test design (which she may be not experienced in) and automating those tests. I hope she also gets a better salary on these terms.


This approach has at least one weak point – you just need to find somebody proficient in both testing and test automation disciplines, which is, in real life, more like looking for fullstack developer or unicorn. Long story short, while it is still possible, it may be not an easy task.
 
The other thing is by doing this you’re creating a bottleneck leaving little time for both testing and test automation. In this situation usually you will have to either spend less time on test design (which, I think, happens most often, hence “automation first”), or on test automation. In any case, you’re sacrificing something. If this is justifiable is a different question, for now let’s just acknowledge the fact.

You may also go further and get rid of dedicated QA at all, leaving all testing activities entirely to development team.
Usually one gets what she aims for: in this case, you will not have manual tests, and those little tests you will have will be 100% automated. On the other hand, without having solid test strategy and qualified test engineer input, you may end up having gaps in test coverage.

With this approach teams usually end up automating what they can automate, not what they should automate, both automating something which should not be automated and missing something important. Well, not 100% teams do that, I clearly see such tendency.

And yes, you just created an opportunity for people to tell you that they were working on more important feature implementation stuff so couldn’t automate this and that. And you probably also going to believe this.

Test needs model

Please, don't get me wrong, I don't say that any of described approaches cannot work. I am saying it is possible they would not work, especially if implemented wrong or in wrong place. By saying not work, I mean not providing adequate quality control with resources at hand.

My current understanding is that you can achieve effective quality control with automated tests by applying something I call test needs pyramid approach (which is related to agile test pyramid, but not exactly same).

One thing that gets easily forgotten when we speak about test automation, is that ultimate goal for any automated test is to, actually, perform a test. When I think about test automation, I cannot think about it without a context of actual testing on a project, and therefore I am suggesting following test needs model.

Generic test needs

Overall, when we speak about generic test needs, we speak about verification and validation. Do we build product right (which is internal quality) and do we build a right product (which is external quality). If you want to achieve sustainable quality assurance, you will need both internal and external quality control in place.

While model is given in a form of a pyramid, I wish to stress, that in many cases it is possible that you don’t really care about internal quality, and all you’re aiming for is external quality. This is right strategy for many occasions.

However, I am convinced that it will be hard to achieve sustainable delivery of external quality without having internal quality in place.

Manual test needs pyramid

Even if you don’t have manual testers or manual testing phase on a project, the fact is, you still do at least some manual quality control operations on any project. You will perform manual code review and (hopefully) manual test design. Sometimes, when it is not enough, you may also employ manual test execution, and, specifically, manual exploratory testing.
The most important is that, even with no manual test execution in place, test design is crucial element for your success. One can argue that BDD acceptance criterion or Model-based testing techniques (as well as combinatory techniques and other way to automate test scenarios generation) substitute test design. The thing is, however, when you write BDD acceptance criterion, it is a test design. With Model-based and other test generation techniques, you will still manually create models (which is a test design). Necessity of skilled quality-focused specialist work here is unavoidable.

Automated test needs pyramid

Automated test needs pyramid looks pretty much similar to agile test pyramid, but has slightly different meaning. While agile test pyramid advocates proper test distribution, automated test needs pyramid advocates that you need to build lowest level before going upwards. Without solid unit testing, automated acceptance testing will not be able to provide effective quality control, not to speak about automated E2E testing.

Moreover, chances are, that once you really build properly lower levels, you may not need upper level at all. For instance, it may be possible, that on your project limiting automated testing to unit-testing only is enough to provide sufficient quality feedback.

Applying test needs model

Usually, as quality specialist, we will always advocate for better quality, while business needs only sufficient quality in place. By sufficient, I mean that overall cost of prevented bugs is same or bigger then investment into quality control procedures. Based on context driven test school, lets consider two polar cases.

Internet chat room application.

In this application, you are providing some nice features, attracting your users into using your app. You’re getting money from showing adds on a page and by selling some advanced features to users. Your asset is a user base, which is a basement for your revenue.
From quality control perspective, problem is that before you start letting bugs into production, you usually cannot actually tell what is a cost of bug is, and what is benefit from preventing this bug. You can probably guess, that there’re some features attracting most users, having biggest priority, and also adds showing logic is crucial. What you don’t yet know, is how likely your team is going to put bugs into production. So you both don’t know what bug may cause to you, and how likely you’re going to produce a bug.
For such a project, I see quality control set up as an empirical process of building lower levels of automated and manual test needs, learning from experience, and then deciding about if we need to move further. For instance, it may be possible, that after code-review, unit testing and some manual ad-hoc testing (or without it) you’re already providing sufficient quality to sustain your business. In this case, before quality becomes a problem you may not need to invest into upper pyramid levels, and you probably don’t need dedicated test and test automation roles at all.

Software providing medical tests based on a human blood

In this application, you are providing information about patient health state based on his blood. Software is tightly coupled with hardware, and any change to software after production has big financial impact. You’re making money out of selling devices with software and hardware in place and also from maintenance of these devices.
In this case, bug may lead to inadequate or absent treatment, ultimately leading to human death and millions of dollars compensation and abandonment of your device in overall industry. You do know what cost of the bug may be. What you yet don’t know, is how likely you’re going to produce bugs into production.
For such  a project, one might suggest implementing all test needs levels, and then decide if we need that much. So, for instance, if after a year we notice that historically upper level didn’t contribute into quality (didn’t catch anything), we may consider minimize our attention these levels or get rid of them entirely. Thus, we may start from having both development, test and test automation teams working on a product, and then decide if we need to get rid of some roles.

Summary

This article has become too long and has covered several topics in one place, so I suggest a key takeaways from this article:
  1. Should you have separate roles for test, test automation and development on a project you may achieve better quality control, but with bigger associated cost. It also usually increases time to market because of longer feedback period (I will cover this aspect in later posts).
  2. Should you have dedicated role for both test and test automation, please make sure that the one taking this role is capable of doing both test design and test automation well, and also has enough time for both activities.
  3. Should you not have dedicated test/test automation role, please make sure that developers really pay sufficient attention to QA-related activities and.
  4. In any case, you choice on a team set up should be based on estimated quality needs for you project, and should never be considered set up in stone. In contrast, it should be empirical process of seeking something that fits current needs for your project
  5. On a long term project, without having solid coverage of lower level needs, you will not be able to build solid coverage on upper level needs. In rare cases when you will be able to do so, it will be associated with bigger costs.
I encourage you also to have a look at another insight to the very same problem outlined in this brilliant blog post: https://www.linkedin.com/pulse/developertester-ratio-what-really-means-me-sachin-natu
 Thank you for reading this post and please let me know your opinions about it.


Comments

  1. Hi Alexander Pushkarev. I went through your blog and I could see what you are trying to say. I do agree with most of it. Well said. Software industry is maturing and recovering from phase where testing efforts investments were improper for many reasons. I guess such thoughts helps people to start doing balancing act.

    ReplyDelete
    Replies
    1. Hello Sachin Natu, thanks a lot for your feedback, as I think you have a great expertise in this area your feedback is very important for me. I also think that you "test activities" model is very relevant to this topic, so I will add a link to this in article to make it topic coverage wider.

      Delete

Post a Comment

Popular posts from this blog

Test automation framework architecture. Part 2 - Layered architecture

Test automation framework architecture. Preface.

Test automation framework architecture. Part 1 - No architecture