Test needs model - when test and test automation meets

So, you decide that your project needs test automation because manual testing is too long or too costly to address your needs. How would you approach that? In essence, there’re two opposite approaches to test automation. Let’s call them “test-first” and “automate first”

Test-first approach 

In this realm, you have the luxury of having somebody actually designing a test case for you so you can automate it. Quite often, this is a place where many “test automation” engineers would like to be. These means they are actually being developers because they have requirements (which is a test case) and a product (which is an automated test case). They also have a clear goal (automate that) and they are not responsible for the quality of the product under test. In contrast, they are responsible only for the quality of their automated tests.

All things considered, this approach is actually quite sustainable from a quality control perspective. Provided automated tests are accurate, stable (which is test automation engineer’s work) and chosen wisely, the quality will be controlled as good as provided test cases are and as stable as test automation solution is.

On the other hand, it, basically, means having one developer (“test automation engineer”) working on non-production code at least part of the time. This also means she is going to work on something not very exciting, and therefore less likely to attract the most experienced specialist into the role. To make things worse, those specialists may decide to do all automation on UI level only, thus falling into IceCream trap.

It is also may lead to increased costs, both because you have to hire two people for one thing, and because those people may work with a different tempo, so sometimes time waste occurs.

You can address this by not having a dedicated test automation engineer role and ask the developer to perform test automation, but guess what? You just created an opportunity for people to tell you that they were working on more important feature implementation stuff so they didn't have a chance to automate this and that. And you probably also going to believe this.

 Automate-first approach

In this case, you may decide that you cannot afford to have a manual testing phase on a project, and you should have 100% automated testing. For instance, you may choose to get rid of "manual" testers and making a “lucky” test automation engineer responsible for all test-related activities.

In this case, you make your poor automation guy responsible for both providing a solid test strategy and test design (which she maybe not experienced in) and automating those tests. I hope she also gets a better salary on these terms.

This approach has at least one weak point – you just need to find somebody proficient in both testing and test automation disciplines, which is, in real life, more like looking for full-stack developer or unicorn. Long story short, while it is still possible, it may be not an easy task.

 

The other thing is by doing this you’re creating a bottleneck leaving little time for both testing and test automation. In this situation usually, you will have to either spend less time on test design (which, I think, happens most often, hence “automation first”) or on test automation. In any case, you’re sacrificing something. If this is justifiable is a different question, for now, let’s just acknowledge the fact.

You may also go further and get rid of dedicated QA at all, leaving all testing activities entirely to the development team.

Usually one gets what she aims for: in this case, you will not have manual tests, and those little tests you will have will be 100% automated. On the other hand, without having a solid test strategy and qualified test engineer input, you may end up having gaps in test coverage.

With this approach teams usually end up automating what they can automate, not what they should automate, both automating something which should not be automated and missing something important. Well, not 100% teams do that, I clearly see such tendency.

And yes, you just created an opportunity for people to tell you that they were working on more important feature implementation stuff so couldn’t automate this and that. And you probably also going to believe this.

Test needs model

While all the approaches mentioned above can work, they have certain inefficiencies. In an attempt to address that I suggest "Test needs model" approach

One thing that gets easily forgotten when we speak about test automation is that ultimate goal for any automated test is to, actually, perform a test. When I think about test automation, I cannot think about it without a context of actual testing on a project, and therefore I am suggesting following test needs model.

Generic test needs

Overall, when we speak about generic test needs, we speak about verification and validation. Do we build product right (which is internal quality) and do we build the right product (which is external quality). If you want to achieve sustainable quality assurance, you will need both internal and external quality control in place.

While the model is given in a form of a pyramid, I wish to stress, that in many cases it is possible that you don’t really care about internal quality, and all you’re aiming for is external quality. This is the right strategy for at least some occasions.

However, I am convinced that it will be hard to achieve sustainable delivery of external quality without having internal quality in place.

Manual-test needs pyramid

Even if you don’t have manual testers or manual testing phase on a project, the fact is, you still do at least some manual quality control operations on any project. You will perform a manual code review and (hopefully) manual test design. Sometimes, when it is not enough, you may also employ manual test execution, and, specifically, manual exploratory testing.

The most important is that, even with no manual test execution in place, test design is a crucial element for your success. One can argue that BDD acceptance criterion or Model-based testing techniques (as well as combinatory techniques and another way to automate test scenarios generation) substitute test design. The thing is, however, when you write BDD acceptance criterion, it is test design. With Model-based and other test generation techniques, you will still manually create models (which is a test design). The necessity of skilled quality-focused specialist work here is unavoidable.

Automated test needs pyramid

Automated test needs pyramid looks pretty much similar to Agile test pyramid, but has a slightly different meaning. While Agile test pyramid advocates proper test distribution, an automated test needs pyramid advocates that you need to build the lowest level before going upwards. Without solid unit testing, automated acceptance testing will not be able to provide effective quality control, not to speak about automated E2E testing.

Moreover, chances are, that once you really build properly lower levels, you may not need the upper level at all. For instance, it may be possible, that on your project limiting automated testing to unit-testing only is enough to provide sufficient quality feedback.

Applying test needs model

Usually, as a quality specialist, we will always advocate for better quality, while business needs only sufficient quality in place. By sufficient, I mean that the overall cost of prevented bugs is the same or bigger than investment into quality control procedures. Based on context-driven test school, let's consider two polar cases.

Internet chat room application.

In this application, you are providing some nice features, attracting your users into using your app. You’re getting money from showing adds on a page and by selling some advanced features to users. Your asset is a user base, which is a basement for your revenue.

From a quality control perspective, the problem is that before you start letting bugs into production, you usually cannot actually tell what is a cost of a bug is, and what is benefit from preventing this bug. You can probably guess, that there’re some features attracting most users, having the biggest priority, and also adds showing logic is crucial. What you don’t yet know, is how likely your team is going to put bugs into production. So you both don’t know what bug may cause to you, and how likely you’re going to produce a bug.

For such a project, I see quality control set up as an empirical process of building lower levels of automated and manual test needs, learning from experience, and then deciding if we need to move further. For instance, it may be possible, that after code-review, unit testing and some manual ad-hoc testing (or without it) you’re already providing sufficient quality to sustain your business. In this case, before quality becomes a problem you may not need to invest in upper pyramid levels, and you probably don’t need dedicated test and test automation roles at all.

Software providing medical tests based on a human blood

In this application, you are providing information about patients' health state based on his blood. The software is tightly coupled with hardware, and any change to software after production has a big financial impact. You’re making money out of selling devices with software and hardware in place and also from the maintenance of these devices.

In this case, a bug may lead to inadequate or absent treatment, ultimately leading to human death and millions of dollars compensation and abandonment of your device in the overall industry. You do know what cost of the bug maybe. What you yet don’t know, is how likely you’re going to produce bugs into production.

For such a project, one might suggest implementing all test needs levels, and then decide if we need that much. So, for instance, if after a year we notice that historically upper level didn’t contribute to quality (didn’t catch anything), we may consider minimize our attention these levels or get rid of them entirely. Thus, we may start from having both development, test and test automation teams working on a product, and then decide if we need to get rid of some roles.

Summary

This article has become too long and has covered several topics in one place, so I suggest a key takeaway's from this article:

1. Should you have separate roles for a test, test automation and development on a project you may achieve better quality control, but with bigger associated costs. It also usually increases time-to-market because of a longer feedback period (I will cover this aspect in later posts).
2. Should you have a dedicated role for both test and test automation, please make sure that the one taking this role is capable of doing both test design and test automation well, and also has enough time for both activities.
3. Should you not have dedicated test/test automation role, please make sure that developers really pay sufficient attention to QA-related activities and.
4. In any case, your choice on a team set up should be based on estimated quality needs for your project, and should never be considered set up in stone. In contrast, it should be an empirical process of seeking something that fits current needs for your project
5. On a long term project, without having solid coverage of lower-level needs, you will not be able to build solid coverage on an upper level needs. In rare cases when you will be able to do so, it will be associated with bigger costs.

I encourage you also to have a look at another insight into the very same problem outlined in this brilliant blog post: https://www.linkedin.com/pulse/developertester-ratio-what-really-means-me-sachin-natu

 Thank you for reading this post and please let me know your opinions about it.