An alternative to ubiquitous UI-level checking - Subcutaneous tests
Let's assume a hypothetical situation - you were assigned to a project to help with the "test automation" initiative. You have a huge "test plan" as an input, containing hundreds (if not thousands) of "test cases" and you need to do something about it, quick.
Problem statement
Your first urge (if it is a Web application) may be to write some UI-level automated checks using tools like, you know, Selenium. In fact, there's a huge demand for such "Selenium test automators" in the industry these days. But please, please, don't do this.
There're lots of things which make UI-level automated checking the least desirable approach:
- UI-level automated checks will be slow. There's just no way to avoid it. You can parallel them or do some other tweaks to speed them up somehow, but they still will be slow.
- UI-level automated checks will be flaky. Partly - because they're slow. Partly because Web browser and UI interface were never designed to be operated by a computer. (I have to admit, this is now changing, and it is not necessarily good change).
- UI-level automated checks are, actually, the most difficult to write and maintain. Because they check too much. And because people usually try to automate "manual test cases" as is, making very ineffective automated checks (This one can be addressed, for sure)
- Even though some claim that UI-level automated checks emulate a real user, they don't. A user won't look for a field based on ID or XPath locator. A user won't fill a form with a speed of light and then "fail" to submit because something didn't appear in focus in the given timeframe. And now, when browsers are designed to be operated by a computer - what you automate is not a browser used by a user, but just a (very good) emulation. This list can be continued forever.
- Some would argue that there's functionality that is difficult to test otherwise. However, if there's a functionality, which can be checked using UI-level automated checks only (provided it is not UI logic itself) - it may be a good sign of architectural problems in the product.
The only real benefit of UI-level automated checks is that it will allow writing some usable checks without actually diving deep into the product codebase. Which is hardly a benefit in the long run.
An alternative solution
So, let's assume we have a user creation form which requires you to provide a valid user name. Once you give a user name that satisfies the rules - a new User object is created and stuffed into the database.
The application itself can be found here: https://github.com/senpay/login-form. Be warned, though - as you can guess from the screenshot, it is very buggy.
Let's try to think about the list of "test cases" for this application:
Let's try to think about the list of "test cases" for this application:
| Number | Steps | Expected results |
|---|---|---|
| 1 | 1. Enter a valid user name 2. Click "Log in" button | 1. 2. A new user is created. |
| 2 | 1. Enter an empty user name 2. Click "Log in" button | 1. 2. The error message is given. |
Does it look simple? Yes, it does! Can we automate it using Selenium? Sure we can. Code with automated UI-level checks "covering" "test cases" can be found in LoginFormTest class if you check out uitests git tag:
$git checkout uitests
Code coverage statistics:
Class: 100%
Method: 93.8% (30/32)
Line: 97.4% (75/77)
Now, let's consider the possibility that Funtctional (System-level) automated check may be written without touching UI at all. This technique is called subcutaneous tests and was suggested by Martin Fowler quite a long time ago [1].
When people think about "non-UI" level Functional checks, they usually think about an API. And by API they usually mean REST/SOAP or some other complicated stuff. However, an API (which, for a second, stands for application programming interface) may be a way faster and simpler thing.
If we dig into the product code, we may find these interesting lines:
It turns out, that whenever we click something on the UI, those two methods are being invoked - a user is added, then the user list is presented. So, what if we use those methods directly? It does look like an API, after all!
Then, we can end up with something like:
(can be found at subctests tag, $git checkout subctests)
Unexpectedly, test methods themselves become slightly smaller. What happens if we run them?
Time to execute: ~21 milliseconds
Code coverage statistics:
Class: 77.8%
Method: 78.1 (30/32)
Line: 78.7 (75/77)
Well! The speed boost is impressive. It is more than 600 times faster!
On the other hand, we somewhat lost the coverage. If this coverage loss is significant enough? That depends. What we lost was actually some of a glue code, which is (or may be) important (feel free to find what code is lost from the coverage yourself). But if it has to be checked using extensive UI-level checks - that's a different story.
At this point, we can:
And a little plug - I provide one day and two days workshops on this topic. So if you're interested - please ping me on Twitter, LinkedIn or via email.
Stay tuned!
Follow @aqaguy$git checkout uitests
A couple of things to note about those two tests would be:
Time to execute: ~12 seconds (12 seconds 956 milliseconds when I last checked)Code coverage statistics:
Class: 100%
Method: 93.8% (30/32)
Line: 97.4% (75/77)
Now, let's consider the possibility that Funtctional (System-level) automated check may be written without touching UI at all. This technique is called subcutaneous tests and was suggested by Martin Fowler quite a long time ago [1].
When people think about "non-UI" level Functional checks, they usually think about an API. And by API they usually mean REST/SOAP or some other complicated stuff. However, an API (which, for a second, stands for application programming interface) may be a way faster and simpler thing.
If we dig into the product code, we may find these interesting lines:
It turns out, that whenever we click something on the UI, those two methods are being invoked - a user is added, then the user list is presented. So, what if we use those methods directly? It does look like an API, after all!
Then, we can end up with something like:
(can be found at subctests tag, $git checkout subctests)
Unexpectedly, test methods themselves become slightly smaller. What happens if we run them?
Time to execute: ~21 milliseconds
Code coverage statistics:
Class: 77.8%
Method: 78.1 (30/32)
Line: 78.7 (75/77)
Well! The speed boost is impressive. It is more than 600 times faster!
On the other hand, we somewhat lost the coverage. If this coverage loss is significant enough? That depends. What we lost was actually some of a glue code, which is (or may be) important (feel free to find what code is lost from the coverage yourself). But if it has to be checked using extensive UI-level checks - that's a different story.
At this point, we can:
- Add one UI-level check, to verify that the glue code is still working, or
- Provided that we don't intend to change the glue code - leave it without automated checking, or
- Provided that there's some human-driven testing involved - we accept that with quite big probability all possible glue-code related issues would be identified by human anyway, or
- Any other possible solution
Summary
- Functional automated check don't have to be on UI or REST/SOAP API level. There's an alternative approach, called subcutaneous tests, which allows checking the same functionality at much higher speed
- This approach has a caveat though - some of the code coverage gets lost in the way
- To overcome this limitation, one can use "Functional Tests Model", which will be described later
- But in anycase, the speed boost (and check stability improvements) are so astonishingly significant, that I don't really know why not everyone uses the approach yet.
And a little plug - I provide one day and two days workshops on this topic. So if you're interested - please ping me on Twitter, LinkedIn or via email.
Stay tuned!
Comments
Post a Comment