A QA-guy blog

While there're lots of information published on the internet about how one should develop/buy a Test Automation solution, there's an elephant in the room - which is the fundamental question "Do I need to do it at all?". That is a good question. While I think that there’re some legitimate contexts where product/project/team can benefit from having a sophisticated test automation solution/framework, my general rule of thumb is “If you need a framework - you do it wrong” . There’re at least several decent papers showing the benefit of unit-testing ([1],[2], [3]), while I could not find any research proving that system level tests could bring better results. Please let me know if you can prove me wrong. There’s at least one half-scientific paper showing the benefits of unit tests in comparison to integration/system level tests [4] There’s a good paper bringing a notion of running/writing and maintaining test vs. the benefit of having/running it at...

Test automation at the Web UI level (you know, Selenium stuff) is usually brittle and painful. It is usually also the only test-automation approach people are aware of/interested in (for whatever reasons). Below are several suggestions on how one can make her/his UI test automation at the Web UI level less painful and flaky. 1. Retry failed steps Each test consists of several steps. Unfortunately, those steps from time to time tend to fail with no apparent reason. Retrying a failed test step may save your whole test from failing. Helps with: Unresponsive, slow UI Elements being shown with a delay Wrong moon phase [1] Drawbacks: Sometimes it is just a waste of time, especially when the real issue is being "retried" Even worse: sometimes it may hide real issue that would go away after some time or page refresh 2. Retry failed tests From time to time - odds are just against you with a specific test run. Everyone, whoever had the misery of working with Web UI...

As we've learned in previous posts of the series - Layered (also known as Tiered) architecture is a de-facto standard of the industry. There're two major variations of this architectural pattern that one should be aware of: 3-layered architecture (described here ) and 2-layered architecture . This post is going to concentrate on 2-layered architecture . 2-layered architecture is typically seen in UI-specific test automation frameworks. The key feature of this pattern is the absence of Business-layer, so most of the business-related logic are actually to be put into Page Objects. In order to avoid " The God class " anti-pattern and still be able to contain business-related logic in Page Object, one usually uses libraries or frameworks that allow simplifying DOM-elements location, thus letting Page Objects to concentrate less on locating elements and more on business-logic itself. The good thing in this architectural pattern is that it actually allows you...

Let's consider an example of what Layered (Tiered) architecture may look like for a test automation framework. As a system under test, we're going to use this simple and neat "Todo list" application. First, let's do a brief analysis of the application. From my brief exploration I came up with the following checklist: Area/Group Things to check Comments Todos CRUD 1 Create item Basic scenario, UI-level case 1.1 Create an item with the same name The case is not valid. There's no validation for that - nothing to check really. 1.2 Create an item with an empty/not valid/malicious name While it would be a nice thing to test manually, from test automation standpoint I would expect this to be filtered on service or repository level, so either integration or unit test . 2 Mark item complete Basic scenario, UI-level case 2.1 Un-mark item complete Basic scenario, UI-level case 3 Delete item Basic scenario, UI-level case 3.1 Delete item ...

Preface: I am in the middle of writing a series of posts about test automation frameworks architecture. I am still going to continue that series, even though this posts kind of devaluate the whole test automation framework concept a bit. Sorry for that, just can't stop ranting. Looking through the internet I spotted a couple of posts where some test automation specialists were talking about "page object" "pattern"/"model", as it was something special they had invented. Well, I also have something to say about "page object". “Java test automation engineers were told that there are different patterns than a PageObject” ( http://classicprogrammerpaintings.com/post/153817288474/java-test-automation-engineers-were-told-that ) "Page object" may be described as a pattern that allows us to decouple things you can do with the web page (external interface describing test/business logic) from the real implementation code you will h...

Probably the most popular architecture pattern used for test automation frameworks (TAF) is layered architecture. This pattern is so well known that on job interviews for some companies when they ask you about TAF architecture you are supposed to describe this one. If you don't - they think you know nothing about the architecture altogether. I suggest you first read a brilliant description of the pattern at the O'Reilly web page , cause in this post I am going to describe the pattern in a way it is usually applied to build test automation solution. Usually, there're three distinct layers, which may have different names, but follow the same logic mostly. Sometimes those layers called test layer , business-layer and core layer , but there're no standard names really. Key rules for layered architecture are the dependency direction (each level depending on the level below) and call direction (no level can call/reference code described in the level above). The rough...

Most of the "UI test automation" tutorials I have seen describe the Test Automation Solution where Selenium Web Driver is used directly from test methods and no additional abstraction layer exists. This "architectural pattern" is so ubiquities, that I decided to describe this as well. I think we can call this "architectural pattern" as No architecture . The structure of a test automation solution created with No architecture pattern presented (in a very rough way) on the picture below: Such test automation solution usually consists of some amount of test classes, each containing some number of test methods. All orchestration of interaction with System Under Test (SUT) is done either (using JUnit terms) in setUp and tearDown methods, or test methods themselves. Tool, whatever it is - Selenium, RestAssured, Selenide - called directly in test methods. Such an approach is the industry well-known anti-pattern. However, in some cases, it may be ok...

Some time ago I wrote this post describing my understanding (at that time) of the architectures used to create test automation solution (framework). While there's some information I still agree with, my understanding has evolved and I want to share this understanding with others. First, lets probably coin the terminology I use (which is not necessarily would be the right one - feel free to suggest something different) Test automation framework It is a framework that allows one to write automated tests. Usually one means the specialized framework, i.e. framework that is specialized for one or several related applications under test. A framework does not include tests themselves Test automation solution It is is a complete solution used for test automation. It includes everything needed to perform the automated tests, including tests themselves. A solution may be based on a framework, however, it is not mandatory. Architecture May be described as an imaginary model t...