A QA-guy blog

While there're lots of information published on the internet about how one should develop/buy a Test Automation solution, there's an elephant in the room - which is the fundamental question "Do I need to do it at all?". That is a good question. While I think that there’re some legitimate contexts where product/project/team can benefit from having a sophisticated test automation solution/framework, my general rule of thumb is “If you need a framework - you do it wrong” . There’re at least several decent papers showing the benefit of unit-testing ([1],[2], [3]), while I could not find any research proving that system level tests could bring better results. Please let me know if you can prove me wrong. There’s at least one half-scientific paper showing the benefits of unit tests in comparison to integration/system level tests [4] There’s a good paper bringing a notion of running/writing and maintaining test vs. the benefit of having/running it at...

Let's assume a hypothetical situation - you were assigned to a project to help with the "test automation" initiative. You have a huge "test plan" as an input, containing hundreds (if not thousands) of "test cases" and you need to do something about it, quick. Problem statement Your first urge (if it is a Web application) may be to write some UI-level automated checks using tools like, you know, Selenium. In fact, there's a huge demand for such "Selenium test automators" in the industry these days. But please, please, don't do this. There're lots of things which make UI-level automated checking the least desirable approach: UI-level automated checks will be slow. There's just no way to avoid it. You can parallel them or do some other tweaks to speed them up somehow, but they still will be slow. UI-level automated checks will be flaky. Partly - because they're slow. Partly because Web browser and UI interface w...

Test automation at the Web UI level (you know, Selenium stuff) is usually brittle and painful. It is usually also the only test-automation approach people are aware of/interested in (for whatever reasons). Below are several suggestions on how one can make her/his UI test automation at the Web UI level less painful and flaky. 1. Retry failed steps Each test consists of several steps. Unfortunately, those steps from time to time tend to fail with no apparent reason. Retrying a failed test step may save your whole test from failing. Helps with: Unresponsive, slow UI Elements being shown with a delay Wrong moon phase [1] Drawbacks: Sometimes it is just a waste of time, especially when the real issue is being "retried" Even worse: sometimes it may hide real issue that would go away after some time or page refresh 2. Retry failed tests From time to time - odds are just against you with a specific test run. Everyone, whoever had the misery of working with Web UI...