Some tip to fix flaky Web UI tests

-
Test automation at the Web UI level (you know, Selenium stuff) is usually brittle and painful. It is usually also the only test-automation approach people are aware of/interested in (for whatever reasons). Below are several suggestions on how one can make her/his UI test automation at the Web UI level less painful and flaky.

1. Retry failed steps

Each test consists of several steps. Unfortunately, those steps from time to time tend to fail with no apparent reason. Retrying a failed test step may save your whole test from failing.

Helps with:
  • Unresponsive, slow UI
  • Elements being shown with a delay
  • Wrong moon phase [1]
Drawbacks:
  • Sometimes it is just a waste of time, especially when the real issue is being "retried"
  • Even worse: sometimes it may hide real issue that would go away after some time or page refresh

2. Retry failed tests

From time to time - odds are just against you with a specific test run. Everyone, whoever had the misery of working with Web UI-test automation has seen this. Sometimes if you just rerun the problematic test – it will pass. Retrying the whole test or less similar to the previous suggestion, but takes more time and helps more often.

Helps with:
    • Unresponsive, slow UI
    • Elements being shown with a delay
    • Wrong moon phase

Drawbacks:
    • Sometimes it is just a waste of time, especially when the real issue is being "retried", and
    • Even worse: sometime it may hide real issue that happens from time to time

3. Smart test results reporting

Not all failures are of the same importance. Some failures indicate minor bugs, some show that disaster has happened. If you sort failures smartly it won’t make your test run pass, it will help you a lot with results investigation and getting valuable information about product quality (which is why we started this “automated testing/checking” in the first place). Also, it may be the case that all test failures were due to one/two issues, and sorting (for example by exception class) may help to understand a failure reason faster.

Helps with:
  • Finding what is wrong (or the biggest problem) when lots of tests had failed
  • Sorting/Prioritizing issues using “minor” to “catastrophic” scale
Drawbacks:
  • You will have to implement this, or integrate solutions from the market, like Allure or Report Portal

4. Break complex test into several simple ones

When Google wanted to know where those flaky tests did come from [2] they found an interesting correlation between the test size and the probability it would be a flaky test. Lesson learned: the shorter you test is, the less likely it will be flaky or fail due to the wrong Moon phase. It turns out that when tests are smaller and shorter, they are also easier to understand, maintain and investigate.

Helps with:
  • Wrong Moon phase
  • Scenarios which test too much (which is – more than one thing)
Drawbacks:
  • Sometimes it actually makes sense to have one big-uber test for a specific scenario, so this suggestion not always applicable. It is a heavily underused technique, though.

5. Do Arrange/Assert on a level below UI

If you don't touch UI, you obviously have fewer issues with UI!

Quite often, the test case itself is short, but there're lots of additional steps required. For example - when you test a Log In form you need not only navigate to the form itself and fill it with the correct data, you also need to make sure that the required user was already stored in the user database. Usually, this preparation is done using UI as well, which is highly inefficient. Instead, one can do Arrange and Assert steps using API or even DB directly. If you haven't heart about Arrange, Act, Assert (aka 3A) - feel free to check this post.

Helps with:
  • All pains of touching UI
Drawbacks:
  • It requires knowledge of lots of things in addition to Selenium, but one needs this at some point anyway.

6. Write as little Web UI tests as possible

Integrated tests are a Scam (as was nicely explained by J. B. Rainsberger) [3]. Web UI tests often are even bigger Scam. They don't force your design to be better. They don't help you with Emerging Architecture [4]. They don't emulate how the user will interact with your page (unless you think that user will really navigate elements using CSS classes and XPath). Automated testing does not really exist, it is not possible [5].  So, why not break the illusion and use more helpful techniques (like TDD) and better test levels (like unit test [6])?

Helps with:
  • All pains of touching UI
  • Untestable, rigid architecture

Drawbacks:
  • It requires knowledge of lots of things in addition to Selenium, but one needs this at some point anyway.
Please stay tuned - at some point, I am going to give code examples for the suggestions above.


[1] Wrong Moon Phase - Russian idiom, means that something happened for a reason which is either too complex or too costly to identify
[2] Google Testing Blog: Where do our flaky tests come from?: https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html
[3] Integrated tests are a Scam - J. B. Rainsberger: https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam
[4] Emergent Architecture - just enough, just in time: https://www.agilealliance.org/resources/sessions/emergent-architecture-just-enough-just-in-time/
[5] Automated testing is not possible - my takeaway from this excellent twitter thread: https://twitter.com/noahsussman/status/1066461811669696514
[6] What are the differences between unit test, integration test and end to end test in automated testing? - J. B. Rainsberger: https://www.quora.com/What-are-the-differences-between-unit-test-integration-test-and-end-to-end-test-in-automated-testing/answer/J-B-Rainsberger?ch=10&share=efcede70&srid=jbr

Comments

Post a Comment

Popular posts from this blog

Test automation framework architecture. Part 2.1 - Layered architecture example

-
Image
Let's consider an example of what Layered (Tiered) architecture may look like for a test automation framework. As a system under test, we're going to use this simple and neat "Todo list" application. First, let's do a brief analysis of the application. From my brief exploration I came up with the following checklist: Area/Group Things to check Comments Todos CRUD 1 Create item Basic scenario, UI-level case 1.1 Create an item with the same name The case is not valid. There's no validation for that - nothing to check really. 1.2 Create an item with an empty/not valid/malicious name While it would be a nice thing to test manually, from test automation standpoint I would expect this to be filtered on service or repository level, so either integration or unit test . 2 Mark item complete Basic scenario, UI-level case 2.1 Un-mark item complete Basic scenario, UI-level case 3 Delete item Basic scenario, UI-level case 3.1 Delete item
Read more

Test automation framework architecture. Part 2 - Layered architecture

-
Image
Probably the most popular architecture pattern used for test automation frameworks (TAF) is layered architecture. This pattern is so well known that on job interviews for some companies when they ask you about TAF architecture you are supposed to describe this one. If you don't - they think you know nothing about the architecture altogether. I suggest you first read a brilliant description of the pattern at the O'Reilly web page , cause in this post I am going to describe the pattern in a way it is usually applied to build test automation solution. Usually, there're three distinct layers, which may have different names, but follow the same logic mostly. Sometimes those layers called test layer , business-layer and core layer , but there're no standard names really. Key rules for layered architecture are the dependency direction (each level depending on the level below) and call direction (no level can call/reference code described in the level above). The rough
Read more

Efficient test automation - Secret #1 | Test Club

-
Image
It seems everyone talks about efficient test automation, but not so many people have actually seen it! To cover this I decided to make several videos regarding the secrets of efficient test automation. Today we will talk about the first (and, arguably, the main) secret - realistic expectations. Very often, test automation projects or initiatives are started with the following unrealistic expectations: "Test automation will replace 'manual' testing" "Test automation will provide fast results and benefits" "Test automation efficiency depends solely on test automation engineers" Unfortunately, all those expectations are not realistic! Watch the video to know more!
Read more