Test automation framework architecture. Part 2.1 - Layered architecture example

Let's consider an example of what Layered (Tiered) architecture may look like for a test automation framework. As a system under test, we're going to use this simple and neat "Todo list" application.

First, let's do a brief analysis of the application. From my brief exploration I came up with the following checklist:

Area/Group	Things to check	Comments
Todos CRUD	1 Create item	Basic scenario, UI-level case
	1.1 Create an item with the same name	The case is not valid. There's no validation for that - nothing to check really.
	1.2 Create an item with an empty/not valid/malicious name	While it would be a nice thing to test manually, from test automation standpoint I would expect this to be filtered on service or repository level, so either integration or unit test.
	2 Mark item complete	Basic scenario, UI-level case
	2.1 Un-mark item complete	Basic scenario, UI-level case
	3 Delete item	Basic scenario, UI-level case
	3.1 Delete item when there're multiple items with the same name	Knowing that there's no validation this is a nice scenario to try. However, I don't think that it is a good scenario for UI verification. Basically what we check there is that some other field (not name) is used for delete query generation (assuming it is stored in DB). Thus it is probably a great scenario for integration or even unit test, depending on the realisation.
Filtering	4 All	Basic scenario, may be combined nicely with some other scenario, just to save time, UI-level case
	5 Active	Basic scenario, UI-level case
	6 Completed	Basic scenario, UI-level case
Bulk action	7 Delete several completed items	Basic scenario, UI-level case
	7.1 Delete one completed item	Wonder what the difference from the first case? One interesting thing is that it seems the "bulk action" button should appear if we have at least one item selected. Apart from that deleting one time or 100 items should be just slightly different SQL-queries generated (assuming DB is used), so I would rather not add this as a separate scenario.
	8 Select all items	Pretty simple pure UI thing to do. Depending on the UI implementation may be a good candidate for JS unit-test
Other	9 Valid item list shown	Nothing special. Maybe combined nicely with some other scenario. Pretty minor thing anyway so I would rather not add a separate UI case for that.

Please notice, that  I did checklist first - thus I make test scenarios to drive my framework implementation, not vice versa. I think it is an important premise as our final product are tests. Let us call it "intent-driven test automation design" (or choose/suggest any other meaningful name).

Typical anti-pattern would be to start working on PageObjects first, then on Service/Step layers and only after that on test scenarios. Thus you usually get ugly and unreadable scenarios.

A typical layered framework is divided into three layers - Web (contains Page objects ), Business (contains business abstractions and exposing "steps" abstraction above) and Tests themselves (check this short post for additional info). For a record, I don't like thinking about Tests as a layer of a framework (rather it is a layer of a solution, please check this) but common vocabulary differs from mine.

Traditional three-layered framework/solution

Most test automation frameworks I worked with were "plain code" frameworks, where test scenarios are done using some XUnit library as pure test methods, resembling typical unit tests. The example I have created uses this approach as well, with a major difference - I inversed dependency so my lower level layers depend on a higher layer (this is inspired by Clean Architecture by Uncle Bob and Hexagonal Architecture pattern).

Mapping between layers and code itself:

Test layer	/src/main/java/gmail/alexspush/test /src/test/java/gmail/alexspush/test
Business (Steps/Service) Layer	/src/main/java/gmail/alexspush/service
Core Layer	/src/main/java/gmail/alexspush/utils /src/main/java/gmail/alexspush/driver /src/main/java/gmail/alexspush/page

All in all, this is a good enough solution. Tests are descriptive and tell the reader what they're testing (instead of how they do it). Of course, as you add test scenarios one should be very careful to preserve this communication, but still good enough already. The weakest point of this solution is a standard reporting, but it can be improved using external libraries (or ignored if you run tests often enough so you know a source of failure without reporting)


Github link: https://github.com/senpay/layered-test-framework-example

Traditional three-layered framework/solution + Serenity for reporting

A good compromise between "Vanilla BDD" and classical test automation would be using tools like Serenity. Without changing much of implementation I was able to add Serenity to the layered framework to produce readable and descriptive HTML-report.

The downside of this was that I had to use Serenity build in dependency injection mechanism, which killed my idea with interfaces. I am sure it can be worked around, but I decided that it was not necessary to prove a point - there's no much difference between frameworks disregard of what reporting/test layer specifics one choose.

Mapping between layers and code itself:

Test layer	/src/main/java/gmail/alexspush/test /src/test/java/gmail/alexspush/test
Business (Steps/Service) Layer	/src/main/java/gmail/alexspush/service
Core Layer	/src/main/java/gmail/alexspush/utils /src/main/java/gmail/alexspush/driver /src/main/java/gmail/alexspush/page

Github link: https://github.com/senpay/layered-test-framework-example-serenity

Vanilla BDD three-layered framework/solution powered by Serenity and Jbehave

The main difference of this in comparison to previous examples is that steps now have to worry about test state (while in two previous examples test state was handled by test code itself). Another drawback is the inability to run a single test instead of a full suite (not a problem if you have good CI though).

All in all steps in a Business Layer were changed only cosmetically and JBehave annotations were used to map Gherkin scenarios to steps.

The mapping between layers and code itself:

Test layer	/src/test/resources/stories
Business (Steps/Service) Layer	/src/main/java/gmail/alexspush/service
Core Layer	/src/main/java/gmail/alexspush/utils /src/main/java/gmail/alexspush/driver /src/main/java/gmail/alexspush/page

Github link: https://github.com/senpay/layered-test-framework-example-serenity-jbehave

I hope that these were useful examples - I have a couple of other nice examples in my Github, feel free to check them as well.