What does the Test Pyramid mean?

Have you seen the "Test Pyramid" yet? There are dozens of versions going around in the Internet, most look somewhat like these two:

Common test pyramids seen on the Internet and in Testing Literature

Have you ever wondered what these pyramids actially represent? What are the underlying assumptions of these models?

What they all have in common is that they start with "unit test" at the bottom.

The first question we need to answer is: "What is a unit test?" Unit test specifically implies testing a component in isolation. This is often understood as "The tests that developers do [on their machines]." Where this concept exists in the back of people's heads, the test pyramid represents -from bottom to top- a separation of concerns and the breadth of the pyramid connotates distance from real customers. Is that the intention?

Let us dig into the left model.

UI-Acceptance-Unit Pyramid

There are three basic assumptions in this model:

1. UI testing and Acceptance testing are two different things
2. Acceptance Testing does something other than unit testing
3. The UI does not get unit tested.

Think about it for a minute. 

What are "UI tests" that have nothing to do with product acceptance? What do unit tests actually do if they do not verify acceptance criteria of the product/ feature/ requirement? Why is the UI, if it's not a unit (= component) of the product? Why would you not unit test one of your units?

A proper understanding of unit testing and the application of modern UI testing frameworks reveals the inconsistencies in this model. 

UI-Service-Unit Pyramid

This model makes similar assumptions:

1. The UI tests are not part of service tests (i.e. the UI does not belong to the service)
2. Service Testing does something other than unit testing.
3. The UI does not get unit tested.

Take your time to reflect on these assumptions.

Where is the UI, when it's not part of a service? What happens during "service testing"? Why do we need "service tests" after we have proper unit coverage? 

This model can be made consistent by clarifying some of the underlying assumptions:

1. "UI" tests actually mean "user testing", where real people use the real product.
2. "Service" tests actually mean "integration testing", or unit interaction testing.

With these clarifications, the model correlates to maturity of the product, and therefore, to the timeline on which tests are made (up = late, bottom = early). It also makes sense to assume that most tests should be made early. 

This model presumes an undesirable structure: The assumption that these activities are indeed separate and impossible to merge or refactor. Therefore, we get stuck with the process in it's current state. Organizational redesign for optimizing the pyramid is pre-empted by the underlying assumptions.

A different Test Pyramid

This model also has underlying assumptions, but they are completely different from the previous models:

1. On an E2E level, the object under test would be the entire product as set up to run in a production-like environment.
   However, that is expensive and difficult to implement and slow to execute. When defects are found, they could have been caused anywhere, anytime in the system, making debugging difficult and tedious.
2. On integration level, the object under test is narrowed down significantly.
   Integration tests are significantly cheaper and easier to implement and faster to execute than E2E. When defects are found, they can be much easier to locate.
3. On component level, the object under test is reduced to minimal scope, which is - in a perfect state - just the single method under test.
   Component tests execute in milliseconds and defects coincide with the failing test. Debugging can quickly pinpoint the faulty line of code.

This model proposes that complexity correlates to the depth of the pyramid. It connotes that you can start anywhere, yet it is desirable to reduce testing complexity by refactoring tests down in the stack.

Summary

No model is really wrong, but some are inconsistent. Models serve to facilitate a conversation. In our case, we created the Test Pyramid as a model for the purpose of explaining the relationship between complexity and effort of testing. Our model is intended to foster an understanding of why high component test coverage is desirable and how to discover if your test suite is adequately built.