Testing Pyramid - A Case Study
Test automation is prevalent in the software development community. Practices like TDD and BDD are widespread and applied almost unquestionably. However, several organisations have struggled in attempting to scale automated test suites, which very often become slow, brittle, non-deterministic, unreliable and difficult to maintain.
One common issue reported by many teams and that I have also experienced many times is the inverted testing pyramid.
A year ago I joined a project that was going in that direction. I could see us making the same mistakes again. If we had kept going we would have ended up with the slow and hard to maintain, melting ice-cream cone. However, this time we tried a different approach. Our test strategy was heavily based on the concept of Shallow Depth of Tests, which means testing the code as close as possible to where it is written using preferably either unit or integration tests. We only automated high level test journeys at the UI level.
Now, one year later, we have a stable and fast build, which gives us an extremely high level of confidence.
Here is our project's testing pyramid, of which we are very proud:
Some observations:
- Only 12 tests through the UI take 13 minutes to run
- 1748 unit tests take only 1 minute
- 273 JavaScript unit tests take less than 1 second. Treating JS as a 1st class language helped us as well
Feel free to share your testing pyramid as well. And always keep an eye on it… It can make a big difference to your project.