Home > Technical > Testing Pyramid - A Case Study

Testing Pyramid - A Case Study

March 5th, 2012

Test automation is prevalent in the software development community. Practices like TDD and BDD are widespread and applied almost unquestionably. However, several organisations have struggled in attempting to scale automated test suites, which very often become slow, brittle, non-deterministic, unreliable and difficult to maintain.

One common issue reported by many teams and that I have also experienced many times is the inverted testing pyramid.

A year ago I joined a project that was going in that direction. I could see us making the same mistakes again. If we had kept going we would have ended up with the slow and hard to maintain, melting ice-cream cone. However, this time we tried a different approach. Our test strategy was heavily based on the concept of Shallow Depth of Tests, which means testing the code as close as possible to where it is written using preferably either unit or integration tests. We only automated high level test journeys at the UI level.

Now, one year later, we have a stable and fast build, which gives us an extremely high level of confidence.

Here is our project’s testing pyramid, of which we are very proud:

Testing Pyramid Fabio Pereira

Some observations:

  • Only 12 tests through the UI take 13 minutes to run
  • 1748 unit tests take only 1 minute
  • 273 JavaScript unit tests take less than 1 second. Treating JS as a 1st class language helped us as well

Feel free to share your testing pyramid as well. And always keep an eye on it… It can make a big difference to your project.

Technical ,

  1. March 6th, 2012 at 01:13 | #1

    Are you maintaining this as a visual control?

  2. March 6th, 2012 at 01:47 | #2

    @Jason: on this project we only updated the pyramid manually a couple of times, but it’s a great idea for the next one. The diagram seems to carry the message pretty well so I can see the benefit in keeping it up to date - even maybe generating it from our CI server.

  3. March 7th, 2012 at 19:36 | #3

    Love it. Do you have stats that could overlay coverage on each layer?

  4. March 8th, 2012 at 02:24 | #4

    @simon, we do keep coverage on Sonar, but not separately per layer, that’s a good idea tough.

  5. June 11th, 2012 at 20:36 | #5

    Why does it take 13 minutes to run 12 functional tests?

  6. June 11th, 2012 at 22:33 | #6

    @Nick Carroll The functional tests through the UI are very high level “User Journeys”, so they are very full scenarios that take longer than one usual test. That’s why they take longer…

  1. No trackbacks yet.