Software Metrics is a very controversial topic. A quick search on Google for good software metrics to track comes with lots of results. Ask any tester, manager or ask this question on online forums or raise it in tester gatherings and you will get a mix of answers. Some of them are mentioned below:
- Test cases passed & Failed
- Execution Productivity
- Number of Defects Raised
- Number of test cycles
- Number of defects rejected by root cause
- Number of duplicated defects
- Cost of quality
- Defect rejection ratio
- Manual test case execution days
- Number of code builds
- Number of successful code builds
- Duration of code builds
- % availability of test environments
- Average number of testers
- Average manual test runs per tester per test execution day
- Average manual runs per tester per effective test execution day
- Total runs per tester per test execution day
- Runs per manual test case
- Number of defects by severity and total defects
- Number of defects at end of UAT by severity
- Automated test runs
- Manual test runs
- Total test runs
- Effective test execution days
- Total test cases for project
- Number of manual test cases prepared during project
- Number of test case writers
- Effective test case preparation days
- Defect Leakage
- Return on Investment
- Downtime log
- Test Script Execution Time
- Code Coverage
- Requirement Volatility
- Test case effectiveness
- Defect Detection rate
Metrics can help guide your team and help you measure your team’s progress towards its goals. Metrics, on the other hand can also be a pit of wasted effort, numbers recorded for the sake of numbers.
Why do you need Metrics?
There are some really good and bad reasons to track metrics. Lisa Crispin and Janet Gregory write in their book Agile Testing
When Metrics are used as guideposts, telling the team when it’s getting off track or providing feedback that it’s on the right track, they’re worth gathering. Is our number of unit tests going up every day? Why did code coverage take a dive from 75% to 65%? It might have been a good reason – maybe we got rid of unused code that was covered by tests. Metrics can alert us to problems, but in isolation they don’t usually provide value.
Before you try to decide what metrics to collect, you should ask yourself this question. What am I going to do with this information? Some reasons why you might need to collect metrics are as follows:
- To help estimate for future projects based on past experience.
- To monitor progress within a programme or project.
- To identify training needs of individuals or groups.
- To estimate cost to completion.
- To monitor progress of staff.
Identify the reason first and then decide what metrics you need to collect. Lastly, decide how to collect this information.
If your goals are measurable, the measurements you need to gather to track the metrics will be obvious.
Lean Measurements and ROI:
In their book, Implementing Lean Software Development: From Concept to Cash,Mary and Tom Poppendieck, go into detail about how one can apply lean initiatives to testing and developments. They explain that fundamental lean measurement is the times it takes to go “from concept to cash”, from a customer’s feature request to delivered software. They call this measurement “cycle time”. The focus then is on the team’s ability to “repeatedly and reliably” deliver new business value. The team tries to continuously improve their process and reduce cycle time. Another lean measurement used is financial return. If a team is developing a profitable product, it needs to understand how it can achieve the most profit. Even if there is no profit involved, a team should look into the Return on Investment to make sure its delivering best value. (1)
This also echoes my point made above about knowing how to delight your customer and knowing what the ROI of this activity is. Don’t forget there is a cost overhead in collecting metrics:
- Tooling and code manipulation to collect metrics.
- Reporting of metrics to parties.
- Analysis of metrics – to understand the impact points.
Your activity of capturing metrics should pass the test of economics, wherein the benefits derived are far greater than the cost expended.
What Not to do with Metrics:
Metrics that involve the whole team are more likely to drive you towards success than are measures confined to isolated roles or groups. Metrics can also be used in a terrible way. One of these is to use them as a basis for an individual team member’s performance evaluation. One of my colleagues who works for a test consultancy in India was telling me about one of her previous work experiences where the company judged testers by the number of defects they found.
How does this help them do their jobs better? Bug counting results in the generation of meaningless metrics. I’ve raised 40 bugs in a half day of testing once; the code was still working, it just needed lots of tidying up before being released. On a different project, I raised two bugs in a single day of testing, each one of which on its own would have been sufficient to halt testing and forced developers back to rework things. When Microsoft released Windows 2000, there were over 65,000 known issues (bugs), yet is was a very successful product. IBM released an ALGOL compiler in the 1960s with one bug: it wouldn’t load. In short, not all bugs are equal.
Similarly, is it safe to say that a development team that produces a higher number of defects per lines of code is doing a bad job? Or a team that finds more defects is doing a good job? Even if the thought holds up, how motivating is it for a team to be whacked over the head with numbers? (2)
Some popular Metrics to Track:
Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it.
– Alan Perlis, American Scientist
McCabe’s cyclomatic complexity is a software quality metric that quantifies the complexity of a software program. Complexity is inferred by measuring the number of linearly independent paths through the program. The higher the number, the more complex the code. Measurement of McCabe’s cyclomatic complexity metric ensures that developers are sensitive to the fact that programs with high McCabe numbers are likely to be difficult to understand and therefore have a higher probability of containing defects. The cyclomatic complexity number also indicates the number of test cases that would have to be written to execute all paths in a program.
Another traditional metric to see how much of the code is exercised by tests. A lot of commercial and open source code coverage tools available which can be integrated into the build pipeline so you will immediately know if the coverage has gone up or down. One thing to note here is that these tools only measure the coverage of the code you have written. If you have missed some functionality, then this will not be spotted by these tools. You can have 90% of code coverage but you can also be missing 10% of functionality. TDD can help with that. (3)
Lots of metric combination can be found in this category. An important one for me is the number of defects found in production. This will tell you how your team did in the last release but it won’t tell you how good you are progressing in the current one. On the other hand it will give you an indication on what processes to change to reduce the number of defects.
- How much work has the team completed
- How much is left to do
- Iteration burndown
- Estimated vs Actual time taken for tasks.
Automation and Manual Test Pass Rate:
Track the number of tests at each level. Unit, Functional, Story Level, GUI level, Performance etc. Although we get happy seeing the number of tests go up, but the numbers on their own are not enough. If you have a 1000 tests but they only give you 10% coverage, then this is not a good position to be in.
Cem Kaner has provided a framework for understanding and rethinking software metrics. Read the following two short papers to know almost everything about software metrics.
(2,3) – Agile Testing