Software Metrics: What are they and how to track them


Software Metrics is a very controversial topic. A quick search on Google for good software metrics to track comes with lots of results. Ask any tester, manager or ask this question on online forums or raise it in tester gatherings and you will get a mix of answers. Some of them are mentioned below:

  • Test cases passed & Failed
  • Execution Productivity
  • Number of Defects Raised
  • Number of test cycles
  • Number of defects rejected by root cause
  • Number of duplicated defects
  • Cost of quality
  • Defect rejection ratio
  • Manual test case execution days
  • Number of code builds
  • Number of successful code builds
  • Duration of code builds
  • % availability of test environments
  • Average number of testers
  • Average manual test runs per tester per test execution day
  • Average manual runs per tester per effective test execution day
  • Total runs per tester per test execution day
  • Runs per manual test case
  • Number of defects by severity and total defects
  • Number of defects at end of UAT by severity
  • Automated test runs
  • Manual test runs
  • Total test runs
  • Effective test execution days
  • Total test cases for project
  • Number of manual test cases prepared during project
  • Number of test case writers
  • Effective test case preparation days
  • Defect Leakage
  • Return on Investment
  • Downtime log
  • Test Script Execution Time
  • Code Coverage
  • Requirement Volatility
  • Test case effectiveness
  • Defect Detection rate

Metrics can help guide your team and help you measure your team’s progress towards its goals. Metrics, on the other hand can also be a pit of wasted effort, numbers recorded for the sake of numbers.

Why do you need Metrics?

There are some really good and bad reasons to track metrics. Lisa Crispin and Janet Gregory write in their book Agile Testing

When Metrics are used as guideposts, telling the team when it’s getting off track or providing feedback that it’s on the right track, they’re worth gathering. Is our number of unit tests going up every day? Why did code coverage take a dive from 75% to 65%? It might have been a good reason – maybe we got rid of unused code that was covered by tests. Metrics can alert us to problems, but in isolation they don’t usually provide value.

Before you try to decide what metrics to collect, you should ask yourself this question. What am I going to do with this information? Some reasons why you might need to collect metrics are as follows:

  • To help estimate for future projects based on past experience.
  • To monitor progress within a programme or project.
  • To identify training needs of individuals or groups.
  • To estimate cost to completion.
  • To monitor progress of staff.

Identify the reason first and then decide what metrics you need to collect. Lastly, decide how to collect this information.

If your goals are measurable, the measurements you need to gather to track the metrics will be obvious.

Lean Measurements and ROI:

In their book, Implementing Lean Software Development: From Concept to Cash,Mary and Tom Poppendieck, go into detail about how one can apply lean initiatives to testing and developments. They explain that fundamental lean measurement is the times it takes to go “from concept to cash”, from a customer’s feature request to delivered software. They call this measurement “cycle time”. The focus then is on the team’s ability to “repeatedly and reliably” deliver new business value. The team tries to continuously improve their process and reduce cycle time. Another lean measurement used is financial return. If a team is developing a profitable product, it needs to understand how it can achieve the most profit. Even if there is no profit involved, a team should look into the Return on Investment to make sure its delivering best value. (1)

 This also echoes my point made above about knowing how to delight your customer and knowing what the ROI of this activity is. Don’t forget there is a cost overhead in collecting metrics:

  • Tooling and code manipulation to collect metrics.
  • Reporting of metrics to parties.
  • Analysis of metrics – to understand the impact points.

 Your activity of capturing metrics should pass the test of economics, wherein the benefits derived are far greater than the cost expended.

What Not to do with Metrics:

Metrics that involve the whole team are more likely to drive you towards success than are measures confined to isolated roles or groups. Metrics can also be used in a terrible way. One of these is to use them as a basis for an individual team member’s performance evaluation. One of my colleagues who works for a test consultancy in India was telling me about one of her previous work experiences where the company judged testers by the number of defects they found.

How does this help them do their jobs better? Bug counting results in the generation of meaningless metrics. I’ve raised 40 bugs in a half day of testing once; the code was still working, it just needed lots of tidying up before being released. On a different project, I raised two bugs in a single day of testing, each one of which on its own would have been sufficient to halt testing and forced developers back to rework things. When Microsoft released Windows 2000, there were over 65,000 known issues (bugs), yet is was a very successful product. IBM released an ALGOL compiler in the 1960s with one bug: it wouldn’t load. In short, not all bugs are equal.

Similarly, is it safe to say that a development team that produces a higher number of defects per lines of code is doing a bad job? Or a team that finds more defects is doing a good job? Even if the thought holds up, how motivating is it for a team to be whacked over the head with numbers? (2)

Some popular Metrics to Track:


  • Cyclomatic Complexity:

Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it.
– Alan Perlis, American Scientist

McCabe’s cyclomatic complexity is a software quality metric that quantifies the complexity of a software program. Complexity is inferred by measuring the number of linearly independent paths through the program. The higher the number, the more complex the code. Measurement of McCabe’s cyclomatic complexity metric ensures that developers are sensitive to the fact that programs with high McCabe numbers are likely to be difficult to understand and therefore have a higher probability of containing defects. The cyclomatic complexity number also indicates the number of test cases that would have to be written to execute all paths in a program.

  •  Code Coverage:

Another traditional metric to see how much of the code is exercised by tests. A lot of commercial and open source code coverage tools available which can be integrated into the build pipeline so you will immediately know if the coverage has gone up or down. One thing to note here is that these tools only measure the coverage of the code you have written. If you have missed some functionality, then this will not be spotted by these tools. You can have 90% of code coverage but you can also be missing 10% of functionality. TDD can help with that. (3)

  •  Defect Metrics:

Lots of metric combination can be found in this category. An important one for me is the number of defects found in production. This will tell you how your team did in the last release but it won’t tell you how good you are progressing in the current one. On the other hand it will give you an indication on what processes to change to reduce the number of defects.

  •  Iteration Metrics:

  1. How much work has the team completed
  2. How much is left to do
  3. Iteration burndown
  4. Estimated vs Actual time taken for tasks.
  • Automation and Manual Test Pass Rate:

Track the number of tests at each level. Unit, Functional, Story Level, GUI level, Performance etc. Although we get happy seeing the number of tests go up, but the numbers on their own are not enough. If you have a 1000 tests but they only give you 10% coverage, then this is not a good position to be in.

Further Reading:

Cem Kaner has provided a framework for understanding and rethinking software metrics. Read the following two short papers to know almost everything about software metrics.

  1. Software Engineering Metrics: What do they measure and how do we know
  2. The Darker Side of Metrics


(1) – Implementing Lean Software Development: From Concept to Cash

(2,3) – Agile Testing


Should Testers know how to code?

I was actively involved in recruitment of testers for in 2012 and came across an age old discussion once again which made me write about this topic. Should Testers know how to code?

Adding to this, there was also a statement made by Adam Goucher at CAST2012..

This is the last generation of testers that don’t know how to code

In my opinion, it totally depends on the situation. For example, in companies like Microsoft and Google, virtually all testers must know some form of coding. There are still companies out there which only require manual testers. If coding know how was not a requirement, I would also prefer to hire a tester who has critical thinking, analytical, investigative skills, good communication, understanding of risk and knowledge of common areas where bugs tend to hide as compared to someone who can just write C#.

 I also agree with Cem Kraner’s view below:

Testing within a business involves at least two types of knowledge: knowledge about testing and knowledge about the type of application under test. Much of the PR about the need for “software development engineers in test” has come from software companies. For them, software is the subject matter. For them, expecting someone to understand code is like a bank expecting someone to understand financial statements. It is unfortunate that the software publishers and software service providers have been speaking with a louder megaphone than the other industries: the result has been that some non-software companies are changing recruiting standards to look for people with stronger software knowledge instead of stronger industry (e.g. banking) knowledge.

Vacancies for “Developers in Test” or “QA Developers” are coming out more and more, especially in Agile environments. We need to remember, automation testing can never replace the need to do manual testing. There has to be a balance between the two. Skilled testers are adaptable people and should not be threatened by test automation. It’s a valuable skill to have and it’s not like learning how to code will make you forget manual testing. Secondly, the more the tester knows about coding, they’ll be able to do their jobs better, and the more career opportunities they will have (more on this later – I have some stats to prove this). Additionally, talking to developers in a language they can understand, writing clear bug reports which put your message across correctly and quickly has to be a good thing, right?

As a recruiter, you also need to use a common sense approach. I have seen companies that seem to use a standard list of demands, that they once created, and keep on using because they have been told it is a good list by the consultant they hired to create it, or they see other companies ask for the same skills, so they assume they are vital. You should look at the job spec, the role of a tester in your organisation and your team structure. What does a tester’s role involve, instead of what others testers in your company do/can do. Sometime you may find that the ability to write code is desirable instead of a requirement. If it is a requirement, state the level of familiarity with coding that you require. This way you will avoid putting off good candidates who will just be overwhelmed by the long list of requirements. Here at the Trainline, I have amended our requirements in job specs to be inline with the role of a QA Developer.

Lastly, what all programming skills should a tester have? This totally depends upon the job requirement and differs from one company to the other. Some positions are for automated testers and for this you require a good set of programming skills to create test frameworks. A recommendation for a manual tester or a tester thinking of improving their programming skills would be to:

  • Learn enough in any language (look at your target market or see research below) so you can write some test automation helper code, create classes, make asserts etc.
  • Learn enough of general programming concepts so that you can write code for typical easy programming exercises available on the internet.
  •  Once you have mastered this, drill deeper into object oriented design skills/concepts and create a mock test framework testing a website or an application.
  •  Also attend tester gatherings and talk to other testers/managers and see what skills they seek in a tester.

Now, let’s have a look at testing job specs taken from 2012. This research has been done by a UK based recruitment company.

Percentage of jobs that require Good/Exceptional coding skills:

Q1/Q2 2012 – 63%

Q3/Q4 2012 – 84%

Breakdown of languages required:

  Q1/Q2 2012  –  28% – Java, 26% – C#, 23% – Ruby, 16% – C, 7% – PHP

Q3/Q4 2012  –  39% – Java, 35% – C#,  8% – Ruby, 12% – C, 6% – PHP

The remaining  37% in Q1/2 2012 and 16% in Q3/4 2012 were a mixture of “we don’t need someone with any coding experience”, or “we purely need a Manual tester.”

Research also shows that salaries for testers with exceptional coding abilities have risen by £3500 per annum in the past 6 months.

What this means:

The research above suggests there are 21% more people being hired in the past 6 months alone with good to exceptional coding knowledge than without, however, there is clearly still a requirement for Manual testers.

Java is a still the main scripting requirement but Ruby has become increasingly popular.

Top Test Automation Technologies are:

  •   Selenium, including SeleniumRC and Webdriver
  •   QTP
  •   XUnit frameworks such as JUnit, NUnit, TestNG, etc
  •   BDD tools like Specflow/Cucumber
  •   Watir or Watin.

The above results are for the UK. A similar study was done in the US not long ago and you can read the details about that here.

Final Words:

Adam Goucher is probably right. But for now, manual testing and manual testers are here to stay. As more and more companies move towards the “Developer in Test” mentality, I see future testers taking up coding, especially if they want to work for the best companies and not to forget, these roles pay considerably more as compared to manual testing roles.

Do not expect everyone to understand testing!

As a tester, it is your job to tell others what you need in order to do your job properly. A tester is largely affected by the choices made by management and programmers. If other stakeholders are unclear about their plans or design a product which is hard to test, then this will affect your test plan. I’ts not that they do not understand or care about testing and software quality. They probably just do not understand what impact their actions can have on the test process. An important part of testing is to explain testing to other stakeholders.

Your explanations about testing are like a flu vaccine, healthy and not painful, but the effect wears off, so you have to do it over and over again.


Never dream or wish to have absolute control over the release of a product.

When testers control a release, they must also bear full responsibility of the quality of a product. What I have seen happen is that if any bugs are missed by the testers and make their way to production, the rest of the team (in most cases) will blame the testers. Why did the tester release such a buggy product? On the other hand, if a tester delays the release then they will come under pressure for being perfectionists or quality fanatics.

The people who control the project are the best suited to hold the responsibility to release the product.  So if you are ever given the authority to control a release, insist on sharing that authority with others in the team.


What is your Quality Philosophy?

I recently went to a Tester Gathering where this question was brought up. There were many answers to it. Some of these were:

  • Software should be stable.
  • The software does what it’s supposed to do from a functional perspective.
  • To deliver defect free code as best possible, accounting for schedule, risk and resources.
  • Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and is maintainable.
  • To deliver what is needed, in a quality that is accepted by the client.
  • Test early and test often

This turned out to be a very controversial topic and we had a very good debate about it. We talked about Six Sigma, Total Quality Management and even discussed Deming’s 14 Point Philosophy ( I recommend reading his work, more on him can be found here. A brief introduction is below).

Dr. W. Edwards Deming was a statistician who went to Japan to help with the census after World War II. Deming also taught statistical process control to leaders of prominent Japanese businesses. His message was this: By improving quality, companies will decrease expenses as well as increase productivity and market share. After applying Deming’s techniques, Japanese businesses like Toyota, Fuji, and Sony saw great success. Their quality was far superior to that of their global competitors, and their costs were lower. The demand for Japanese products soared – and by the 1970s, many of these companies dominated the global market.

I used to often ask this question when conducting interviews. My purpose for asking this question was to know how the candidate thinks. I often had candidates recite texts memorised from books which lead me to ask clarifying questions to satisfy my curiosity.  Many used to get caught out as they did not think about the answers and just memorised some text. This allowed me to initially separate good candidates from the average ones.

Quality philosophy is also dictated by the circumstances. Sure there are basic ideas of what quality means but if you’re looking for a quality philosophy, then it also depends on the domain, the company management, the company culture, the teams and the project.  It can also include quality in:

  • How a project is planned, designed, implemented and reviewed.
  • Reporting statuses among departments and department heads.
  • How requirements are written so Developers can produce the design with ease
  • Documentation Standards
  • Testing
  • Release Management etc.

Churchill said: “They say no one is perfect. Than they say that practice makes perfect. I wish they made up their damn mind!”

Talking about perfection or getting things absolutely right – As much as we would like our software to be perfect, it is a futile attempt to make it so. There comes a point where the time and expense to make a program perfect gives diminishing returns. In my experience so far, it is impossible to deliver ‘perfect quality’. There is always this or that little bit that could be improved, simplified, re-factored or documented better etc.

I also strongly believe in the fact that the client/customer has final say over quality – if the client is satisfied, the quality is ‘ok’. If the client is happy, the quality is ‘good’. If I ask this question in an interview I expect to hear “customer” somewhere in the answer. You must seek to identify the customer, whether external or internal to the organization, then advocate for that customer with your QA skill set. The QA Engineer should be happy when they think the customer will be happy. We try to measure that “happiness” with metrics and quality gateways before release, but it is the customer feedback after release that counts. The quality of project management, documentation standards, requirements acquisition and testing, all must be in sync with the ultimate customer requirements regarding quality. The cost of producing the end result, including all interim steps, must reflect end user/customer requirements to quality.

As I said earlier, it is a very controversial question but one which has no correct answer. Like your favourite colour, or car, or your love for Justin Bieber, your quality philosophy is what makes you a unique individual. There is no “right” answer to this, but it might indicate as to what ways you think about such things.

So what is my philosophy I hear you say? Well it’s this:

  1. Know that nothing is perfect and never will be.
  2. Be a professional and passionate about what you do. Give it your best shot and try to do better every day.

This is valid whether it is about testing, development,doing business, life in general or just being human.