If Your Security Operations Center is Measuring MTTR, Read This

One of the most common metrics in security is Mean Time to Resolve (MTTR), which measures the average time it takes for a security incident to be resolved. This is a useful metric for understanding how efficient the organization is at incident handling, as the quicker an incident can be resolved, the more likely some losses can be mitigated.

However, despite MTTR being one of the most common metrics measured, it should actually be thrown out the window. Yes, you heard that right. The reason? MTTR has a glaring issue that makes it borderline useless in the context of the Security Operations Center (SOC): how it handles unresolved incidents.

How to Calculate Mean Time to Resolve

MTTR has a rather simple calculation: over a defined period of time, take the average (mean) time to remediate across all incidents. This simplicity is a major contributor to its commonality within security teams.

However, because incidents in the SOC are an ongoing process, what you choose for your "defined period of time" can significantly impact the resulting calculation. For example, let's say you are putting together a presentation on SOC performance over the prior month. As part of this, you are asked to include the MTTR.

You write a query and pull all the incidents from that past month. Here is where you run into a problem: What about incidents that started prior to last month but were resolved during that month? What about incidents that started last month but were not resolved until this month? And most importantly: what about incidents that were active during last month but are still not resolved?

Many security teams decide to pull all incidents with resolution timestamps within the measurement period (for example, last month) and then calculate the MTTR. But as we can see above, this can become very misleading, as it does not take into account incidents that were not resolved within the time period being analyzed. You can have 100 incidents that were not resolved, yet this would not be reflected in your MTTR calculation.

Survival Analysis and Median Survival Time

So, what are security teams to do if they want to improve the accuracy of their MTTR? The good news is that this is already a well-defined problem in statistics, and as such, a solved one. The answer is survival analysis.

In essence, survival analysis is a statistical method for analyzing time-to-event data (such as the time-to-remediate) while accounting for events that have not yet experienced the event being measured (which is called censored data). Survival analysis is essentially answering the question, "what is the probability that a subject will or will not experience the event up to a specific time, given that not all subjects have yet experienced the event?"

For example, framed around time-to-resolve: what is the probability that an incident will or will not be resolved after _____ hours/minutes?

One of the primary outputs of survival analysis is a Kaplan-Meier Survival Curve, which plots the likelihood of not experiencing the event over time (called surviving). In the case of time-to-resolve, the survival curve plots the likelihood an incident will not yet be resolved over time.

The time interval of the curve at a survival probability of 0.5 is called the Median Survival Time. This is the duration at which point 50% of incidents are resolved. For example, if the median survival time for time-to-resolve is 5 hours, then we can interpret this as 50% of incidents will be resolved in 5 hours or less, and 50% of incidents will be resolved after longer than 5 hours. We can also say that there is a 50% chance that a given incident will be remediated within 5 hours.

Making the Switch from MTTR to Median Survival Time

While using the Median Survival Time instead of MTTR does take some additional work, it's work well spent. In fact, I would argue using MTTR causes more harm to your security program than not measuring anything at all, as inaccurate metrics used as decision tools can lead to worse decisions than those made without.