In our last article, Part 5: Assigning Harms and Severity, we addressed the importance of doing your “homework” before attempting to assess the severity of identified hazards/harms, clearly defining the use environment and user profile being considered, and distinguishing between harms that occur as a direct result of the hazard/hazardous situation and harms that will only occur as the result of a potential series of subsequent events. This article addresses the other major element of risk, the Probability of Occurrence of Harm.
If you have missed any of our previous articles in this series, you can find them on our website at here.
The Elements of the Probability of Occurrence of Harm
Risk is composed of two elements: Severity and the Probability of Occurrence of Harm. Unlike the assessment of Severity, which is typically based on qualitative criteria used to describe the harm caused to people, property or the environment; the estimation of the Probability of Occurrence of Harm is typically based on quantitative criteria (i.e., hard numbers derived from field experience and/or engineering studies) – but that doesn’t mean it’s any easier.
First, there are a few things that you need to keep in mind:
- A failure is not necessarily a hazard. The failure of a device is only a hazard if it can cause harm to people, property, or the environment through a foreseeable sequence of events. There are few medical device failures that could not conceivably result in harm—even a device that fails in a safe condition could result in a delay of treatment—but the “failure” vs. “hazard” distinction is important to keep in
- If there is no exposure to a hazard, there can be no harm. TIR 24971 states that “Medical devices only cause harm if a sequence of events occurs that results in a hazardous situation, which then causes or leads to harm. Sequences of events can include a chronological series of causes and effects, as well as combinations of concurrent events. A hazardous situation occurs when people, property or the environment are exposed to one or more hazards.“ (Section 5.4.2). So “exposure” is a key element in the estimation of risk—more on this later.
- Estimating Probability of Occurrence of Harm requires a clearly defined harm. In most cases, a hazard can be associated with a range of harms (e.g., a cut could lead to infection, and then illness, permanent damage to the limb, and possibly death). In our previous article, Assessing Severity, we describe an approach for determining the harm (and associated Severity score) that is most appropriate for a particular hazardous situation. If there are multiple, potential harms, there is likely a different Probability of Occurrence for each of those harms. Typically, harms that occur only as the result of a long sequence of events are less likely to occur than harms that are an immediate result of a device failure. You will need to clearly define the harm that is the subject of your analysis (and the sequence of events leading to that harm) to effectively estimate the Probability of Occurrence of
- The Probability of Occurrence of Harm estimate needs to be “per use.” Too often we come across risk analyses where the Probability of Occurrence units are unstated or ambiguous. The only practical way to characterize these estimates is on a “per use” basis. For single-use devices, it’s pretty simple: the Probability of Occurrence of Harm is the likelihood that the harm occurs per every 1,000; 10,000; 1,000,000, etc. uses of the device. If you assume that all single-use devices are used for their intended purpose, you can use device sales as the denominator for calculating occurrence rates. The numerator is the number of times that the harm occurs (e.g., “In 2019, we sold 100,000 units and received reports of 2 injuries that year;” therefore, the Probability of the Occurrence of Harm is 2 injuries per 100,000 uses, or a probability of 0.00002).
For reusable devices, you’ll need to make some assumptions about the expected number of times the device will be used per year and how many are in the field (e.g., If you have 1,000 units in the field and each are used about 250 times per year, that’s 250,000 uses per year). The full benefit of making “per use” estimates becomes clear once you start collecting complaint data. With risk and complaint data described in the same terms, you’ll be in a much better position to identify when gaps exist between expected occurrence rates and reality.
The standard (ISO 14971:2019) brings all the elements of risk estimation together in one graphic in Annex C (see Figure 1). In addition to “Severity of Harm” and “Probability of Occurrence of Harm,” this graphic introduces two new terms:
- The probability of a hazardous situation occurring (P1); and
- The probability of a hazardous situation leading to harm (P2).
The product of those two terms is the Probability of Occurrence of Harm (P).
Figure 1: Components of Risk (From ISO 14971:2019 Figure C.1)
The consideration of both terms is a key element of the standard, but it is overlooked by many device manufacturers. That’s not too surprising—the standard doesn’t provide much guidance on how to work with these two terms.
Conceptually, the use of the two terms is pretty straight-forward . . . for example, if there’s a hot surface (the hazard) on a device, the hazardous situation (P1) occurs when the user/patient comes into contact with that surface (i.e., exposure to the hazard). The probability that the hazardous situation leads to harm (P2) depends on the harm. A first-degree burn might be fairly common, a second-degree burn less common, and a third-degree burn very rare. If you assign probabilities to it, P1 may occur once every ten uses (P1 = 0.1). P2 will vary with the harm so the resulting Probability of Occurrence of Harm will also vary as shown in Table 1.
Table 1: Example of the use of P1 and P2
Note: In this example, the P2 results are independent events (i.e., you can’t have multiple type of burns) that sum to 0.1101. This result implies a probability of 0.8899 that there is no harm when exposed to the hot surface (i.e., 88.99% of the time the user is not harmed when exposed to the hot surface).
Most manufacturers do not explicitly estimate both P1 and P2 and report only a single “Probability of Occurrence of Harm” score in their FMEAs—the two terms (P1 and P2) are implicit in this single score. The problem is that the single score doesn’t provide you with much information. Does a “Moderate” score mean that that there is a moderate chance that the hazardous situation (exposure to the hot surface) will occur and a moderate chance that it will lead to harm? Or does it mean that there is a high chance of exposure but only a slight chance that it will result in harm—leading to an overall “Moderate” probability of harm? There is no way to tell.
Does that matter? . . . Maybe. It may affect how you decide to mitigate the risk. If the surface is always hot, a design change to reduce the temperature or place a guard over the surface may be the best approach. If the surface gets hot only once every ten uses, an alarm to warn the user may be appropriate. (Of course, an alarm wouldn’t make sense in the first scenario because it would be on all the time.)
These two risk control options illustrate how P1 and P2 differ. If you are able to reduce the temperature of the hot surface, P1 would not be affected—exposure to the hot surface would stay the same, but the probability of the exposure leading to harm would decline (you’d be less likely to cause a severe burn if the temperature is lower). Alternatively, if you put a guard around the hot surface, P1 should go way down (if the guard is effective, exposure should be rare). But if you do come into contact with it, the P2 values would be unchanged.
In a perfect world with sufficient information, you could create a table of all relevant device hazards, hazardous situations, and harms, and list out the P1 and P2 probabilities, multiply them together to get a value for the Probability for the Occurrence of Harm, and then assign a score to the probability level using a tool like the one in Table 2.
Table 2: Example of Semi-Quantitative Probability Levels (based on TIR 24971, Table 5)
Combining the information from Table 1 and Table 2 (and your Severity ratings), you can create a risk analysis that looks like the following:
Table 3: Example Risk Table with P1 and P2 Broken Down
In this example, the estimates from the P1 and P2 columns are multiplied and the results are recorded in the “Prob. Occ. of Harm” column. Those results are given a rating based on Table 2, and the Occurrence Rating and Severity Rating are assessed to provide an overall Risk Rating (i.e., Acceptable or Not Acceptable). [Note: we’ll discuss Risk Acceptability more in a future article.]
Alternatively, the risk analysis can be set up to go right to the Probability of Occurrence of Harm rating as shown in Table 4. As you can see, there is a lot less information describing how you got to the final rating. You know that 1st Degree Burns will happen frequently (just as described in Table 3), but there is less information describing why.
Table 4: Example Risk Table with the Probability of Occurrence of Harm (POH) Rating Only
So which approach should you take? It depends on how much information you have about your device. Working though the P1 and P2 scenarios should help you to think through the chain of events that lead to a harm and, a year or two down the road, provide you and others with a better understanding about how the estimates were established. That said, if you have a novel device and little experience with the likelihood of these events, attempting to break the estimates down into its component parts may not be valuable.
One approach to avoid is to assign “1 to 5” occurrence ratings to both P1 and P2 and then try to combine those scores to produce a final Probability of Occurrence of Harm rating. Since each 1 to 5 occurrence rating cover a wide numeric range, you would need to establish additional rules to use it consistently. For example, if you give P1 an occurrence rating of “2” (based on Table 2), does that mean the value is 10-5, 10-6, or something in-between? If you want to break the Probability of Occurrence of Harm into P1 and P2, use numbers. If you don’t want to assign numbers to P1 and P2, stick with a single “1 to 5” Probability of Occurrence of Harm rating for each harm.
This seems like a lot of work . . .
Yes, it can be. The question is what level of rigor is needed to produce an effective risk analysis—and by “effective analysis,” we mean an analysis that helps you make effective decisions for how to improve the safety of your device. As we described in an earlier article on setting priorities before your team jumps into the details of an FMEA, you should take a high-level view of the risks associated with your device to identify the high priority areas. The rigor that you apply to the analysis of specific aspects of your device and its uses should be proportional to the significance of the potential risks.
If the only imaginable harms associated with your device are of “Minor” severity, you may be able to go right to your FMEA and not have to worry too much about dissecting the Probabilities of Occurrence of Harm into all of its component elements. But if there’s any indication that your device could cause a serious injury or death, it’s your responsibility to identify those priority areas and complete a thorough analysis. That analysis may involve thinking through P1 and P2 as we’ve discussed in this article, constructing FTAs to better understand how best to mitigate particularly complex hazardous situations, or applying some other tool that is appropriate to the particular situation. The key is that you use these tools to gain as much of an understanding of the device risks as possible before releasing your product to the market—so that you can provide patients and users with the safest possible products.
MEDIcept … Trusted Solutions, Rapid Response …
About Us
MEDIcept Inc. is an international consulting firm specializing in medical device, IVD, and biotechnology Regulatory, Quality, and Clinical Services. Since 1996, we have worked with thousands of companies to solve their most critical FDA and ISO issues. Our integrated solutions are rooted in our direct experience and span all stages of the product life.
MEDIcept is committed to providing our clients with what they need. We are committed to quality deliverables because we value our clients’ time and resources. This is why 90% of our clients come back to us again and again to solve new issues.
For additional information, please contact Susan Reilly at SReilly@MEDIcept.com.