Introduction
One of the most challenging elements of risk analysis is the assignment of Harms and Severities to a particular hazardous situation. On the surface, describing the consequences of the sequence of actions that lead to exposure to a hazard seems pretty straight-forward: establish a set of criteria describing increasing levels of Harm (e.g., from “Negligible” to “Catastrophic”) and select the Severity level most appropriate for each hazard in your analysis (typically scored on a 1 to 5 scale). Unfortunately, like most things, the devil is in the details. If you’ve ever participated in the assignment of Harms in a risk analysis, the following discussion among Risk Analysis Team members may sound familiar:
Facilitator: Alright, moving on to line 37, the harm is an electrical shock to the user due to improper grounding. Are there any recommendations for a Severity score?
Engineering: That’s not likely to happen; it’ll be designed to 60601-1 standards.
Facilitator: Okay, but the Probability of Occurrence is a separate discussion. If it did occur, what is the severity of the harm?
Engineering: It’s just 120 AC, I’d say “Minor,” maybe “Moderate.”
Quality: But what if the user has a heart condition? Wouldn’t that be the worst case? And if he or she had a heart attack and no one was around to help them it could be serious—certainly hospitalization and maybe even death.
Facilitator: Susan [the Clinician], could a 120 AC shock cause cardiac arrest?
Clinician: I don’t know if it’s the most likely scenario, but it is possible.
Facilitator: Well, the standard does require that we consider “foreseeable sequences of events.”
Quality: I agree, I think death is the worst case so it should be “Catastrophic.” It’s best to be conservative.
Facilitator: Okay, does everyone agree?
All: Yes . . . Sure . . . OK . . .
Facilitator: Okay, the next one is where the device loses power during an operation. Since this is a life sustaining device, I think this type of failure would be quite severe. Do you agree that the effect could be “Catastrophic”?
Clinician: I agree. If this failure occurs during the procedure, the patient would be at great risk of serious injury or death if backup equipment isn’t available.
Facilitator: Are we agreed?
All: Yes . . . Yes . . . Yes . . .
Independent Team Member: It seems like we have a lot of hazards that are rated “Catastrophic,” but some seem much worse to me than others. Is this process really helpful?
Facilitator: This is how the process works. The risks will be different once we assign Probabilities of Occurrence. Don’t worry.
Independent Team Member: Okay, but if everything is going to end in “death,” I don’t see how we’re going to be able to prioritize mitigations.
Facilitator: Don’t worry. I think you’ll see how we do it once we get to that part of the process. Now for the next one: the user gets cut on a sharp corner of the device. I think we should consider the potential for infection, which if not treated promptly.
I think you see where this is going.
The hero of our story, the “Independent Team Member,” makes an important point: the objective of a risk analysis is to help the team make decisions about which hazards represent the greatest risks, and thus where risk reduction efforts should be focused. As a wise professor once said, “Information that doesn’t help you make a decision has no value.” If all of your risks fall into the high Severity region of your risk acceptability table (when, in fact, the harms and associated severities are quite different), the time and effort that your team spends in the conference room updating your FMEAs will not help you develop a safer device.
In addition, if you are in a situation where you have a novel device and you’re unable to estimate the probability of occurrence of harm, the Guidance (TIR24971) recommends that the risk evaluation be based on the Severity of Harm alone (section 5.5.3). If everything has a Severity of 5, you won’t be able to differentiate the risks.
So what do you do? Unfortunately, the Guidance is not always helpful when discussing this challenge. The discussion of Severity in section 5.5.4 is limited to a few points:
- While severity is, in reality, a continuum, the use of a discrete number of Severity levels simplifies the analysis.
- The manufacturer decides how many categories of Severity are needed and how they are to be defined.
- The category definitions should not include any element of probability.
- The Severity levels need to be chosen and justified by the manufacturer based on the harms that could result for a particular medical device.
In subsequent sections, the Guidance recommends that three to five levels are typically appropriate for both Severity and Probability of Occurrence (section 5.5.5) and offers the following example of qualitative Severity levels:
Negligible: Results in inconvenience or temporary discomfort
Minor: Results in temporary injury or impairment not requiring professional medical intervention
Serious/Major: Results in injury or impairment requiring professional medical intervention
Critical: Results in permanent impairment or irreversible[1] injury
Catastrophic/Fatal: Results in Patient Death
So far, this is all very helpful. But you can still run into the problem of creating a “sequence of events” for each Hazardous Situation that leads to death. The old version of the Standard did exactly that, with a table of five electromagnetic energy, chemical, biological, and functional Hazardous Situations that all ended in death. The new Guidance does a better job of addressing this point (the replacement table only has one of four rows ending in death – Table F.1).
If you’re creative enough, any hazard may lead to death – it’s just that some may require a longer “sequence of events” to get there. Once probabilities are factored into the risk calculation, you may see that the “high severity” events are extremely rare events, while the “low severity” events are more common. Whether that “extremely rare, high severity event” should be entered into the risk analysis at all is up to the judgement of your cross-functional team.
[1] Note: “Irreversible injury” was added in 2020. The old example used “life threatening injury”, which was always confusing: “life threatening” has an element of probability in it (i.e., it might cause death) and a Severity of 4, while “Results in Death” had a Severity of 5. The new example is clearer.
So how does your team resolve this problem? At MEDIcept, we recommend a three-step process when assessing severity:
- Do your Homework
- Define Your Use Scenario
- Focus on Direct Effects
Do Your Homework
One of our primary recommendations to clients with regard to conducting risk analyses is to do their homework before jumping into detailed analyses. By “homework” we mean conducting research into the type of hazards and failures (and associated harms) that can occur as a result of the use of your device. There are a variety of tools and approaches that your team can use to develop a strong understanding of likely hazards/failures/harms before beginning an analysis. A few of these include:
TIR 24971:2020 Annex A: The Annex A title, Identification of hazards and characteristics related to safety, pretty much says it all. This annex to the Guidance provides 37+ questions that manufacturers can use to begin to identify potential safety issues associated with their device. Many manufacturers include the answers to these questions in their Risk Management File. The answers can help you to identify where to focus your risk assessment efforts.
ISO 14971:2019: Table C.1 of the Standard, Examples of Hazards, provides a list of generic hazards, which can be a very helpful, top-down tool for thinking about potential hazards. The list is fairly redundant with the Annex A questions in the Guidance, but given the different perspective and wording, it can be helpful to consider both.
IEC 62366-1: 2015 Medical Devices – Application of Usability Engineering to Medical Devices: Like the Risk Management Standard, this standard provides some good tools to help identify potential safety hazards. In addition, this standard provides very useful tools to help you define the “use scenario” for your device and develop “user profiles,” which will help to establish the context within which your device will be used and, therefore, what types of hazards and harms could occur.
Complaints Data: Reviewing historical data is invaluable. If the subject device is a new member of a family of devices, complaint data on other “family members” is a great place to learn about the hazards/hazardous situations/harms that should be anticipated with your planned device. If the device technology is new to your company, the FDA’s MAUDE database will help you learn about the types of safety issues that other manufacturers and users are reporting for similar devices.
Field Research: There is only so much that you can learn about your device within the walls of your facility. Seeing how existing, similar devices are actually used in the environments where they are intended to be used will certainly open your eyes to issues that you otherwise may not have anticipated.
Define Your Use Scenario
When assessing the severity of harms that could occur during the use of your device, it is very important to clearly establish the “use scenarios” and “user profiles” being considered in your analysis. For example, if your device is indicated for use with the general population, your “user profile” will be defined quite broadly: e.g., from pediatric to elderly patients. The challenge is that the severity of harm resulting from a specific hazard may be quite different for a healthy adult when compared to an infant or elderly adult (e.g., a hazard that may result in moderate harm to an adult might be catastrophic to an infant or a patient that has an impairment that is not explicitly contra-indicated for your device).
So, how do you come up with a single Severity score for a particular harm when you have a potentially broad and complex range of users, patients, and use environments? There are a few basic approaches:
Worst-Case Scenario: Using this first approach, you define the “worst case” in which your device is intended to be used and base the whole analysis on that scenario. The benefit of this approach is that the team has a clear, consistent scenario to consider throughout the assessment. The downside includes the potential for conflicts when there is no single worst case (e.g., in some cases, the severity of a harm is greater for a child than an elderly adult, and in other cases it may be the reverse).
Typical Scenario with Exceptions: Using this approach, the analysis would be based on the User Profile and Use Scenario expected in most cases. Then, when the harm associated with a specific hazard varies significantly based on the user, patient, or environment, add a separate line in the analysis to address that “exceptional” situation. For example, small amounts of air in an I.V. line might cause headaches or illness in a healthy adult (a Minor/Moderate harm) but could be Catastrophic for an infant. When there are a limited number of “exceptional” situations, this approach can be very efficient. But you need to be careful: teams can get caught up in creating too many exceptions, thereby creating a great deal of redundancy and over-complicating the analysis.
Hazard/Failure-Specific Scenarios: At the most extreme end of the spectrum, a specific use scenario can be established for each hazard. If a team decides to follow this approach, they should work to limit themselves to just a few specific scenarios and identify which of these scenarios is being considered for each hazard. Otherwise, this approach can easily become overly burdensome.
The bottom line is that your team needs a shared understanding of the use scenario and user profile to effectively identify appropriate Harms and associated Severities. The approaches described above simply provide a few options for dealing with situations where a “typical user” approach is not appropriate.
With your homework complete and the use scenario(s) established, your team should have a good idea of the types of hazards and hazardous situations that are likely to occur and a good understanding of the type of user or patient who will be affected. The risk management standard is clear that a Hazard cannot cause harm on its own, and the previous discussion makes it clear that the harm may vary depending on the health and condition of the patient or user exposed to the hazard and hazardous situation. Figure 1, below, illustrates how these three elements, together, lead to characterization of the harm that could occur.
Figure 1: Interaction of Elements Leading to Harm
By clearly establishing the context in which the harm occurs, your team will be in a much stronger position when it works to assess the severity of that harm. In addition, when you return to the analysis a few months down the road (when assessing the impact of proposed design changes or conducting an annual review), you will have a much better sense of why particular harms received specific Severity scores.
Focus on Direct Effects
The last piece of the puzzle is to more clearly distinguish between harms that occur as a direct result of a hazard/hazardous situation and those that only occur following a series of subsequent events. To illustrate how these distinctions can be made, we have taken the five examples from Table E.1 in ISO 14971:2007 (Note: it’s the old version of the Standard, but the examples illustrate the point well). We’ve added more information regarding potential “Use Scenarios / User Profiles” and the associated harms – including both direct harms and harms that could occur if the direct harm is not adequately addressed (see Table 1).
This table provides one view of direct and subsequent harms that could result from the hazards/ hazardous situations described in Table E.1. We’re sure others may characterize these harms differently, but this example will help to illustrate the approach. The key point is that by separating direct harms from subsequent harms, your team will be better able to assign Harms and Severity scores that are appropriate to the situation and, as a result, help to establish meaningful priorities.
In the original version of Table E.1, the severity of each harm was arguably the same (death was one of the potential outcomes). In this revised table, we gain a better perspective on the context in which the harm occurs and a better sense of the immediacy of the harm. For example, on Line 4.0 the device is being used on a pediatric, diabetic patient at home. If the device fails, the patient may become hypoglycemic, resulting in coma and then, ultimately, death. On Line 5.0, we have an elderly patient at home suffering from arrhythmia that does not receive an expected, life-sustaining defibrillation shock. While both are high severity events, the immediacy of the harm in Line 5.0 suggests that it should receive a higher Severity score.
Some manufacturers integrate this “temporal” aspect of severity directly into their Severity criteria by using language along the lines of “Catastrophic = Imminent death.” Even with this additional language in the criteria, you are left with the decision of how to interpret “imminent.” For example, is there a difference in the Severity score when death is a “direct harm” as opposed to a “tertiary harm”? . . . Probably. Is there are difference in the Severity score when death is a “direct harm” as opposed to a “secondary harm”? . . . Maybe not.
To make effective use of this approach, a team could establish a procedure where, for example, any hazard/ hazardous situation that results in death as a direct or secondary harm is considered “Catastrophic (5),” while situations where death is less immediate would be considered “Critical (4).” Table 2 illustrates how such a rule could help to distinguish among our five example hazards/situations.
Ultimately, it is up to each manufacturer to establish risk analysis procedures and criteria that are appropriate to the nature of their device and the environment in which it is used. The value of the approaches described above is that they allow your team to assign Severity scores that make appropriate distinctions among the identified hazards. In addition, you’ll have a clear record on why certain scores were assigned.
By completing the homework, explicitly identifying the use environment and user profile, and assessing the immediacy of the anticipated harm, your team will have greater confidence that their analysis accurately represents the risks of their device.
Next Steps
In our next article, we will continue to explore specific challenges associated with components of the FMEA process by digging into the estimation of the probability of occurrence.
If you’ve missed any of our previous articles, you can look them up on here.
MEDIcept … Trusted Solutions, Rapid Response …
About Us
MEDIcept Inc. is an international consulting firm specializing in medical device, IVD, and biotechnology Regulatory, Quality, and Clinical Services. Since 1996, we have worked with thousands of companies to solve their most critical FDA and ISO issues. Our integrated solutions are rooted in our direct experience and span all stages of the product life.
MEDIcept is committed to providing our clients with what they need. We are committed to quality deliverables because we value our clients’ time and resources. This is why 90% of our clients come back to us again and again to solve new issues.
For additional information, please contact Susan Reilly at SReilly@MEDIcept.com.