Patient Safety Event Reporting, also commonly referred to as incident reporting or occurrence reporting, is the system and process in which frontline staff document events that cause or have the potential to cause patient harm, allowing for analysis that can lead to improved care. Event reporting systems include third-party vendor applications, EHR-embedded tools, in-house built electronic tools, and manual paper-based processes.
In 2018, American Data Network Patient Safety Organization (ADNPSO) conducted a deep-dive analysis of the event type “Other.” Over a four-year period, this catchall category accounted for over 40% of events reported by ADNPSO members. Among the myriad of striking revelations, the death rate of incidents categorized as Other was 3 times higher than ALL remaining event categories combined. The results of the Other analysis proved so compelling that ADNPSO was selected to present at the National Association for Healthcare Quality (NAHQ) Next Conference in November 2018. The findings underscore the valuable insight buried in the Other event category.
As part of its mission, ADNPSO conducts deep-dive analyses to continuously improve its understanding of the data and disseminate best practices to members.
Among patient safety events submitted through ADN’s Patient Safety Event Reporting Application, the event type “Other” continuously ranks as one of the most frequently reported. With aggregate data from April 2014 through May 2018, ADNPSO determined that Other accounted for over 40% of all the submitted patient safety events, a trend other PSOs around the country are also finding.
Other tends to serve as a catchall category for events that do not fit neatly into the types standardized by the Agency for Healthcare Research and Quality’s (AHRQ) Common Formats. Moreover, because there is no nationally-defined or uniform collection for Other events, the data captured is largely narrative, which makes it more difficult to analyze and may result in missed opportunities to mitigate risk and harm.
In the project, ADNPSO set out to further refine the management process for Other events to help members better identify and prioritize emerging subcategories, and to analyze associated vulnerabilities, improvement opportunities and sustainability efforts.
The Need for the Project
Patient safety incidents, near misses and unsafe conditions occur constantly throughout healthcare organizations, whether or not facilities know about all—or even most—of them. The Iceberg model (used for guiding systemic thinking in healthcare for years)1 is a strong visual to illustrate that what you see above the waterline is only the tip of the problem (in this case, patient safety incidents). However, what lies beneath the surface provides a much deeper understanding of the problem. For example, near misses and unsafe conditions can reveal fractured systems and processes leading to errors. Focusing primarily on incidents forces staff into a reactive posture, rushing to diagnose and fix what has already happened. Many organizations are striving for a more proactive focus by concentrating on near misses and unsafe conditions. Patterns and trends, what influences them, and assumptions about the systems in place are all important drivers in regards to patient safety.
ADNPSO flipped the Iceberg model upside down and created the Whirlwind Model2 in order to operationalize it. The Whirlwind Model illustrates the key functions, parameters, and compliance efforts that must be coordinated, collaborated, and communicated in order to deliver safe patient care.
There is no straight line from admission to discharge as patients are impacted by all the blowing whirlwinds; i.e., interactions with multiple healthcare professionals and navigation through thousands of process steps that create fertile ground for defects and errors. Organizations must embrace the complexity and execute well-designed, targeted and coordinated care to mitigate patient risk and harm. A first step is appropriately categorizing patient safety data and delving below the surface to reveal fractured systems and processes.
How to Categorize Patient Safety Data
AHRQ Common Formats, created to standardize patient safety data collection, capture 3 top-level categories: incidents, near misses, and unsafe conditions. An incident is a safety event that reached the patient, regardless of harm. A near miss is an event that did not reach the patient. And an unsafe condition is a circumstance that increases the probability of a safety event.
AHRQ’s Common Formats define 8 standard event types in addition to Other:
Blood or Blood Product
Medication and Other Substance
Device or Medical/Surgical Supply, including Health Information Technology
Healthcare Associated Infection
Surgery or Anesthesia
ADNPSO’s deep-dive analysis of the 4-year period of data began with the large concentration of events in the Other category.
As the Other event type accounts for 43.56% of overall events, it’s critical to understand the data and to aggregate similar events wherever possible for learning purposes. ADN’s event reporting application also captures data about preventability. Strikingly, the analysis revealed that 62.12% of the Other events almost certainly or likely could have been prevented.
One of the most noteworthy findings from the ADNPSO analysis was that the mortality rate of incidents categorized as Other was 3 times higher than ALL remaining event types combined.
In 2014, ADNPSO conducted an extensive analysis of the catchall Other event type, identifying reporting trends and creating resultant subcategories in ADN’s event reporting application as a means for further segmentation. The list of 33 subcategories allows providers to better isolate and understand the types of events that do not fit neatly within the AHRQ Common Formats.
In reviewing the subcategory utilization between April 2014 to May 2018, Other Other accounted for 20.56% of the events followed by Specimens representing 16.03%. Using this to drive data collection priorities, ADNPSO carved Specimens out of the subcategory listing and designated it as an independent event type.
Of equal importance are the subcategories that do not have many events but should have higher volumes. It raises questions about the reporting culture or the staff’s understanding of what constitutes a safety event. This analysis can lead to the revelation of the need for more education around the definitions of the subcategories so frontline staff doesn’t default to the path of least resistance and use the Other Other subcategory.
A good example of how frontline staff’s perception of event categorization can affect your data is the subcategory Telemetry, which only accounted for 2.52% of all subcategories. Does staff not recognize Telemetry issues as a patient safety event? Or are those being reported but buried in the Other Other subcategory?
For Patient Safety and Risk Managers, it’s common to go after the easy targets of events that are High Volume. But Low Volume/High Risk should also be a regular target of focus.
Death and the Pareto Principle (80/20 Rule)
When assessing areas that need improvement, a quality team’s efforts can be focused to maximize impact by using the Pareto Principle (also known as the 80/20 rule), which states that roughly 80% of effects result from 20% of the causes.
A notable application of this rule can be seen in analyzing events where the Final Harm Level is Death. Among all incidents reported to ADNPSO between April 2014 and May 2018, 83.05% of all deaths occurred in 3 event types, specifically Other, Healthcare Associated Infections, and Surgery or Anesthesia.
One of the most remarkable discoveries was revealed when comparing the Final Harm of Other incidents to that of the remaining 8 event types combined. Death among Other events was almost TRIPLE.
With that in mind, ADNPSO used the Pareto Rule to dig deeper and concluded that 81.81% of Other deaths were attributed to just 3 of the subcategories, specifically Code/Emergency, Unexpected Death, and Other Other.
Focusing on just 21.42% (3 of the 14) subcategories with deaths will likely deliver the highest return, which is always a best practice when deciding where to focus time and energy. Identifying areas for improvement using the Pareto Principle can often bring high ROI with just small changes.
To further dissect Other deaths, ADNPSO looked at these 3 subcategories by department, discovering that just five department types accounted for 84.45% of deaths. Evaluating death by time of day at the department levels is recommended to ascertain if a pattern emerges or to determine what unit-specific factors correlate with the times of death.
Validating the Trends
To validate the four-year findings, ADNPSO evaluated data for CY 2018. The goal was to confirm whether the same trends are reflected in the most recent data set and to expose any new insights. The team ran aggregate reports for CY 2018 and also selected 500 random Other Other events for manual, narrative reviews.
The comparison of the two time frames revealed that Other events remained the most frequently reported type in CY 2018, followed by Medication, Falls, and Surgery/Anesthesia. The subcategory analysis revealed no significant changes from the analysis of the first data set. The highest reported subcategories were Other, Specimens, Delay in Treatment and Failure to Follow Policy/Order.
Also, over 50% of incidents continue to be reported as preventable. While the majority of events did not result in harm, the trend of Other events having a 3X higher death rate than all other events held true.
Miscategorization of Events as Other Other
Of the 500 random Other Other event narratives reviewed, ADNPSO recommended that 70% (or 346) should be recategorized to one of the more specific subcategories or event types. If these 346 events were recategorized as recommended, a shift would occur as to which patient safety subcategories top the list.
If your team is using the 80/20 Rule to prioritize time and resources, events that are miscategorized and hiding in Other may be skewing your data, meaning you may not actually be focusing on the top priorities. Risk Managers should review and appropriately categorize events prior to closing them to produce a more accurate set of priorities. ADNPSO also recommends using miscategorizations as an opportunity to educate frontline staff and curb future errors.
Strikingly, nearly all of the Other events that resulted in death were miscategorized. Of the 500 random Other Other events reviewed, 7 resulted in death; and 6 of the 7 warranted recategorization to more specific subcategories such as Delay in Treatment, Code/Emergency or Medications.
Key Takeaways and Suggested Action Items
The goal of ADNPSO’s Other event analysis project was to determine what type of patient safety issues were captured in this catchall category, then use our findings to aggregate similar events for learning purposes, improve future data collection efforts and isolate key issues that should be evaluated at individual organizations.
Perhaps the biggest takeaway from this deep dive is the incredible potential hiding in the event type Other, especially considering these could have a mortality rate 3X higher than all remaining event types combined.
Patient Safety and Risk Managers should make time to review and recategorize Other events before they are closed and used in analytics reports to ensure your team is focusing on an accurate set of priorities.
Organizations may be missing analytic opportunities for spotting key trends in patient safety data if using paper processes, EHR add-on modules or in-house built databases to collect events. Not to mention, your team is likely spending significant time to manually enter events and build meaningful reports; time that could be reallocated to analysis and improvement efforts. Get in touch with our team if you could benefit from the type of systems thinking and analytics seen in this project and would like more information about ADN’s event reporting application.
Suggested Action Items
Stratify your organizational event data by Incidents, Near Misses and Unsafe Conditions and compare to the Iceberg and Whirlwind models.
Run an Event Distribution on your organizational data to see if Other ranks among the most reported event types. Apply the Pareto Rule to prioritize efforts.
Perform deep dives into the catchall Other category, and aggregate like events into subcategories to prioritize emerging topics. Evaluate for miscategorizations and mortalities.
Evaluate Final Harm Distribution for Other events and consider comparing to the Final Harm for all remaining event types combined.
Charter multidisciplinary teams with subject matter experts to address the diverse safety issues submitted within Other.
Communicate deep dive findings and resulting team recommendations to all staff, via multiple avenues.
Utilize improvement tools, such as Rapid Cycle PDSA (Plan-Do-Study-Act) to drive change.
References and Resources
“Whirlwind Model” created by ADNPSO, copyright pending