Inter-Rater Reliability: What It Is, How to Do It, and Why Your Hospital’s Bottom Line Is at Risk Without It
Inter-Rater Reliability (IRR) assessments can help your facility evaluate the accuracy and integrity of your data abstractions. IRR assessments are performed on a sample of abstracted cases to measure the degree of agreement among reviewers.
⏰ 8 min read
Published on August 26, 2019
What is Data Abstraction Inter Rater Reliability (IRR)?
Inter-rater reliability (IRR) is the process by which we determine how reliable a Core Measures or Registry abstractor’s data entry is. It is a score of how much consensus exists in ratings and the level of agreement among raters, observers, coders, or examiners.
By reabstracting a sample of the same charts to determine accuracy, we can project that information to the total cases abstracted and thus gauge the abstractor’s knowledge of the specifications. We perform IRR often due to the dynamic aspect of measures and their specifications.
Inter-Rater Reliability (IRR)
IRR assessments are performed on a sample of abstracted cases to measure the degree of agreement among reviewers.
Agreement can be expressed in the form of a score, most commonly Data Element Agreement Rates (DEAR) and Category Assignment Agreement Rates (CAAR), which are recommended by The Joint Commission and Centers for Medicare and Medicaid for evaluating data reliability and validity.
Data Element Agreement Rate (DEAR)
The Data Element Agreement Rate, or DEAR, is a one-to-one comparison of consensus between the original abstractor and the re-abstractor’s findings at the data element level, including all clinical and demographic elements.
To calculate the DEAR for each data element:
Count the number of times the original abstractor and re-abstractor agreed on the data element value across all paired records.
Divide by the total number of paired records.
Convert to a percentage and evaluate the score. DEARs of 80% of better are acceptable.
DEAR results should be used to identify data element mismatches and pinpoint education opportunities for abstractors.
It is also important to analyze the DEAR results for trends among mismatches (within a specific data element or for a particular abstractor) to determine if a more focused review is needed to ensure accuracy across all potentially affected charts.
The Category Assignment Agreement Rate, or CAAR, is the score utilized in the CMS Validation Process which affects Annual Payment Update.
CAAR is a one-to-one comparison of agreement between the original abstractor and the re-abstractor’s record-level results using Measure Category Assignments.
MCAs are algorithm outcomes that determine numerator, denominator and exclusion status and are typically expressed as A, B, C, D, E.
In other words, the same numerator and denominator values reported by the original abstractor should be obtained by the second abstractor.
How to Calculate Your CAAR
To calculate the CAAR, count the number of times the original abstractor and re-abstractor arrived at the same MCA; then, divide by the total number of paired MCAs.
Again, convert to a percentage for evaluation purposes.
A score of 75% is considered acceptable by CMS, while TJC prefers 85% or above.
CAAR results should be used to identify the overall impact of data element mismatches on the measure outcomes.
CAAR mismatches can then be reviewed in conjunction with associated DEAR mismatches to foster abstractor knowledge.
Remember, CAAR results are also the best predictor of CMS validation results.
Add Total Paired MCAs (Denominator) (21+9+9+27) = 66
Divide Numerator by Denominator (61/66) = 92.42%
When to Use Inter-Rater Reliability
Incorporating Inter-Rater Reliability into your routine can reduce data abstraction errors by identifying the need for abstractor education or re-education and give you confidence that your data is not only valid, but reliable.
When to Use Inter-Rater Reliability
After a specifications manual update.
For new abstractors.
When new measure sets are introduced.
During performance evaluations.
Rules of Engagement
The IRR sample should be randomly selected from each population using the entire list of cases, not just those with measure failures.
Each case should be independently re-abstracted by someone other than the original abstractor.
The IRR abstractor then inputs and compares the answer values for each Data Element and the Measure Category Assignments to identify any mismatches.
The results are reviewed/discussed with the original abstractor and case is updated with all necessary corrections prior to submission deadlines.
If the original and IRR abstractor are unable to reach consensus, we recommend submitting questions to QualityNet for clarification.
Lessons learned from mismatches should be applied to all future abstractions.
Results should be analyzed for patterns of mismatches to identify the need for additional IRR Reviews and/or targeted education for staff.
Core Measures and Registry Data Abstraction Service can help your hospital meet the data collection and reporting requirements of The Joint Commission and Centers for Medicare & Medicaid Services. We will work directly with your facility to provide a solution that fits your needs – whether it’s on site, off site, on call, or partial outsourcing. Plus, it is not necessary to use ADN’s data collection tool; our experienced abstraction specialists will work with whatever Core Measures vendor you use.
As a vendor since the inception of Core Measures, ADN has developed a keen understanding of the measure specifications, transmission processes, and improvement initiatives associated with data collection and analytics. In addition, ADN can train your abstractors on the changes to the measure guidelines and conduct follow-up Inter Rater Reliability assessments to ensure their understanding.
Our data abstraction services allow your hospital to reallocate scarce clinical resources to performance improvement, utilization review and case management. Or, use ADN personnel to complement your existing data abstraction staff to provide coverage for employees on temporary leave or to serve as a safety net for abstractor shortages or unplanned employee departures.