Inter-rater reliability (IRR) is the process by which we determine how reliable a Core Measures or Registry abstractor’s data entry is. It is a score of how much consensus exists in ratings and the level of agreement among raters, observers, coders, or examiners.
By reabstracting a sample of the same charts to determine accuracy, we can project that information to the total cases abstracted and thus gauge the abstractor’s knowledge of the specifications. We perform IRR often due to the dynamic aspect of measures and their specifications.
IRR assessments are performed on a sample of abstracted cases to measure the degree of agreement among reviewers.
Agreement can be expressed in the form of a score, most commonly Data Element Agreement Rates (DEAR) and Category Assignment Agreement Rates (CAAR), which are recommended by The Joint Commission and Centers for Medicare and Medicaid for evaluating data reliability and validity.
The Data Element Agreement Rate, or DEAR, is a one-to-one comparison of consensus between the original abstractor and the re-abstractor’s findings at the data element level, including all clinical and demographic elements.
To calculate the DEAR for each data element:
DEAR results should be used to identify data element mismatches and pinpoint education opportunities for abstractors.
It is also important to analyze the DEAR results for trends among mismatches (within a specific data element or for a particular abstractor) to determine if a more focused review is needed to ensure accuracy across all potentially affected charts.
Get More Info on Outsourcing Data Abstraction
(using a partial AMI data element list)
DEAR
*n/a in the table above represents fields disabled due to skip logic.
Related: Top 3 Reasons Quality-Leading Hospitals are Outsourcing Data Abstraction
The Category Assignment Agreement Rate, or CAAR, is the score utilized in the CMS Validation Process which affects Annual Payment Update.
CAAR is a one-to-one comparison of agreement between the original abstractor and the re-abstractor’s record-level results using Measure Category Assignments.
MCAs are algorithm outcomes that determine numerator, denominator and exclusion status and are typically expressed as A, B, C, D, E.
In other words, the same numerator and denominator values reported by the original abstractor should be obtained by the second abstractor.
To calculate the CAAR, count the number of times the original abstractor and re-abstractor arrived at the same MCA; then, divide by the total number of paired MCAs.
Again, convert to a percentage for evaluation purposes.
A score of 75% is considered acceptable by CMS, while TJC prefers 85% or above.
CAAR results should be used to identify the overall impact of data element mismatches on the measure outcomes.
CAAR mismatches can then be reviewed in conjunction with associated DEAR mismatches to foster abstractor knowledge.
Remember, CAAR results are also the best predictor of CMS validation results.
Overall CAAR
Incorporating Inter-Rater Reliability into your routine can reduce data abstraction errors by identifying the need for abstractor education or re-education and give you confidence that your data is not only valid, but reliable.
When to Use Inter-Rater Reliability
While conducting IRR in house is a good practice, it is not always 100% accurate.
Often abstractors correct for physician documentation idiosyncrasies or misinterpret Core Measures guidelines.
American Data Network can provide an unbiased eye to help you ensure your abstractions are accurate. Click here for a free quote!
About American Data Network Core Measures Data Abstraction Service
Core Measures and Registry Data Abstraction Service can help your hospital meet the data collection and reporting requirements of The Joint Commission and Centers for Medicare & Medicaid Services. We will work directly with your facility to provide a solution that fits your needs – whether it’s on site, off site, on call, or partial outsourcing. Plus, it is not necessary to use ADN’s data collection tool; our experienced abstraction specialists will work with whatever Core Measures vendor you use.
As a vendor since the inception of Core Measures, ADN has developed a keen understanding of the measure specifications, transmission processes, and improvement initiatives associated with data collection and analytics. In addition, ADN can train your abstractors on the changes to the measure guidelines and conduct follow-up Inter Rater Reliability assessments to ensure their understanding.
Our data abstraction services allow your hospital to reallocate scarce clinical resources to performance improvement, utilization review and case management. Or, use ADN personnel to complement your existing data abstraction staff to provide coverage for employees on temporary leave or to serve as a safety net for abstractor shortages or unplanned employee departures.
