A new study calls into question whether the Leapfrog Group's Hospital Safety Grade system should continue to use self-reported data.
The research, conducted by a team at the University of Michigan, looked at 2013 hospital data and found that most facilities that participate in Leapfrog’s Hospital Survey say they comply with most, if not all, of the group’s Safe Practices. However, hospitals that self-reported high compliance did not necessarily have lower hospital-acquired infection rates or lower federal penalties for readmissions.
“The Safe Practices part of the Hospital Safety Grade is based on whether a hospital self-reports that it has adopted certain protocols, and not whether those protocols are actually followed,” Jennifer Meddings, M.D., the study’s senior author and an assistant professor of internal medicine at the UM Medical School, said in an announcement.
One example, Meddings said, is that hospitals self-report what hand-hygiene protocols they have in place, but not how often clinicians are actually washing their hands. In Leapfrog’s analysis, the Safe Practices score is based on a comparison with the average of participating hospitals, so there is a significant risk to the overall grade associated with a poor self-reported score, according to the study.
The researchers concluded that ratings systems should put more emphasis on safety measures that are reported by objective sources. “It’s time to have a new discussion about how to measure new metrics and use them in innovative ways to evaluate hospitals,” Meddings said in the announcement.
The Leapfrog Group said in a statement to FierceHealthcare that providers that try to “game” the organization’s rating system are “generally frustrated” by the effort. The group’s ratings are made of up of 30 factors, which is hard to statistically predict, and hospitals are graded on a curve, so comparisons would not be accurate until all hospitals are scored.
Leapfrog also “goes to extreme lengths” to confirm self-reported data, it said in the statement. Each survey response is verified independently, it said, and documentation for Safe Practices responses is required from a random group of responding hospitals.
“We believe that hospitals that adhere to the Safe Practices are on the right path toward better outcomes. There are many other steps along this path, so we have never assumed they correlate 100% with safety outcomes,” Leapfrog said. “Indeed, that is why we consider both processes and outcomes in scoring the Leapfrog Hospital Safety Grade.”