New guide helps assess hospital scorecards

A set of 25 guiding principles can now help organizations assess hospital rankings and scorecards.

The Association of American Medical Colleges (AAMC) issued the list in response to an increase in the number of institutions that issue hospital quality reports. The scorecards often use different formulas and methodologies and produce disparate, contradictory results.This variability "points to concerns about validity and reliability among the measures used by these groups," according to the document.

An expert panel developed the 25 principles, which fall into three overarching categories:

Purpose: Since various organizations rank hospitals from a variety of perspectives, such as improvement, purchaser decision-making and consumer education, they should clearly state their target audience and intended purpose, according to AAMC. "Stakeholders may have differing opinions on how well the measures and methodology meet the intended purpose, however, a discussion on divergent viewpoints cannot occur if the purpose is not well defined," the document states.

Transparency: Readers must have all information necessary to understand an organization's data, according to AAMC, including data collection methods, data sources and measure specifications. "Details should be sufficient for independent replication of the results," the document states. "Limitations in the data collection and methodology and relevant financial interests also should be disclosed."

Validity: Methodology, data collection, benchmarks and scoring must be as accurate and reflective of the measured characteristic as possible, according to the document. Organizations must field test measures and methodology and receive endorsement from the National Quality Forum whenever possible. "Validity is necessary to ensure that results are accurate and that providers are appropriately characterized," according to AAMC.

Critics argue that there are flaws in the methodology of many ranking organizations. For example, U.S. News & World Report, which critics accuse of overemphasizing hospital reputations in its rankings, will give increased weight to patient safety in its new formula. And several hospitals questioned the Leapfrog Group's methodology after they dropped in the rankings simply for failing to fill out the group's survey, FierceHealthcare previously reported.

To learn more:
- read the AAMC principles (.pdf)