Section 1115 waivers in the Medicaid program are meant to be experiments for states to test coverage approaches that don't meet federal requirements.
However, most of those evaluations fall far short, lacking the rigorous study design needed to properly evaluate them, according to a new commentary in the New England Journal of Medicine.
The evaluations usually use before-and-after comparisons and difference-in-difference analyses, both of which contain numerous methodological limitations. Moreover, the Government Accountability Office has found these evaluations often contain small sample sizes and no comparison groups. They also fail to test certain hypotheses and report all outcomes.
But demonstration projects are highly consequential and evaluating them rigorously is a “legal, ethical, and practical imperative," the authors wrote. Instead, they should use randomized controlled trials, the gold standard in study design.
Medicaid demonstration waivers have been a major source of controversy since the Centers for Medicare and Medicaid Services (CMS) began allowing states to pursue work requirement demonstrations in January. Opponents said work requirements will force many beneficiaries out of the program, effectively blocking their access to health services and harming their health.
A proper evaluation should examine how a waiver affects beneficiaries' health, according to Kristen Underhill, J.D., an associate professor of law at Columbia University and lead author of the article. Those evaluations are particularly important with new work requirements in Arkansas, and now Kentucky, following a recent reapproval from CMS.
“Evaluations can (and should) generate an important and useful body of evidence about how waiver terms affect population health,” Underhill told FierceHealthcare via e-mail.
There are financial implications to this problem, too. More than a third of Medicaid dollars are spent on these Medicaid demonstrations, so it's critical that states and CMS can rely on their evaluations.
RELATED: CMS reapproves Medicaid work requirements in Kentucky
Currently, demonstration projects between states can be evaluated with a randomized design, but one particular demonstration shouldn’t need to count on similar states pursuing—and being approved for—similar ones. Using a randomized design for a given demonstration would eliminate that issue.
Some may argue randomized trials are too challenging to design and implement—that states lack the administrative capacity to do so, there are too many methodologic choices involved, and there is not sufficient uncertainty around how the waiver will affect beneficiaries for such a design.
But addressing these issues is not as daunting as it seems, the authors argue.
States already apply different benefits to different populations (e.g. pregnant women aren’t subject to work requirements), methodological issues “are surmountable with evaluation expertise,” and subjecting all beneficiaries to an untested policy is no better than subjecting controlled groups, they wrote.