NYU Langone took a page from other industries' playbooks to improve care. Here's how

Businessman video-chats with doctor on laptop
Iterative randomized testing is helping NYU Langone Health refine its patient-facing best practices. (Getty/AndreyPopov)

Sometimes, it's a simple tweak that makes a big difference.

At NYU Langone, a randomized quality improvement test revealed a small change to the wording of a call script would generate much better results—potentially adding 392 appointments to the schedule per year for annual billings of over $30,000.

It's just one example of how the health system has begun using the nimble, low-cost approach known in other industries as A/B testing. Since 2018, they've applied it to improve everything from tobacco counseling to health worker interventions in the emergency department, according to a recent article published in The New England Journal of Medicine

While the process is nearly ubiquitous among tech companies, healthcare has been slow to adopt it, according to study author Leora Horwitz, M.D., associate professor of population health and medicine and director of the Center for Healthcare Innovation and Delivery Science.

“I don't know why it's taken so long,” she told FierceHealthcare. "We've done a clinical investigation—where we're trying to sort out, 'Is one drug better than another drug?'—for decades and have been really good at it. And we kind of assume once we figure out which thing is better that everybody just gets the right treatment. We've been slow to really understand the importance of healthcare delivery, the process of healthcare.

"And hospital administrators, who put these programs into place, are not researchers so they're not thinking about analysis in the same way that researchers do," she said.

RELATED: HHS awards $107M to support quality improvement in 1,200-plus health centers

How they do it

A/B testing is a randomized experiment comparing two variables against each other to determine which performs better. For instance, in technology, one of two versions of a webpage might be shown to random visitors of the page, and data is analyzed to see which performed better. 

In the first year of the program at NYU Langone, one project found that changes to their outreach telephone scripts could make the calls shorter and increase appointment rates for annual exams. Another showed that changing the text of a provider-targeted prompt about giving tobacco counseling in an office generated a higher rate of prescriptions.

Some projects made an even bigger impact. “We randomized our community health worker intervention in the emergency department, which is a big program, and expensive and resource-intensive,” Horwitz recalls. Since the program operates beyond its capacity, targeting interventions more efficiently represents a high-profile project with potential for substantial positive changes.

The lightweight, quick nature of the testing means the hospital not only can identify processes that don’t work but can make changes and keep testing until they become effective—or even reallocate resources more efficiently. Introducing randomization also keeps the analysis from getting bogged down in the statistical machinations required to make retrospective reviews work.

“Often, part of my group’s job is to help evaluate interventions that have been going on for a while, and it’s intensely frustrating because there are huge biases in the way people get into interventions and data are collected,” Horwitz said. “Now, we set it up from the get-go that we randomize who we approach and I have confidence in the data.”

RELATED: These 6 healthcare leaders say quality improvement is an organization-wide effort and a cultural imperative

Since these tests require the involvement of frontline staff, NYU Langone designs the studies around data they already capture routinely. That ease of implementation helps with staff buy-in, but Horwitz believes it’s even more important to establish a high level of transparency and trust.

“You’re working with people to assess their own performance and to see if what they are doing every day is working, so you have to build a lot of trust to start with because you might find that what they’re doing all day isn’t helpful. They have to set it up with the attitude that if we find out that this doesn’t work, that’s great. Then we have a chance to make it better,” she said.

Horwitz cautions that as useful as randomization is for quality improvement work, human-subject research is a different ballgame. As more hospitals turn to randomized testing, however, Horwitz said more clarity about where the ethical line between research and quality improvement lies would be a huge help for organizations figuring out how to navigate that territory.

“All we’re doing is trying to get patients more often to get evidence-based care—we’re not testing whether a new treatment is better than another. We’re saying, we already know washing your hands is a good idea, so how can we get people to do that more?” she said.