Better late than never goes the old adage, which is fitting for the authors of a just-published article in the American Journal of Preventive Medicine which concludes that "rigorous research is needed to examine the potential, as well as the challenges, of utilizing mobile technologies to improve health outcomes." The article is based on the results of an August 16, 2011 mHealth Evidence Workshop at the National Institutes of Health (NIH), which gathered together 50 researchers, policymakers, government and regulatory officials from around the world.
FierceMobileHealthcare spoke with Robert M. Kaplan (pictured left), Ph.D., director of NIH's Office of Behavioral and Social Sciences Research (OBSSR), and Wendy Nilsen, Ph.D., a health scientist administrator at OBSSR, who co-authored the AJPM article. Dr. Nilsen's work in multiple trans-NIH mHealth initiatives include: leading the development of the NIH mHealth Public-Private Partnership, co-leading the NIH mPower mHealth group, convening meetings to address methodology and barriers to the utilization of mobile technology in research; serving on numerous federal mHealth initiatives; and leading the mHealth training institutes.
FMH: NIH, in partnership with NSF, Robert Wood Johnson Foundation and McKesson, gathered 50 researchers, policymakers, government and regulatory officials from around the world at NIH's mHealth Evidence Workshop. What were the results as reflected in your article recently published in the American Journal of Preventive Medicine?
Nilsen: We brought this meeting together because we were looking at all the excitement around mobile and we were seeing a real lack of science. And, so, we were hearing from the science community concerns that some of the hype around this was really outstripping the science. We brought everybody together to ask the questions: is there a way to speed up the science so that mobile health can really have an evidence base? Many of the participants at the meeting actually were authors on the paper so this has been an iterative process, which is one of the reasons it took awhile. As the science was changing, we were adapting and growing the paper. It all worked around what the central tenets of the meeting were. 
FMH: Despite the rapid growth and proliferation of mHealth over the past decade, systematic research on the impact of new mobile technologies on health outcomes remains scarce, according to an article in the Journal of Medical Internet Research. How can we address the sparse evidence base for the efficacy of mHealth while keeping pace with the rapid growth of technology?
Nilsen (pictured right): One of the things we were trying to get at in this paper was that there are methods that are not being used consistently and that there really isn't a culture of evaluation in mobile health. We have people trying things that they think make sense. We're often bringing in people from industry, from engineering and from computer science that have very different ways of evaluating. It's how many you sell, does it work, can you turn it on, is it transmitting? That's not the criteria we use in health. So, one of the questions we really looked at was whether some of the methods of evaluation were reasonable and appropriate ways to evaluate and bring them into the mobile health sphere?
Kaplan: I think one of the big issues in this area is that research takes different forms and uses different standards. The one thing that has struck me about this field is that people are doing a lot of work and publishing a lot of papers, but they might come from research disciplines that haven't evolved the same sorts of standards that we see in medicine and healthcare. So, for example, if we look at randomized clinical trials in medicine, we have evolved fairly high standards for the reporting of those trials now. Drugs and devices need to register in advance in clinical trials.gov and there are specific standards for randomization is done and pre-declarations of what the outcome variables ought to be. It actually makes it very hard to find significant effects. I think these difficult standards aren't appreciated in a lot of other areas of science quite yet.
FMH: A recent review of randomized mHealth trials found many studies to be of poor quality and few with low risk of bias and very few with clinically significant benefits for the interventions. As Dr. Francis Collins, NIH director, pointed out at the 2012 mHealth Summit, only about 20 randomized clinical trials involving mHealth tools or services have been conducted in the U.S. since 2008 under NIH auspices. And, as Collins revealed, more than half of them have failed to document clear evidence of improved outcomes. How can we best establish the reliability and validity of mHealth assessment methods?
Kaplan: When we looked at all these trials we only focused on 20 of them because only 20 of them that we encountered met the standard of randomized trials. There's a certain amount of rigor that we expect in the biomedical sciences that, again, I don't think is well appreciated in some of the other sciences. So, Wendy and some others have been trying to pull this together with training institutes for young scholars to learn about how to merge these different ways of thinking about things.
FMH: An International Journal of Medical Informatics article recently looked at 215 mHealth studies of which 81.8 percent used a classical randomized trial design. Should randomized clinical trials remain the gold standard for evaluating mHealth interventions?
Kaplan: If a randomized trial is powered properly, you get an answer to the question. In randomized clinical trials in clinical medicine, no results are actually pretty common. In fact, we've been learning recently that they're perhaps more common than positive results. And, those results often lead to something that's good for patients.
Nilsen: We can randomize in ways that we haven't randomized in the traditional clinical trials. There are some new methods. There are some adaptive trials. There are some optimization methods we can use. We actually think that the technology will improve the way we evaluate this. We can help people adhere to the trials. We can measure more frequently. And, the measurement issue I think is huge because if we can measure more frequently we may be able to move the end points into a shorter time span. One of the calls to action in the paper is for people to work on methods in this area.
FMH: A resource is being developed by the Center for Communication Programs at Johns Hopkins University's Bloomberg School of Public Health. It's an online mHealth evidence database that aims to catalog, categorize and grade all of the known peer-reviewed and grey literature on mHealth in high-, middle- and low-income countries. Are you familiar with this project and do you think it will help?
Nilsen: I'm familiar with it. I think trying to catalog the evidence is good. Unfortunately, as you well know, this is an exploding area with things happening all of the time, so cataloging them is difficult and as soon as you get done with one area there's other things piping up. I think it's also important to remember, as Dr. Kaplan said, things like clinicaltrials.gov and having people register their trials in advance which keeps them targeted on an outcome.
Kaplan: If you think about how are we going to move the field of mHealth forward and get it the appropriate attention, I think the way that is achieved is by getting a few really good publications in key medical journals. A really good study published in the New England Journal of Medicine, even one or two, has the potential to get the attention of the medical establishment and perhaps the payers as well. The way to do it is in trusted sources and in forums that people recognize as legitimate peer-reviewed outlets.
FMH: Is it fair to say that there's a lack of evidence in mHealth?
Nilsen: Yes, I think that is a very fair statement. It doesn't mean that mHealth doesn't have a huge potential or that things aren't changing in a good way. It just means that there are many ongoing trials. There's more literature coming. NIH is funding more every year. But, right now, I don't think the literature is there.
Kaplan: When we here at NIH say "evidence" we have something very specific in mind, because we are confronted with colleagues who are very picky in what they call evidence. And, so, the standard that they are applying is maybe quite different than the standard that is being applied on the commercial side. This is part of the cultural divide that we need to attend to in order to bring the different sectors together so we can develop a common language.
Editor's note: This interview has been edited for length and clarity.