Experts say IBM Watson’s flaws are rooted in data collection and interoperability

IBM Watson
IBM Watson promised to revolutionize cancer care. So far, experts say it's fallen well short.

Despite being backed by a significant budget and the marketing power of a major technology company, IBM Watson appears to be falling short of expectations when it comes to revolutionizing cancer care.

The shortcomings of IBM’s premier artificial intelligence system—made famous by its appearance on Jeopardy in 2011 and later co-pted to provide support for oncologists—are linked to a number of factors, according to an in-depth investigation by Stat that included interviews with doctors and artificial intelligence experts from around the world.

It’s not the first time health experts have questioned the impact of Watson. Last year, researchers pointed out Watson’s high costs weren’t meeting expectations. Earlier this year, venture capitalist Chamath Palihapitiya called Watson “a joke” and said the company is leaning on marketing prowess rather than its software. Palihapitiya later told CNBC he "should have been more careful" with his words, but maintained smaller AI companies are getting "somewhat outmarketed" by larger companies like IBM.

RELATED: Plenty of buzz for AI in healthcare, but are any systems actually using it?

But two key failures of the system, pitched by IBM’s marketers as a tool that can generate new insights and approaches to cancer care, are some of the same problems that currently plague most health IT systems: interoperability and data collection.

A specific issue that came up during IBM Watson’s partnership with MD Anderson Cancer Center—one of the first providers to partner with IBM—was navigating the incongruent data points within medical records. Some noted that inputting data could be time-consuming and onerous for a relatively small payoff in which Watson’s recommendation aligned with the physician. MD Anderson backed out of its partnership with Watson earlier this year.

“Teaching a machine to read a record is a lot harder than anyone thought,” Lynda Chin, M.D., who previously oversaw the Watson partnership at MD Anderson before taking a position at the University of Texas System in 2015, told Stat.

RELATED: Stanford researchers: Bring expectations for artificial intelligence back down to earth

The other obstacle was the sheer amount of data required to feed Watson’s deep learning network. Moreover, integrating socioeconomic data or patient-generated information, which could identify new patterns for certain subsets of patients, was difficult or impossible. 

“You may have 10,000 patients for lung cancer. That is still not a very big number when you think about it,” Chin told Stat.

A third concern revolves around the fact that Watson’s main source of diagnostic information comes from a group of experts at Memorial Sloan Kettering. Although the hospital is considered one of the most renowned cancer institutions in the world, that approach has led some to say Watson’s results are biased and pull only from a wealthy subset of patients.

The idea of AI bias has been a mounting concern, especially considering the data disparities that exist for certain populations like LGBT and minorities. Other experts have voiced concern that companies like IBM that are eager to capitalize on AI in healthcare first need to solve the industry’s data accessibility issues.