Generative AI models can identify social determinants of health (SDOH) in doctors’ notes, a new study finds.
SDOH like housing and income are major drivers of health outcomes. But these factors are notoriously underdocumented in existing EHR structured data, study authors wrote in npj Digital Medicine. And while a subset of billing codes exists to capture SDOH data for quality improvement and health equity efforts, they are sorely underutilized.
To tackle the issue, a team at Mass General Brigham led by Danielle Bitterman, M.D., a physician in the Department of Radiation Oncology at Brigham and Women’s Hospital, fine-tuned language models to spot rare references of SDOH in visit notes of cancer patients getting radiotherapy.
Their study showed that the models, when finely tuned, could identify the vast majority of patients (nearly 94%) with an adverse SDOH. This could augment efforts to identify patients who may benefit from social resource support, the study said.
Health disparities linked to SDOH are particularly important in cancer care, where patients face a complex and demanding medical journey. The distance a patient lives from a major medical center or the support they have from a partner can substantially influence outcomes.
“We should use all the information that we have on these patients in a safe way to try and support their care proactively,” Bitterman told Fierce Healthcare.
Bitterman is also a faculty member of the AI in Medicine Program, housed at Mass General Brigham. The program is a collaborative of about half a dozen labs that focus on translating safe and effective AI into clinical practice.
To train the language models, Bitterman’s team manually reviewed 800 clinician notes from the Department of Radiation Oncology at Brigham and Women’s Hospital. Because only 3% of sentences in the training set contained any SDOH mention, they also used ChatGPT to generate 900 additional synthetic examples to further train the models.
They tagged sentences that referred to one or more of several predetermined SDOH: employment status, housing, transportation, parental status if the patient has a minor, relationships and presence or absence of social support.
The team then tested their models using an additional 400 clinic notes from patients treated with immunotherapy at Dana-Farber Cancer Institute and from patients admitted to the critical care units at Beth Israel Deaconess Medical Center.
Additionally, the researchers compared the language models they trained to much larger generalist models like OpenAI’s GPT-4. They found that the specialized models the team used were less likely than OpenAI’s GPT-4 to change their determination about an SDOH based on a person’s race, ethnicity or gender — in other words, the smaller models were less prone to bias.
Large language models like GPT-4 are more expensive to use and often less accessible than smaller, more specialized models, per Bitterman.
Further, "if we start relying on the proprietary models, not the open-source models, we start to get into the question of who has access,” Bitterman said.
As a next step, Bitterman hopes to quantify how this type of data extraction could improve the physician experience and patient outcomes. But she is under no delusions about how difficult it can be to operationalize such an application of AI. Many things have to be considered, from clinician alert fatigue to mitigating bias to patient consent.
“We’re at just the inflection point of these models entering clinic,” Bitterman said. “The risk of bias is very real.”
Many SDOH can be sensitive in nature, and a patient may not want them brought up, she added. It behooves healthcare professionals to think about how each patient might want or not want AI to be used on their data.
“AI models, because they can process so much data, … can find hidden correlations that may reveal sensitive information about patients,” Bitterman said.
Key to mindful development and integration of AI is to include patients, Bitterman said. Her lab actively collaborates with patient advocacy organizations and is planning studies on how patients want to interact with language models.
“The public right now does not have enough of a voice in how AI is being developed and implemented,” she said. If practitioners begin using AI to inform care and not telling patients about it, that could cause harm.
“We’re going to lose a lot of trust early on and lose a lot of the amazing opportunity of AI to support healthcare,” Bitterman said.