In a new independent assessment of AI scribe technologies, the Peterson Health Technology Institute (PHTI) found evidence that AI scribes decrease burnout and cognitive load for clinicians but have not proven a financial return on investment.
The AI scribe market has boomed in the last year, but the healthcare industry does not have a standardized set of metrics to determine the value of the nascent, and expensive, technology.
As health systems are deciding whether to invest beyond pilot stages, the PHTI compiled a report summarizing existing evidence of the benefit of AI scribes alongside eight health systems, 10 vendor companies and outside experts in AI.
Though the organization typically rates the efficacy of digital health companies' solutions in a given category—like diabetes management platforms, remote monitoring for hypertension and musculoskeletal solutions—it decided to take an early look at AI scribes because of their firestorm entrance into the industry.
“There is no technology in recent memory that has been adopted more enthusiastically by clinicians or has scaled so uncharacteristically fast, absent a regulatory mandate,” the PHTI wrote.
The PHTI categorizes the measures into process metrics—like time spent in notes and percentage adoption—clinician experience metrics, financial metrics and quality metrics.
AI scribes have not achieved significant return on investment to date, the PHTI found. And, while provider organizations are mainly adopting AI scribes to reduce physician burnout, it will become increasingly important for the organizations to provide financial returns, the PHTI writes.
“Most health systems have secured executive support for pilot-level initiatives but have not necessarily aligned the requisite data analytics and measurement tools to conduct a rigorous assessment of impact across a comprehensive set of process, experience, financial, and quality metrics,” the report says. “As a result, executives do not have all the information they need to decide to scale or refine implementation.”
AI scribes decrease burnout, but results are mixed on pajama time
The PHTI found solid evidence from participating provider organizations that ambient scribes decreased the cognitive load and burnout of their physicians.
“[Mass General Brigham] reported a 40% relative reduction in reported burnout in a six-week survey pilot (n=223),” the report reads. “Similarly, MultiCare shared that clinicians surveyed after its pilot reported a 63% reduction in burnout and a 64% improvement in work-life balance due to ambient scribe.”
Generally, physicians seem to keep a significant portion of the notes generated by AI in the final clinical documentation, which reduces the cognitive load of physicians. The notes are also of higher quality than provider-generated notes when a clinician can review the AI note.
In the report, the PHTI cites a statistic by Yale New Haven Health that clinicians keep an average of approximately 80% of the AI-generated draft of the note. There’s also evidence to suggest that the patient’s experience of the visit is improved when an AI scribe is used.
In other categories, however, feedback and academic literature were mixed or too nascent to identify a strong conclusion.
Provider organizations had mixed feedback when asked whether ambient scribes saved clinicians time or reduced the amount of after-hours work they had to spend on charting, known as “pajama time.”
A Stanford analysis found AI scribes can “modestly reduce documentation and EHR time.” At some systems like Mass General Brigham and Intermountain Health, the tools decreased pajama time, but others reported no difference or no definitive way to tell whether pajama time was reduced, the PHTI found.
AI scribe company Abridge has published case studies from pilots at the University of Vermont Health Network and Corewell Health that say provider pajama time decreased by 60% and 48%, respectively, the PHTI reported. The Permanente Medical Group likewise has published a study that found decreased pajama time.
PHTI finds no firm results on financial returns for ambient scribes
Increasing patient throughput is a potential financial benefit for provider organizations. The PHTI found little evidence to support the financial return, though organizations noted that throughput isn’t a current focus of their use of the technology,
Provider organizations reported no increase in the number of patients they were able to see in a given time frame when using an ambient scribing tool. However, providers said they thought they could see more patients or may be willing to take on a last-minute patient because of the tool.
Providence found more patients added to clinic schedules in specialties with ambient scribe.
Many ambient scribe vendors also leverage the insights gleaned from provider-patient interactions to suggest billing codes for the provider. Many vendors claim that their technology can result in higher reimbursement for an interaction because it may include codes for things that providers didn’t pick up on or was a secondary issue in the visit.
While Multicare and Ochsner Health noted increases in billing and accuracy, other health systems reported no impact. Ambient scribe vendor Suki AI claims that users have a 5% higher encounter volume equivalent to $54,000 in additional revenue per user per year, according to the report.
More assessment is needed, PHTI says
The report reflects findings from early adopters of ambient AI scribes. Many organizations have only explored pilots of the technology, which limits their return on investment, Accenture wrote in a recent report.
Healthcare executives involved in the creation of the report said they would welcome standard industry metrics for ambient scribes.
The PHTI’s AI Taskforce is working to develop an administrative technologies assessment framework for healthcare organizations that want to procure products like AI scribes or revenue cycle management AI products, which the PHTI expects to experience a similar boom as AI scribes.
The health systems involved in the report were: CommonSpirit Health, Intermountain Health, Mass General Brigham, MultiCare, Ochsner Health, Providence, UC San Diego Health and Yale New Haven Health.
The vendors involved in the report were: Abridge, Commure, DeepScribe, Fourier Health, Infinitus Systems, Microsoft, Nabla, Oracle, R1 and Suki.