As the adoption of medical AI rapidly grows, healthcare AI companies are shifting from simply generating answers to proving why those answers should be trusted.
OpenEvidence, an AI-powered medical search engine, developed a medical AI copilot feature that grades and visualizes the quality of the published evidence cited in, and used to inform, every answer to a clinical question. The feature, called EvidenceGrade, adds a critical layer of context to aid in incorporating medical evidence into high-stakes medical decisions, the company asserts.
A well-known limitation of AI systems is that when summarizing multiple sources, they gloss over differences in the quality of those sources, according to OpenEvidence. This limitation can have significant impacts in healthcare with the difference between evidence from a randomized, blinded, placebo-controlled trial and evidence from a small observational study in a foreign population.
EvidenceGrade surfaces, quantifies, grades and visualizes the quality of the evidence behind OpenEvidence answers in terms of the safety with which physicians can apply that evidence in real-world clinical practice, according to Daniel Nadler, founder and CEO of OpenEvidence.
Beyond just citing the sources, OpenEvidence's new feature scores the underlying quality of the evidence behind those sources and exposing that assessment to clinicians in real time.
The company provided Fierce Healthcare with a first look at the new feature.
It marks an ongoing shift as healthcare AI companies are increasingly competing on trust, not just accuracy. As generative AI tools become more capable, the differentiator is no longer merely answering clinical questions quickly.
EvidenceGrade was developed by OpenEvidence's team of medical AI scientists, led by physician-scientists Sam Finlayson, M.D., Ph.D. and Travis Zack, M.D., Ph.D. and lead machine learning scientists Evan Hernandez, Ph.D. and Eric Lehman, Ph.D., Nadler said.
EvidenceGrade builds on the GRADE framework — Grading of Recommendations Assessment, Development and Evaluation, a widely adopted standard for evaluating the quality of evidence. It's the methodology behind Cochrane, a non-profit network of health researchers, professionals and patients, the World Health Organization and most major clinical guidelines.
"The team consulted with experts in evidence synthesis and reviewed the approaches used by Cochrane and similar teams. These approaches were then adapted to the unique demands of real-time decision making OpenEvidence is designed to support," Nadler told Fierce Healthcare.
Nadler walked through how EvidenceGrade scores a question: Questions are first classified to determine whether they're suitable for grading, then all retrieved papers are scored for quality, certainty and relevance. The model is trained to weigh study design strength, consistency and precision across sources, and how directly the evidence applies to the question — mirroring how an expert methodologist would appraise a body of evidence, he said.
An "A" grade means the evidence is supported by the best-achievable study designs for the question: Randomized controlled trials or rigorous systematic reviews, well-conducted prospective cohorts or registries, or current evidence-based guidelines with strong recommendations. A “B” grade is moderate evidence supported by appropriate study designs with notable limitations on one or more axes: modest sample sizes, surrogate outcomes, observational evidence applied to a therapy question, or moderate imprecision.
A grade of “C” designates limited evidence that rests on weaker designs such as narrative reviews or expert opinion, or stronger designs with serious limitations across multiple axes. And a “D” grade indicates minimal evidence that rests on case reports, case series, preclinical data or mechanistic extrapolation and offers a directional signal.
EvidenceGrade assigns a U when a grade cannot be assigned.
"Not all evidence is equally certain, and when a clinician acts on an answer, it's imperative they understand how much weight the underlying evidence can bear," said Samuel Finlayson, M.D., Ph.D., Senior Vice President of Medical AI at OpenEvidence. "Expert research appraisal teams do invaluable work, but they can't cover every question that comes up in everyday practice. We built EvidenceGrade to extend their methods to as many questions as possible. We see it as a complement to traditional evidence synthesis, and we're releasing it as a starting point that will improve as the clinical community helps us refine it."
OpenEvidence developed an AI-powered medical search engine and a generative AI chatbot exclusively for doctors that summarizes and simplifies evidence-based medical information. As of July, there were 915,000 medical licensed-verified U.S. clinicians, including nurses, nurse practitioners and physician assistants, using OpenEvidence, with more than 690,000 licensed-verified U.S. physicians.
The company also announced on Thursday a collaboration with NewYork-Presbyterian and its affiliated medical schools, Columbia University Vagelos College of Physicians and Surgeons and Weill Cornell Medicine, to deploy OpenEvidence across all hospitals and care sites.
OpenEvidence will be available to clinical staff at NewYork-Presbyterian, Columbia and Weill Cornell Medicine, the organizations said.
“Every patient is unique, and their care should reflect that,” said Chif Umejei, Chief Information Officer at NewYork-Presbyterian in a statement. “By making OpenEvidence available across our systems, clinicians can use AI to seamlessly access and analyze medical research in real time and use that information to deliver informed and compassionate care to the communities we serve.”
It marks another enterprise-wide integration with a major health system in the New York City area as the company also collaborates with Mount Sinai.
The company gained traction in the market as a free tool for clinicians, but OpenEvidence is now setting its sights on growing its footprint with hospitals and health systems. It also works with Sutter Health and Cedars-Sinai to integrate its AI-based medical search and decision-support platform into an organizations' electronic health record systems.
These health systems are currently using OpenEvidence's ad-supported model. Nadler told Fierce Healthcare in May that an enterprise model is in the works.
"We are pioneering a non-ad-supported, standard AI enterprise model, similar to Anthropic's business model, for large systems like Mount Sinai or Cedars-Sinai who want that option and where there is significant opportunity for enterprise-level customization and value-add beyond the core free-for-physicians OpenEvidence product," he said.