LAS VEGAS—At the HLTH conference in 2023, Google's cloud division unveiled new artificial-intelligence-powered search capabilities to help doctors and nurses quickly access patients' clinical notes and other medical information across different data sources.
Google Cloud's Vertex AI Search for Healthcare tool is designed to make it faster for healthcare professionals and other employees to find critical information within health records and medical documents. The technology enables medically tuned, generative AI searches on a broad spectrum of data including FHIR information and clinical notes, the company said.
Google Cloud continues to enhance its Vertex AI Search platform and build out gen AI capabilities for healthcare.
The company announced Monday at the Healthcare Information and Management Systems Society (HIMSS) 2025 Global Health Conference & Exhibition new gen AI capabilities in Vertex AI Search for healthcare, including a new feature called Visual Q&A that searches tables, charts and diagrams.
"We're helping clinicians to work more efficiently with new multimodal capabilities embedded in Vertex AI Search for healthcare," said Aashima Gupta, global director of healthcare strategy and solutions at Google Cloud, in a statement. "Multimodal analysis processes diverse sources of patient data, like medical images and genetic information, for a more comprehensive understanding and improved decision-making."
Google Cloud also debuted the availability of Gemini 2.0 as one of the models now within the offering.
In healthcare, nearly 90% of healthcare data are in image form such as X-rays, scans or photos. Multimodal technology can process and integrate information from diverse sources such as images, videos and text.
AI models can combine data from medical images, patient history and genetic information to give healthcare professionals a comprehensive understanding of a patient's health. The aim is to use richer data inputs to help providers make more accurate diagnoses and design more personalized treatment plans, according to Google Cloud executives.
The company's visual Q&A capability enables Vertex AI Search for healthcare to receive images such as tables, charts or diagrams directly as an input rather than taking the image and first converting it into text. Google Cloud gives the example that the technology could receive a diagram of a patient's feet indicating where they have an injury alongside a checkbox from a form the patient filled out indicating whether they are wearing therapeutic shoes and any other symptoms indicated on the form. Visual Q&A can also analyze medical forms where information is entered into tables, and, when this information is a relevant part of a search, it will include it in the findings for the clinician.
Gemini 2.0 is Google's latest AI model and what the company describes as faster and more capable than previous models. The use of this AI model enables Vertex AI Search for healthcare to deliver faster, more accurate answers, the company said.
Several health tech companies are partnering with Google Cloud to use Vertex AI Search. Counterpart Health, an AI-powered physician enablement company and a subsidiary of Clover Health, developed a technology platform called Counterpart Assistant that provides real-time, patient-specific insights directly within physicians' workflows.
David Tsay, M.D., Ph.D., chief medical officer at Counterpart Health, said advancements in Vertex AI Search for healthcare provide the company with powerful new tools. "Multimodal AI capabilities will enable clinicians to integrate diverse types of clinical data sources, providing a more comprehensive understanding of their patients' health and facilitating personalized care management," Tsay said.
Electronic medical record company Meditech developed an AI-powered search and summarization capability in its Expanse EHR to enable care teams to find the information they need using Google's familiar search function and gen AI. The company said the new Vertex AI Search features will further advance its EHR capabilities.
Health tech company Suki upgraded its voice assistant technology Suki Assistant to include patient summarization and clinical Q&A for clinicians, built on Google Cloud technology.
Google Cloud is using its tech muscle to build out gen AI capabilities for healthcare and life sciences companies to help offload administrative burden and support assistive technology for clinicians.
At HIMSS 2024, the company unveiled that Vertex AI's search and question-answering capabilities integrated with MedLM, healthcare data engines and cloud healthcare FHIR APIs, making it easier for healthcare and life science organizations to build advanced data analytics and AI solutions.