Risks with AI-enabled health technologies topped the list of the ECRI Institute's annual ranking of health technology hazards.
Also topping the list are lack of technical support for medical device use in the home, cybersecurity threats from third-party vendors and substandard or fraudulent medical devices supplied by unauthorized distributors.
The list is compiled from reports of medical device errors by users, lab testing and conversations with providers.
While most of the safety institute's report focuses on the hazards of medical equipment, like risk of fire near supplemental oxygen, AI and cybersecurity ranked number one and three, respectively, on the top 10 list, signaling the growing risk of software on the healthcare industry.
ECRI recognizes the cost savings and benefits of AI, but says healthcare organizations should keep close track of potential biases of the algorithm, hallucinations and model drift.
“Placing too much trust in an AI model—and failing to appropriately scrutinize its output—may lead to inappropriate patient care decisions,” the report says. “AI offers tremendous potential value as an advanced tool to assist clinicians and healthcare staff, but only if human decision-making remains at the core of the care process.”
ECRI calls for strong AI governance by hospitals and health systems; but, usually only large academic medical centers and other well-funded healthcare organizations have the bandwidth for ongoing performance monitoring of AI, experts say.
Small and rural healthcare facilities could be left by the wayside in the AI revolution because they lack the personnel and budget to vet the performance of new AI technologies.
Healthcare organizations must set expectations for AI, define goals, provide sufficient governance and adequately prepare their data to be used by the algorithm, ECRI says.
Second on ECRI’s top 10 hazards list is the operation of a complex medical device when a patient is being treated at home, whether through a hospital-at-home program or chronic condition management.
ECRI says that unmet technology support needs for homecare patients could lead to improper setup and lack of familiarity with the device.
Healthcare organizations should also ask if the device is usable by a nonclinical expert, if there are physical or structural limitations to use in the home, and if the patient or caregiver has been appropriately trained on the device.
ECRI placed vulnerable technology vendors and cybersecurity threats third on its list of hazardous health technologies. It pointed to the February 2024 cyberattack on Change Healthcare, a UnitedHealth Group subsidiary, which comprised medical claims processing for a large swatch of the industry.
ECRI warned that reliance on a third-party vendors for revenue cycle management or electronic health records could pose a risk to healthcare organizations. If a third-party vendor is attacked or compromised, healthcare organizations could be at significant risk of temporarily discontinuing services or leaking sensitive health data.
ECRI urges healthcare organizations to thoroughly vet third-party vendors and assess security risks.