Hospitals are creating AI committees to manage a legitimate concern: how to adopt AI models without exposing patient information. Too often, however, the review begins and effectively ends with one question: Was protected health information (PHI) used to train the model?
Consider a large language model fine-tuned on clinical notes and a computer-vision model trained to return a blur mask or a binary "in body" or "out of body" label. Both may have been developed using PHI. But one can accept arbitrary prompts and generate open-ended responses, while the other performs a fixed technical task and returns a constrained output.
Under HIPAA, a business associate may use PHI only as permitted by its agreement with the covered entity and must apply appropriate safeguards. That is the legal baseline. It is not the end of the privacy analysis. The harder question is: What realistic pathway exists for the deployed model to expose information about a patient?
One checklist can fail in two directions
The problem with one-size-fits-all review is not simply unnecessary paperwork. It can make hospitals worse at identifying both high-risk and low-risk AI models.
For a generative model, a generic checklist can create false confidence. A committee may confirm that a business associate agreement is in place, data is encrypted and access is logged, yet fail to examine the model's most meaningful exposure pathways: whether users can enter arbitrary prompts, whether the model can be queried repeatedly and adaptively, whether it retrieves information from live clinical records, whether prompts and outputs are retained and whether it can reproduce information from its training or retrieval context.
The same checklist can over-govern a narrow model. A closed classification or segmentation model may have no prompt interface, no generative capability, no exposed model weights or embeddings and no output beyond a label, timestamp, bounding box or redaction mask. Requiring it to undergo the same privacy review as an open-ended generative model ignores the architecture rather than evaluating it.
That can directly increase privacy exposure. A face-detection model must learn from images containing faces, just as an out-of-body detector for minimally invasive video must see the identifiable frames it is intended to flag. Blocking controlled training on PHI can reduce redaction accuracy and preserve manual workflows in which more people view, handle and retain identifiable data for longer.
Collapsing the distinction, thereforem cuts both ways: hospitals can under-govern high-risk generative models while also obstructing narrow AI models that reduce downstream PHI exposure.
'Trained on PHI' describes the development process, not the model’s exposure risk
When a narrow model is fine-tuned, training data influences its mathematical parameters so it can perform a defined task more accurately. That does not automatically make the resulting model a searchable database of patient records.
For a closed, non-generative model that exposes neither its internal representations nor an exploratory query interface, and returns only tightly constrained outputs, there may be no realistic pathway for a hospital user to retrieve an underlying patient image or record. The information influenced the model's weights, but that does not make it meaningfully accessible.
HIPAA itself reflects a practical rather than absolute conception of privacy risk. Under the Expert Determination method, information may be considered de-identified when a qualified expert determines that the risk of identification is "very small" in the relevant context. The standard is not mathematical impossibility. Once information is properly de-identified under the Privacy Rule, it is no longer PHI. Hospital AI governance should apply the same discipline when assessing the residual exposure created by an AI model.
Academic research can identify theoretical attacks against almost any technical architecture under favorable assumptions. A demonstration may require direct access to model weights, extensive knowledge of the training process, enormous numbers of adaptive queries, detailed confidence scores or a closely matched reference dataset. Those conditions should not be treated as equivalent to an exposure pathway available in the actual deployment.
Security teams do not evaluate encryption by asking whether compromise is mathematically conceivable. They examine the algorithm, implementation, attacker access, computational cost and likelihood of success. AI privacy review should apply the same practical discipline.
Demanding proof of zero theoretical risk is not rigorous risk management. It is risk avoidance untethered from probability, and it can obstruct AI models that reduce larger, measurable privacy and clinical risks.
Privacy risk is one dimension, not a universal score
This does not mean that every narrow model is low risk or that every generative model should be rejected.
A narrow model used to guide a critical clinical decision may present little privacy exposure but require extensive review for accuracy, bias, robustness and patient safety. A generative model operating on de-identified information may have a lower privacy risk while still requiring scrutiny for hallucination, inappropriate reliance or cybersecurity vulnerabilities.
"AI risk" is not a single property. Privacy, security, clinical safety, bias and operational dependence are separate dimensions. An AI model can be low risk in one and high risk in another. Applying the same undifferentiated process to every model obscures those distinctions.
Five questions committees should ask
A technically and legally serious privacy review should begin with five questions:
- What can the model accept and produce? Arbitrary clinical text and open-ended answers create a different exposure surface than labels, masks, coordinates or timestamps.
- Can a user probe it? A promptable model supporting repeated, adaptive queries is different from a fixed-function model with no exploratory interface.
- What is actually accessible? Final outputs are not the same as access to confidence scores, embeddings, model weights, training infrastructure or unrestricted APIs.
- What is the credible path to a patient? Reviewers should describe the access, steps and resources required to move from the deployed model to identifiable information. "It may be theoretically possible" is not a complete analysis.
- What risk is created by saying no? Blocking the controlled use of PHI for model training may reduce de-identification accuracy or preserve manual workflows in which more people must repeatedly view and handle PHI, creating additional opportunities for accidental exposure.
The final question is routinely neglected. Committees meticulously scrutinize the privacy risks of approving an AI model without weighing them against either the clinical and operational benefits they may forgo or the additional privacy exposure created by preserving manual, PHI-intensive workflows.
Govern the model that actually exists
Hospitals should scrutinize AI models rigorously. But rigor is not the same as uniformity.
A promptable generative model connected to clinical records, a downloadable foundation model and a closed segmentation model should not receive the same privacy assessment merely because each was trained using patient information. They have different interfaces, outputs, access models and realistic failure modes.
The alternative is not weaker governance. It is more intelligent governance: confirm that the data use is legally authorized, understand the model's actual architecture, identify credible exposure pathways and calibrate the review to the risk that exists.
Treating every AI model alike can create false reassurance where deeper scrutiny is needed and unnecessary paralysis where it is not. Better patient protection will come from more precise governance, not more uniform governance.
Peter Grantcharov is the chief technology officer of Aimbient (formerly known as Surgical Safety Technologies). David Knobel is a partner at Hopp & Partners and a legal advisor to Ambient.