The following is a guest article by Angela Adams, CEO at Inflo Health
Healthcare has spent the past several years adding AI into clinical workflows across imaging, documentation, and decision support. It’s mostly been in service of less manual work, better triaging, and fewer things falling through the cracks, all of which are worthwhile pursuits. What gets less attention amid all the hype, though, is what happens when these tools are wrong.
I don’t mean catastrophically wrong. I mean quietly wrong: a finding that doesn’t route anywhere, a recommendation no one owns, an automated step that fails without anyone noticing. In most health systems today, nobody can answer the most basic governance question about the AI in their stack: when this tool makes a mistake, who catches it, and how fast can they do so?
From imaging to documentation to decision support, healthcare teams have integrated ever more models into their workflows to save time and ensure patient engagement loops are properly closed. Each one adds new capability, but each one also adds a new place where something can fail—sometimes silently. As AI touches more of the patient journey, those failure points multiply faster than anyone’s ability to trace them.
As AI’s role in supporting healthcare workers begins to integrate further with patient care and internal processes, more touchpoints become disconnected and difficult to trace due to a lack of clear governance across the models.
Roughly half of radiology follow-up recommendations are never completed. Patients absorb the delayed diagnoses while health systems absorb the avoidable costs and the liability. In most cases, that failure is hard to notice. There are no alerts. Nothing escalates. The gap becomes visible only when the patient comes back, usually sicker.
What’s Next? Governance that Lives Within the Workflow
The next wave of digital health differentiation will come from practical AI governance that operates at the workflow level. No single platform can govern every model, workflow, and user action across the enterprise, but what if each platform had governance built in?
Built-in governance means the system knows what should happen after every output. Based on embedded rules, it knows who owns the next step, what the time window is, and what triggers escalation if the step doesn’t happen. Audit trails run from finding to completed action. Committee-level governance asks whether a tool should be deployed. Workflow-level governance asks, every day, whether the tool’s outputs are actually turning into safe care.
The federal HTI-1 Final Rule introduced new algorithm transparency expectations for AI and predictive tools in certified health IT, raising the bar for what clinical users should be able to understand about the tools that support decision-making. That direction reinforces a broader market reality: governance needs to be tied to patient safety and measurable outcomes, not committee rituals.
What Practitioners Should Demand from Vendors
As practitioners, it’s incumbent upon us to demand that the EHR-adjacent vendors we welcome into the walls of the hospital actually set our teams up to succeed. If governance will be baked into all health IT predictive tools, that means that health systems can make decisions based on what models are willing to offer. The key things to look out for include:
- Role-Based Output Controls: AI outputs should reach the right person, at the right time, every time; ensuring that information gets into the hands of the person equipped to act on it—and that they have the right permissions—protects patient safety, privacy, and the integrity of the workflow
- Audit Trails that Follow Care: Maintaining clear paths along which teams can easily find the source of mistakes prevents patient harm, nips growing issues in the bud, and allows systemic fixes instead of one-off band-aids
- Escalation Pathways: The most dangerous failing in clinical AI isn’t an incorrect finding, it’s a right answer that goes nowhere; vendors should be able to show what happens when a follow-up stalls: who gets notified, on what timeline, and what prevents the case from simply aging out of view— with clear steps already in place, no incidents go without a resolution plan
When we take on the mantle of providing care to patients, the first promise we make is to do no harm. That bond exists between clinicians and patients, but it exists everywhere else, too—and the promise doesn’t stop applying when the work is done by software. When we consider allowing AI into our hospitals and care relationships, how do we ensure these tools strengthen that promise rather than weaken it? How do we know that we are elevating care and not just making things easier for the sake of making things easier?
Through governance: knowing what the tool touches, who acts on its output, and what happens when it’s wrong. By putting patient safety first in every decision, we keep our oath and build trust with the people healthcare was always supposed to be about.
About Angela Adams
Angela Adams, RN, started her career as a critical care medicine nurse at Duke University Medical Center. Driven to make a broader impact, Angela looked to the emerging healthcare AI segment for solutions that would allow her to help patients as well as assist clinicians to become more effective and efficient in solving complex medical issues. She helped advance AI adoption and overcome skepticism at companies like Jvion (acquired by Lightbeam Health Solutions), where she applied deep machine learning to lower nosocomial event rates and prevent patient deterioration. She went on to create her most recent solution at Inflo Health, where she focuses on missed follow-up radiology appointments.
About Mark Taylor