The following is a guest article by Pooja Walia and Rajat Rawal
Most health systems we work with have passed the pilot stage for AI. Ambient documentation tools are in exam rooms. Revenue cycle models are running against live claims. Clinical decision support is nudging diagnoses across specialties. The tools are in the building.
What we keep seeing is that the governance that looked solid before go-live starts to slip a few months after. The policies are still on the shelf. The risk assessment was signed. The oversight committee met. But the AI that is running today is not quite the AI that was reviewed, and the people meant to govern it have no easy way to tell.
This matters more than it did a year ago, because the regulatory picture has changed. The proposed HIPAA Security Rule overhaul, expected to be finalized in 2026, removes the “addressable” safeguard loophole and brings AI systems that handle electronic protected health information explicitly into scope. The FDA’s revised Clinical Decision Support Software guidance, published in January 2026, narrows the exemptions many AI tools have been operating under. Texas, California, Colorado, and a growing list of states are adding disclosure and governance requirements for AI in clinical decisions. The NIST AI Risk Management Framework, which the federal government increasingly treats as the reference standard, expects ongoing oversight, not a single point-in-time review.
The common thread is that these rules assume health systems know what AI is running, who is accountable for each tool, and whether the governance is still accurate months after go-live. For many organizations, those assumptions do not yet match reality.
Here is where governance tends to break down, and what tends to help.
The Review was a Snapshot, The Tool is a Movie
Most AI governance work is front-loaded. Before go-live, the model gets tested. Bias is checked. Data flows are mapped. A risk assessment is written. On day one, the tool is genuinely well-governed.
Then it runs. Patient populations shift. Vendors push model updates on their own schedule. Clinicians use the tool in ways that were not part of the original design. Edge cases show up that never appeared in the test set.
The risk assessment still describes the system as it existed at launch. The system no longer exists that way. The gap widens quietly, and usually no one notices until something visible goes wrong.
The proposed HIPAA rule expects annual risk assessments that reflect the current state of the system, not the state at deployment. The NIST AI RMF’s MEASURE function expects continuous monitoring for the life of the deployment. Both point to the same practical need.
What helps: set a monitoring baseline at go-live and review it on a schedule. Performance. Drift. Override rates. Vendor update logs. Monthly for routine tools, more often for anything touching clinical decisions. This is not a new committee. It is a standing 30-minute review.
The Workflow on Paper is Not the Workflow in Practice
Governance documents describe how the AI is supposed to be used. A clinician is supposed to review each output. The AI is supposed to be one input among several. The recommendation is supposed to be advisory.
The real workflow often looks different. Busy clinicians rely on the tool more than the design assumed. A suggestion meant to be one data point becomes the anchor. Advisory outputs become the default because the reviewer does not have time to second-guess them.
None of this is negligence. It is what happens when thoughtful design meets an overloaded schedule. But if governance is only looking at the intended workflow, it misses what is actually happening.
This gap has compliance consequences now. The FDA’s updated CDS guidance looks at how the tool is used in practice, not just how it was designed. State laws like California’s AB 3030 require disclosure when AI meaningfully contributes to a clinical decision, which means the organization has to know when that threshold is crossed in the live workflow.
What helps: look at usage data. Which outputs are clinicians accepting without edits? Which are they overriding? Which are they clicking past? The answers tell you how the tool is really being used and where governance assumptions no longer match reality.
Escalation Paths Exist on Paper, Clinicians Cannot Find Them
A pattern we see often: a clinician notices something off about an AI tool. An output looks different. Confidence scores shifted. Results feel inconsistent with what the tool was producing last month. The clinician has a gut sense that something is wrong, and no idea who to tell.
Compare this to how other clinical technologies work. If a medication is wrong, there is a reporting process everyone knows. If an imaging machine misbehaves, biomedical engineering is a phone call away. When an AI tool drifts, the path is usually unclear. Is this an IT ticket? A vendor issue? A safety event? Who owns this?
Under the proposed HIPAA rule, incident response is a formal requirement, and it has to be operational, not just documented. ONC’s algorithm transparency rules expect certified health IT to support similar accountability.
What helps: every AI tool gets a named owner. Clinicians know who to contact. The process does not have to be elaborate. It has to be clear, and people have to know it exists. Treat AI systems the way you already treat other clinical technologies, and most of this gap closes.
Human-in-the-Loop Only Counts if the Human can Actually Review
This is the phrase we hear most often in healthcare AI, and it is also the one most likely to be a formality. Having a clinician click “approve” is not the same as having the clinician meaningfully review the output. If the workflow pushes them to approve twenty outputs in a minute, they are not reviewing. They are rubber-stamping.
Governance that assumes careful review, when the workflow makes careful review impossible, creates a gap between documentation and reality. The record will say a human reviewed each case. The reality will be different. Several state AI laws now explicitly require meaningful human oversight, not just nominal review, which means this gap is increasingly a legal exposure, not just a clinical one.
What helps: design the workflow so that real review is possible in the time clinicians actually have. If a real review is not possible, face that directly. Either invest in the time and structure to do it properly, or pull back on how much the AI is trusted to do on its own. Do not let the phrase carry weight it does not earn.
The Feedback Loop is Usually Missing
The better AI programs we have seen treat what happens after deployment as part of the system, not an afterthought. Clinicians can flag outputs. Overrides are logged. Patterns get aggregated and fed back to vendors or internal teams. Changes to the model or the workflow trigger a review instead of just happening quietly.
Most programs do not have this yet. It is the piece that turns governance from a document into a practice, and it is where the NIST AI RMF’s MANAGE function expects organizations to operate. Without it, the organization is flying on instruments that were calibrated at takeoff and never checked again.
Where this Leaves Us
AI in healthcare is past the stage where governance can stop at the point of deployment. The regulations are catching up fast, and they are catching up in the same direction: continuous oversight, named accountability, meaningful human review, and a feedback loop that captures what the system is actually doing in production.
The organizations that will hold up are the ones that treat post-deployment governance as part of the job. A monitoring baseline. A named owner for every tool. An escalation path that works in practice. A workflow that supports real clinician review. A feedback loop that learns from the live system.
None of this requires a new framework. It requires treating AI the way healthcare already treats everything else that affects patient care, which is as something that needs ongoing attention, not a one-time sign-off.
That is the work now.
About the Authors
Pooja Walia is a seasoned IT professional who works with healthcare organizations to design and operationalize secure, scalable, and compliant AI systems in regulated environments. Her work focuses on translating AI innovation into reliable, real-world systems.
Rajat Rawal is a technology leader who supports healthcare organizations with implementing cloud and AI solutions, with a focus on operational scalability, system reliability, and navigating critical deployment challenges.
The views expressed in this article are the authors’ own and do not reflect the views of their employer.
No comments:
Post a Comment