The following is a guest article by Dror Zelber, VP of Product Marketing at Radware
Healthcare organizations are rapidly deploying AI virtual assistants to help patients schedule appointments, understand medical information, and prepare for a visit. This is incredibly helpful for hospitals facing staff shortages and overloaded call centers, as these tools help provide faster service and a better patient experience.
At the same time, the Large Language Models (LLMs) being used by the AI assistants introduce new risks, especially if the systems behind them aren’t protected adequately. Recent research shows how a healthcare AI assistant used in a pilot program in Utah was manipulated to spread vaccine conspiracy theories, recommend methamphetamine as a treatment for social withdrawal, generate SOAP notes that tripled a patient’s baseline OxyContin dosage, and even provide instructions for cooking methamphetamine.
Cybersecurity is already a constant challenge for healthcare leaders, and AI is now adding another dimension to that risk. When AI assistants interact directly with patients, they create a new type of attack surface. Instead of targeting infrastructure such as servers or databases, attackers may now be able to manipulate the behavior of the system itself through conversation.
A Different Kind of Vulnerability
Traditional healthcare cybersecurity focuses on protecting infrastructure. Security teams work to safeguard networks, medical devices, electronic health records, and other systems that store or transmit sensitive patient information.
LLMs operate differently. These systems generate responses based on instructions embedded in system prompts that guide how the AI should behave. Developers use LLM prompts to define tone and rules about what the assistant is allowed to say, what it should avoid, and how to handle sensitive topics. In healthcare, those instructions often include guardrails such as avoiding diagnosis, referencing trusted sources, or escalating sensitive questions to human clinicians.
However, language models do not distinguish between legitimate and malicious instructions. Models are designed to please users and execute their instructions. This weakness enables what security researchers call prompt injection and model manipulation attacks. In a prompt injection scenario, an attacker hides instructions inside what appears to be a normal user message. The AI assistant processes the message as text and may follow the attacker’s instructions alongside the user’s legitimate instructions.
The attacker does not need to breach the hospital network or bypass authentication controls. The interaction takes place entirely through the chatbot interface.
When an AI Assistant Is Manipulated
Consider how many healthcare organizations are beginning to integrate AI assistants into patient portals, telehealth systems, and digital front doors. If an attacker successfully manipulates the system’s prompt behavior, the consequences may not appear immediately as a technical breach. The hospital’s servers may remain intact and patient records untouched.
Instead, the impact appears in the system’s responses. The assistant may generate misleading medical explanations or present fabricated information as legitimate clinical guidance. It could incorporate false regulatory updates or manipulated treatment guidelines into its recommendations. As the above example illustrates, the system may even generate structured medical documentation, such as SOAP notes that incorporate manipulated information and present it to clinicians as authoritative context.
While none of these scenarios require access to sensitive patient data, they can still influence medical conversations and decision-making. In healthcare, trust plays a central role in patient relationships. If digital tools provide inaccurate or manipulated information, confidence in the institution behind those tools can erode quickly.
Why Healthcare Faces Unique Risks
Many industries are experimenting with AI assistants, but healthcare carries particularly high stakes. Patients tend to view hospital systems as trusted authorities. When information appears on an official hospital website or patient portal, people often assume it has been medically reviewed.
That assumption creates a dangerous dynamic if an AI assistant is manipulated. Even subtle misinformation can influence how patients interpret symptoms, manage medications, or decide whether to seek care. While the system may not be issuing formal diagnoses, its responses still shape patient decisions.
In this sense, AI assistants are becoming part of the clinical information environment. Their outputs influence conversations between patients and providers, which makes their integrity a security issue as much as a technical one.
Key Security Practices for Healthcare AI Systems
Healthcare organizations deploying AI assistants should treat them as operational software systems rather than simple digital chat tools. Since these systems interact directly with patients and clinicians, their behavior must be governed with the same rigor applied to other clinical technologies.
Several security practices can significantly reduce the risk of manipulation.
- Validate and Sanitize User Inputs: Prompt injection attacks often rely on hidden instructions embedded in normal-looking messages; filtering and validating user inputs before they reach the model can reduce the likelihood that malicious instructions will be processed
- Separate System Instructions from User Conversations: System prompts should be isolated from user input so that attackers cannot easily override the guardrails that define how the AI should behave; clear separation between system instructions and conversational content makes prompt manipulation more difficult
- Monitor AI Outputs for Anomalies: AI assistants should be monitored continuously for abnormal responses or behavior patterns; logging and reviewing outputs can help identify situations where the system may be generating misleading or manipulated information
- Conduct Adversarial Testing Before Deployment: Security teams should simulate prompt injection attempts during development and staging environments; red-team exercises can reveal weaknesses in prompt design and system architecture before the AI system interacts with patients
- Adopt Emerging AI Security Frameworks: Guidance such as the OWASP Top 10 for Large Language Model Applications provides a useful framework for understanding common AI risks, including prompt injection, data leakage, and model manipulation; these frameworks help organizations incorporate AI risks into their broader security strategy
As healthcare organizations expand the use of AI-driven patient engagement tools, these practices can help ensure that innovation does not come at the expense of safety, reliability, or trust.
AI Innovation Must Be Secured
AI assistants have the potential to improve healthcare by reducing administrative burdens and helping patients access information more quickly. However, these systems also introduce a new category of cyber risk. Healthcare organizations must treat AI assistants with the same level of scrutiny applied to other clinical technologies. As AI adoption accelerates, ensuring these systems remain trustworthy will require strong governance, security testing, and continuous monitoring.
Dror Zelber is VP of Product Marketing and formerly VP of Management at Radware and a 30-year veteran of the high-tech industry specializing in security, networking, and mobility technologies. He holds a bachelor’s degree in computer science and an MBA from Tel Aviv University.
No comments:
Post a Comment