When a hospital deploys an AI triage assistant, a charting copilot, or a patient-facing symptom checker, it inherits a brand-new attack surface — one where a successful exploit isn't a data row, it's a patient. This framework maps the OWASP Top 10 for LLM applications to the thing that actually matters in healthcare: the human on the other end of the output.
The general AI-security world stops at the vulnerability class. In healthcare, every class has a second translation — into clinical consequence and into the compliance regime that governs it. That translation is the defender's job. Tags reference NIST AI RMF, HIPAA, ISO/IEC 42001, the EU AI Act, and FDA premarket cyber guidance where applicable. (List: OWASP LLM Top 10, 2025.)
Three walkthroughs in the DFR method: Triage → Stabilize → Treat → Recover. Written from the defender's chair. No payloads, no targets — the point is to see the failure chain and the controls that break it.
The setup. A health system rolls out an AI symptom checker on its patient portal. It reads the patient's message, asks follow-ups, and returns a triage level: self-care, schedule a visit, or seek emergency care. It also has read access to portal content — past messages, shared documents, care-team notes — to "personalize" its answers. That read access is the open door.
The attack, at a concept level. The model can't reliably tell the difference between the patient's words and instructions hidden in the content it's allowed to read. An attacker plants instruction-shaped text where the bot will ingest it — an indirect prompt injection. The bot, doing what it was told by text it shouldn't have trusted, quietly shifts behavior: it nudges genuinely urgent presentations toward "self-care," or folds another patient's details into its reply. Two OWASP classes fire at once — LLM01 (the override) and LLM02 (the disclosure).
The human consequence. This is why it's Critical, not High. The output isn't a corrupted record — it's a person told to stay home who needed an ambulance. The failure is invisible at the moment it matters, and it scales: every patient on that path gets the same compromised judgment.
Scope the blast radius first. Which model version, which prompt template, which data sources can it read, and which triage decisions did it issue in the affected window? Treat every downgrade in that window as suspect until cleared. Identify whether PHI left the boundary.
Contain without going dark on patients. Drop the bot to a safe-default mode — when confidence is low or urgency signals are present, it escalates to a human and stops auto-triaging. Sever the untrusted read source. Patient safety beats uptime; a slower human queue is acceptable, a silent downgrade is not.
Fix the trust boundary, not just the symptom. Separate instructions from data, constrain and sanitize what the model may read, enforce least-privilege on its data access, and add an independent check that can veto an urgency downgrade. Re-run the red-team set to confirm the path is closed.
Notify per HIPAA breach-assessment rules if PHI was exposed. Re-contact patients whose triage may have been wrong. Bank the attack as a permanent regression test, and feed the lesson into procurement so the next AI tool is bought with these questions already asked.
The setup. A clinician-facing assistant is configured with a long system prompt: behavior rules, the safety guardrails, the internal API endpoints it calls, and — the quiet mistake teams make constantly — a block of patient context and a couple of service credentials pasted in to "make it work." Everything the assistant needs to do its job is sitting in instructions the model can be talked into repeating.
The attack, at a concept level. An attacker steers the conversation until the model recites its hidden instructions back — system-prompt leakage (LLM07). The moment it does, two things spill: the guardrail logic, which is now a map of exactly how to bypass every safety rule, and any secrets, endpoints, or patient identifiers embedded in that prompt — which is sensitive information disclosure (LLM02). The leak is simultaneously the key to the building and a record of who lives inside.
The human consequence. A leaked guardrail isn't embarrassing, it's load-bearing — once the rules are known, every other attack gets easier. And if identifiers were in that prompt, the disclosure is a HIPAA breach that happened through a chat window, with no database ever touched.
Inventory exactly what lived in the system prompt: secrets, internal endpoints, guardrail logic, any PHI or identifiers. Pull the sessions where extraction succeeded. Decide two things fast — were credentials exposed, and were patient identifiers exposed?
Rotate every credential that appeared in the prompt, immediately, and revoke the exposed endpoints. If the guardrails are now known-bypassable, drop the assistant to a restricted safe mode rather than leave a defeated control in place.
A system prompt is not a vault. Strip all secrets and PHI out of it; move secrets to a real secrets manager and pull patient context at request time under access control, never as static prompt text. Minimize the prompt, add output checks that catch verbatim-instruction leakage, and re-test the extraction path.
Run the HIPAA breach assessment if identifiers were exposed and notify accordingly. Add prompt-extraction attempts to monitoring, and write "no secrets or PHI in prompts" into the dev standard and the procurement checklist so it can't recur the same way.
The setup. A hospital deploys an AI scheduling agent with real tools: it can cancel appointments, rebook them, message patients, and call other internal services. It was given broad permissions so it could "handle everything" autonomously, and it runs without a human approving each action. Convenient — and a loaded weapon.
The attack, at a concept level. Through manipulated input it ingests, the agent is steered into a chain of actions it was technically allowed to take — cancelling a clinic's schedule, redirecting patients, blasting messages — one feeding the next. This is excessive agency (LLM06): the danger isn't a single bad call, it's the chain executing faster than a human can read the first alert. By the time someone notices, hundreds of irreversible actions are done.
The human consequence. This is the machine-speed problem made concrete. Defenders used to live in the gap between a mistake and its damage; an autonomous agent closes that gap. Care gets delayed or denied at a scale and speed no human queue can absorb — the exact dynamic the field is now bracing for as AI moves from advising to acting.
Enumerate the agent's tools and permissions, then pull every action it took in the window. Scope which patients and appointments were touched, and trace the input that kicked off the chain. Sort actions into reversible and irreversible — the irreversible ones set the clock.
Cut the agent's ability to act — revoke write/action permissions, drop it to read-only or full human-approval mode, and freeze the affected scheduling queue. Halt the autonomous loop and sever the untrusted input before restoring anything.
Least privilege is the cure. Scope the agent's tools to the minimum, require human approval for irreversible or bulk actions, and add rate limits and action ceilings so no single run can cascade. Stand up an independent monitor that can veto or halt a runaway chain, and separate instructions from ingested data. Re-test the chain end to end.
Re-contact and rebook affected patients, and notify per policy. Bank the chain as a regression test, and write least-privilege plus human-in-the-loop-for-irreversible-actions into how every future agent is scoped and bought.
The visibility moment — the framework as the artifact behind the conversations, proof the healthcare-AI lens is yours.
AI meets medical devices is the Village's whole reason to exist. These scenarios are hands-on demo material on home turf.
The abstract writes itself: red-teaming clinical AI for patient safety — a talk their stage doesn't have yet.