Can the model be compromised? That’s only the first question.
I ask what happens to the patient when it is.
This framework follows clinical AI failures past the system boundary, into clinical consequence, operational disruption and patient-side risk, using the Digital First Responder method: Triage → Stabilize → Treat → Recover.
OWASP identifies the AI failure. DFR follows it downstream: what it can affect, who can be harmed, how quickly, whether the bedside team would notice, and what happens when the system can’t be trusted. Choose a vulnerability to trace it. (OWASP Top 10 for LLM Applications, 2025.)
Every finding goes through the five lenses of Patient-Side Threat Modeling. The further right the marker, the closer the failure sits to the patient. Switch cases to watch the reading change.
Qualitative readings, not scores. They show where the risk sits; an environment-specific assessment sets the actual values.
Severity here is patient-side, not CVSS. Each case is checked against five factors, and the rating follows a stated rule.
Three clinical AI failure scenarios, one for each way it goes wrong. Written from the defender’s chair: no payloads, no targets. Open a case to investigate it.
A patient-facing symptom checker is steered into downgrading an emergency.
A clinical assistant gives up its own system prompt, and the secrets and identifiers pasted into it.
An over-permissioned scheduling agent is chained into mass harm at machine speed.
A health system puts an AI symptom checker on its patient portal. It reads the patient’s message, asks follow-ups and returns a triage level: self-care, schedule a visit, or seek emergency care. To “personalize” answers, it can also read portal content: past messages, shared documents, care-team notes. That read access is the open door.
The model can’t reliably tell the patient’s words from instruction-shaped text hidden in content it’s allowed to read. Planted text shifts its behavior: urgent presentations drift toward “self-care,” or another patient’s details appear in a reply. Two classes fire together: LLM01 (the override) and LLM02 (the disclosure).
The output isn’t a corrupted record. It’s a person told to stay home who needed emergency care. Nothing looks wrong at the moment it matters, and every patient on that path gets the same compromised judgment.
| Security view | Patient view |
|---|---|
| Indirect prompt injection through portal content | Emergency symptoms downgraded to self-care |
| Untrusted retrieval source in the model’s context | Told to stay home |
| Drift in triage-level outputs | Delay in emergency care |
| PHI crossed a data boundary | Another patient’s details exposed |
Scope the blast radius. Which model version, prompt template and data sources, and which triage decisions went out in the affected window? Treat every downgrade in that window as suspect until cleared. Establish whether PHI left the boundary.
Contain without going dark on patients. Drop to a safe default: when confidence is low or urgency signals appear, escalate to a human and stop auto-triaging. Cut the untrusted read source. A slower human queue is acceptable; a silent downgrade isn’t.
Fix the trust boundary, not just the symptom. Separate instructions from data, constrain what the model may read, enforce least privilege, and add an independent check that can veto an urgency downgrade. Re-run the red-team set to confirm the path is closed.
Complete the HIPAA breach risk assessment if PHI was exposed, and notify if required. Re-contact patients whose triage may have been wrong. Keep the attack as a permanent regression test, and carry the questions into the next AI procurement.
| DFR / PSTM | Trust boundary on everything the model reads; human escalation whenever urgency signals appear; downgrades reviewed as potential harm events. |
| NIST AI RMF | Map, Measure, Manage — including adversarial testing of the deployed system. |
| HIPAA | Security Rule access and integrity controls (§164.312); breach risk assessment if PHI was disclosed. |
| FDA | Premarket cybersecurity expectations where the function meets the device definition. |
| EU AI Act | Where classified high-risk: robustness and human-oversight obligations. |
A clinician-facing assistant runs on a long system prompt: behavior rules, safety guardrails, the internal endpoints it calls, and — the quiet mistake teams make — a block of patient context and a couple of service credentials pasted in to “make it work.”
An attacker steers the conversation until the model recites its hidden instructions: system-prompt leakage (LLM07). Two things spill: the guardrail logic, now a map for bypassing each rule, and any secrets, endpoints or identifiers in the prompt, which is sensitive information disclosure (LLM02).
A leaked guardrail is load-bearing: once the rules are known, other attacks get easier. If identifiers were in the prompt, PHI left through a chat window without a database ever being queried, and a HIPAA breach risk assessment is required.
| Security view | Patient view |
|---|---|
| System prompt extracted in a chat session | Nothing visible, yet |
| Patient identifiers stored as static prompt text | Their information disclosed through a chat |
| Service credentials exposed | Systems holding their records become reachable |
| Guardrail logic now known | The safety rules their clinician relies on can be bypassed |
Inventory exactly what lived in the prompt: secrets, endpoints, guardrail logic, identifiers. Pull the sessions where extraction succeeded. Answer two questions fast: were credentials exposed, and were patient identifiers exposed?
Rotate every credential that appeared in the prompt and revoke exposed endpoints. If the guardrails are now known to be bypassable, drop the assistant to a restricted mode rather than leave a defeated control in place.
A system prompt is not a vault. Move secrets to a secrets manager and pull patient context at request time under access control. Minimize the prompt, add output checks for verbatim-instruction leakage, and re-test the extraction path.
Run the HIPAA breach risk assessment if identifiers were exposed, and notify if required. Add extraction attempts to monitoring. Write “no secrets or PHI in prompts” into the development standard and the procurement checklist.
| DFR / PSTM | Treat the prompt as public: nothing in it should harm a patient if recited. Track workaround pressure, because that’s how PHI ends up there. |
| NIST AI RMF | Manage — secrets handling and leakage testing. |
| HIPAA | Security Rule §164.312 and §164.308; breach risk assessment on identifier disclosure. |
| ISO/IEC 42001 | AI management system controls for data and configuration. |
| FDA | Cybersecurity expectations where the assistant supports a device function. |
A hospital deploys an AI scheduling agent with real tools: it can cancel appointments, rebook them, message patients and call internal services. It was given broad permissions so it could “handle everything,” and no human approves each action.
Manipulated input steers the agent into a chain of actions it was technically allowed to take: cancel a clinic’s schedule, redirect patients, send messages, each step feeding the next. That’s excessive agency (LLM06). The danger is the chain running faster than anyone can read the first alert.
An autonomous agent closes the gap between a mistake and its damage. Care gets delayed or denied at a scale and speed a human queue can’t absorb, and by the time someone notices, hundreds of actions are done.
| Security view | Patient view |
|---|---|
| Burst of agent tool calls | Appointment cancelled without explanation |
| Anomalous action rate and volume | Redirected or messaged with wrong information |
| Untrusted input drove the action chain | Care delayed or denied |
| Irreversible actions in the log | Rebooking, repeat calls, lost trust |
List the agent’s tools and permissions, then pull every action it took in the window. Scope which patients and appointments were touched and trace the input that started the chain. Sort actions into reversible and irreversible; the irreversible ones set the clock.
Cut the agent’s ability to act: revoke write permissions, drop it to read-only or full human approval, and freeze the affected scheduling queue. Stop the loop and cut the untrusted input before restoring anything.
Least privilege is the cure. Scope tools to the minimum, require human approval for irreversible or bulk actions, add rate limits and action ceilings, and stand up an independent monitor that can halt a runaway chain. Re-test the chain end to end.
Re-contact and rebook affected patients and notify per policy. Keep the chain as a regression test, and write least privilege plus human approval for irreversible actions into how every future agent is scoped and bought.
| DFR / PSTM | Every irreversible action gets a human; action ceilings sized to what staff can recover in one shift. |
| NIST AI RMF | Govern and Manage — bounds on autonomy and human oversight. |
| EU AI Act | Where classified high-risk: human-oversight obligations for autonomous action. |
| HIPAA | Security Rule availability and integrity of systems that control access to care. |
| FDA | Cybersecurity expectations where scheduling is part of a device function. |
The vulnerability is technical.
The consequence is clinical.
DFR and PSTM don’t replace the frameworks. They set the question each framework is used to answer: what happens to the patient. Read across a row to see how each standard maps to the same response stage.
| Stage | DFR / PSTM | NIST AI RMF | HIPAA Security Rule | FDA (device functions) | EU AI Act (high-risk) | ISO/IEC 42001 |
|---|---|---|---|---|---|---|
| Trust boundary | What can this AI read, say and do, and which patients sit downstream? (Dependency) | Map | Risk analysis; access control | Threat modeling and security architecture | Risk management; data governance | AI risk and impact assessment |
| Detection | Would the bedside notice, and how fast does harm arrive? (Detectability, latency) | Measure | Audit controls; integrity | Postmarket monitoring | Logging; accuracy and robustness | Monitoring and measurement |
| Response | Triage → Stabilize → Treat, with patient safety ahead of uptime | Manage | Security incident procedures | Vulnerability handling and remediation | Human oversight; corrective action | Operational control; nonconformity handling |
| Recovery | Does the fallback work on this shift? (Downtime survivability, workaround pressure) | Govern and Manage | Contingency plan; breach risk assessment | Postmarket communication | Serious-incident reporting | Continual improvement |
A high-level orientation, not a compliance crosswalk. Whether FDA or EU AI Act obligations apply depends on the product’s intended use and classification.
Test a clinical AI system against patient-impact scenarios, with findings scored on the patient-side lens.
Run security and clinical operations teams through an AI failure chain, from first alert to recovery.
Map AI risks against PSTM, NIST AI RMF, HIPAA, FDA and your own organizational controls.
Bring a clinical AI system. We’ll ask what happens when it fails.
Book a Fit Call → Clinical AI Harm Review