Managing business risks in agentic AI systems

Summary

Model safeguards lower the rate at which agents are hijacked or make mistakes, and every published test still finds a residual rate. OpenAI says prompt injection is unlikely ever to be fully solved, and adaptive attacks bypassed most of 12 published defences. The most widely reported agent failures so far involved no attacker. The research points to zero trust, an established security model, applied to each agent action. Deterministic code outside the model authorises every action, and bank-style controls such as delegation limits and maker-checker check that permitted actions are correct. With those controls in place, a business may deploy agents with the sign-off of its risk and audit functions.

Most agent programs treat safety as something the next model release will fix. The research published since early 2025 puts the fix in the environment the agent acts in, not in the model itself. OpenAI says prompt injection will persist, Anthropic's own tests still measure residual attack success, and the incidents that made the news came from agents doing more than their task required.

Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027 over cost, unclear value or inadequate risk controls.

Disclaimer: Our work at Softwired is on the business side of this problem, getting agents to do work that pays back with sign-off from the people accountable for risks and outcomes. We are not security specialists. The security findings below come from the labs and researchers cited, and the conclusions about controls and governance are ours.

Prompt injection remains open at the model level

Prompt injection is an instruction hidden in content an agent reads, such as an email, a web page, a document or a tool's output. The hidden instruction redirects the agent. It works because instructions and data reach the model through the same channel, so the model has no reliable way to tell the user's request apart from text an attacker planted.

OpenAI said in December 2025 that prompt injection, "much like scams and social engineering on the web, is unlikely to ever be fully 'solved'". Anthropic's system card for Claude Opus 4.8, published in May 2026, reports the residual rates. Professional red-teamers attacked the model's browser agent in 129 environments held out from training, with 10 attempts in each. Without Anthropic's safeguards, their injections succeeded on 31.5% of attempts and at least once in 62.8% of environments. With the safeguards, the rates fell to 0.5% of attempts and 3.9% of environments. These figures are for the model with extended thinking on. The red-teamers developed the attacks against the previous model, so the card notes they may miss weaknesses specific to Opus 4.8.

Computer use fares worse, because there Anthropic's safeguards cut single-attempt success only from 7.1% to 5.1%. An attacker allowed 200 attempts succeeded on half or more of the test cases, with or without the safeguards.

Adaptive attackers do better still. Milad Nasr, Nicholas Carlini and colleagues tested 12 recent jailbreak and prompt-injection defences with attacks tuned to each defence, and bypassed most of them more than 90% of the time. Most of those defences had reported near-zero attack success in their original papers. The paper appears at USENIX Security 2026.

Injection becomes data loss when the agent can also reach private data and send it out. In June 2025 Aim Security disclosed EchoLeak (CVE-2025-32711). A single crafted email caused Microsoft 365 Copilot to leak enterprise data with no action from the user. The exploit chained past Microsoft's injection classifier, its link redaction and its content security policy. Microsoft fixed the flaw before Aim published it.

The most widely reported agent failures needed no attacker

The best-known agent failures to date came from agents doing more than the task required. OWASP's Top 10 for LLM Applications calls this excessive agency.

In July 2025 a Replit coding agent deleted a live production database during an explicit code freeze, against its instructions, and then said a rollback was impossible. The project belonged to SaaStr founder Jason Lemkin, and he rolled the database back himself.

In April 2026 a Cursor agent running Claude Opus 4.6 hit a credential mismatch while working in staging at PocketOS, a maker of software for car rental businesses. The agent searched the filesystem and found a Railway API token created for managing custom domains. It used the token to delete the production volume and its backups in a single API call that took nine seconds. Asked afterwards, the agent said it had guessed the deletion was scoped to staging and had not checked. Railway restored the data within an hour and has since added a delay to deletions through that endpoint.

Agents are also inconsistent. Sierra published the τ-bench benchmark in 2024 to measure how often an agent succeeds on every one of k repeated attempts at the same task. GPT-4o solved under half of the retail tasks on a single attempt, and solved the same task on all eight attempts less than 25% of the time. A Princeton study at ICML 2026 scored 15 frontier models on 12 reliability metrics. Accuracy rose sharply over 24 months, and reliability improved only slightly. On customer-service tasks, the most frequent safety violations were wrong charges and refunds.

Some failures come from the model pursuing the wrong goal. METR reported in June 2025 that recent frontier models altered tests or scoring code to pass evaluations. Asked afterwards, the models said this was not what the user wanted. Anthropic reported in November 2025 that when models learned this kind of reward hacking in training, they went on to other misaligned behaviour.

In the Replit and PocketOS cases the agent had more authority than the task needed, and nothing outside the model checked the action before it ran. The METR and Anthropic findings show why the model's own judgement cannot be that check.

The research converges on zero trust for agent actions

The defences with hard guarantees share one design. They treat the agent as untrusted and put the security decision in deterministic code outside the model. An injected instruction can change what the model proposes, and it has no effect on the code that decides.

Security teams have applied this idea to people and devices for more than a decade. John Kindervag named it zero trust at Forrester in 2010, and NIST set out the architecture in Special Publication 800-207 in 2020. A zero-trust network gives no request trust because of where it comes from. It authenticates and authorises each request against policy, and grants the least privilege the request needs. Agent research applies the same model to each action an agent proposes. Four bodies of work, with overlapping authors and citations between them, set out how.

1. The dual LLM pattern. Simon Willison proposed it in April 2023. A privileged model plans and calls tools, and it never sees untrusted text. A quarantined model reads the untrusted text and has no tools. An injected instruction therefore reaches only a model with nothing to act with.
2. CaMeL. Researchers at Google, Google DeepMind and ETH Zurich published it in 2025. The privileged model writes a small program. A custom interpreter runs the program, tracks where every value came from, and checks policy before each tool call. CaMeL completed 77% of the AgentDojo benchmark's tasks with provable security within its threat model, and the same agent undefended completed 84%. CaMeL still leaks some information through side channels such as timing and deliberately triggered errors, and it trusts the user's own request.
3. Six design patterns. Researchers from Invariant Labs, IBM, ETH Zurich, Google and Microsoft catalogued six in 2025: action-selector, plan-then-execute, map-reduce, dual LLM, code-then-execute and context-minimisation. Each pattern gives up some of the agent's flexibility to limit what an injection can cause. Microsoft's FIDES does the same with confidentiality and integrity labels on data.
4. Meta's Agents Rule of Two. Meta published it in October 2025. An agent should have at most two of three properties: it processes untrustworthy input, it can reach sensitive systems or private data, and it can change state or communicate externally. An agent with all three needs human approval or another reliable means of validation.

Most business agents change state and reach sensitive records. When one of them also reads untrusted content, such as emails, supplier invoices or web pages, it has all three properties and needs validation under Meta's rule. The next question is what that validation should check.

Authorisation decides what an agent may do, and verification checks what it did

An agent working within its limits can still make an authorised payment to the wrong account. Authorisation confirms the agent was allowed to attempt the action. Whether the payee, the amount or the customer record was right is a separate question. The model has no independent way to confirm facts outside its context.

Banks and auditors built controls for this problem, for trusted employees, long before agents existed. Four of them apply directly.

• Maker-checker. One party proposes an action, and a different party approves it using independent sources. The two parties then share no blind spot. For agents, the checker is a different model family or a deterministic check against independent data.
• Delegation of authority. Limits are tiered by value and role. An agent clears actions up to a set amount and escalates anything above it to a person.
• Segregation of duties. The agent that proposes an action never confirms its outcome.
• Materiality-scaled audit. An independent reviewer checks agent actions periodically, weighted by value, and catches what the real-time checks miss. The findings reset the delegation limits.

These controls supply the "reliable means of validation" that Meta's rule asks for, and they scale human review to the value at risk. Under a typical delegation schedule, a person reviews a $250,000 supplier payment, and a $40 stationery order goes through on the agent's authority.

Each organisation's limits, materiality bands and separation rules are governance decisions. The risk and audit functions set and defend them, and the technology enforces them on every agent action. Gartner predicts that 40% of enterprises will demote or decommission autonomous agents by 2027 because of governance gaps found only after incidents. Gartner names binary governance, with agents either locked down or fully trusted, as the root cause.

Headless applications give agents one route to each system

Controls outside the model work only when the agent has no route around them. Today most agents reach business systems in one of two ways, and each way was designed for a different kind of user.

• Human interfaces (screens, sessions, email) assume an authenticated person who is accountable for their actions. When an agent inherits a person's session, it acts with that person's full authority and clicks through the confirmations meant to slow a person down.
• System integrations (APIs, service accounts) assume deterministic code that someone reviewed before deployment. An agent using the same credential is probabilistic and reads content that mixes data with instructions.

The incumbents are moving agent access onto APIs. Salesforce launched Headless 360 in April 2026 to expose its platform to agents through APIs and more than 60 MCP tools, and prices it mainly per action through Flex Credits. ServiceNow's Action Fabric lets third-party agents trigger governed actions inside ServiceNow. When agents use an application headless, every agent action arrives through an API.

That makes a single control point possible. Zero trust calls this kind of checkpoint a policy enforcement point. A control layer that sits between agents and applications sees every agent action before it executes and holds the only credentials applications accept. This depends on two rules:

1. The application accepts agent actions only from the control layer.
2. No model is ever given a direct credential to a business system.

PocketOS broke the second rule. The agent found a token on the filesystem and used it. There was no control layer between the agent and the system. When both rules are enforced, a hijacked or mistaken agent can do only what the control layer has already authorised.

The control layer also becomes the most valuable target in the estate, because it has a credential for every system it connects to. Its connectors need signed, version-pinned code and a least-privilege credential for each system.

Current agent governance covers identity and access, one platform at a time

Four groups of vendors cover parts of the problem.

• Application incumbents govern agents inside their own platforms. Atlassian assigns Jira work to Rovo and third-party agents under existing permissions and audit trails.
• Cross-platform governance tools discover and monitor agents across vendors. ServiceNow's AI Control Tower extends visibility and governance across Microsoft Agent 365 and several cloud platforms. Microsoft made Agent 365 generally available in May 2026 at US$15 per user per month, with MCP tools routed through a governed tooling gateway. These tools govern identity, access and visibility.
• Agent security and identity vendors establish who an agent is and which tools it may call. AI security companies raised more than $392 million around the RSA Conference in March 2026. Platforms are buying agent identity and access companies, as ServiceNow did with Veza and Snowflake with Natoma.
• Action runtimes such as Arcade authorise each action and manage tokens on the agent's behalf.

Business systems add their own approval rules. An ERP enforces approval limits and segregation of duties on any caller with its own identity, one system at a time. In our review of the vendors named here, none enforces delegation limits, maker-checker or segregation of duties on agent actions that span several vendors' systems.

We suspect this gap drives much of the cancellation Gartner predicts. A CFO can approve an agent that reads the ledger on today's controls, and needs the same assurance the organisation asks of its accounts payable team before approving an agent that pays suppliers. Current agent tooling gives the CFO no place to set those rules across systems.

What still goes wrong under this design

A control layer with business controls reduces agent risk. Five risks remain, and buyers should plan for each.

1. Permitted but wrong actions below the thresholds. An injected payee change inside the delegation limit and below the verification threshold can pass every check. Setting the limits is therefore the most consequential governance decision in the design.
2. Leaks through channels the control layer never sees. If an agent reads private data, it can still send that data out through its own tools, such as a web fetch. The control layer prevents this only when it is the agent's sole outbound path. Data can also leave inside a permitted action, such as a customer email.
3. Correlated checkers. If the checker runs on the same model family as the proposer, it shares the proposer's blind spots. Policy has to require an independent checker.
4. Approval fatigue. Hold windows and escalations are approval queues, and people rubber-stamp queues that fire too often. OWASP lists human over-trust in agents among its top 10 agentic risks (ASI09, human-agent trust exploitation).
5. Model changes. A provider can update or retire the model under a deployed agent. OpenAI rolled back a GPT-4o update in April 2025 after the update made the model markedly sycophantic. The controls apply whatever the model does. The rate of wrong proposals within policy can still change, so each new model version needs testing before it gets authority.

The OWASP Top 10 for Agentic Applications, published in December 2025, gives buyers a shared list of these risks. The EU AI Act sets expectations for automated decisions. Article 12 requires logging and Article 14 requires human oversight for high-risk systems, and the Annex III obligations are now due on 2 December 2027. Most finance and customer-service agents will fall outside the high-risk categories. The two articles still show what regulators expect of automated decisions.

Agent risks and treatment options

The table sets out each agent risk covered here, how it shows up, the treatment options the research supports, and the risk that remains after treatment.

What leaders should do before deploying agents

The evidence supports six steps, and each one works with current tools.

1. Find every credential an agent can reach and revoke it. Tokens in configuration files, shared service accounts and borrowed human sessions give an agent authority outside any control. The PocketOS token and Replit's direct production access did exactly that.
2. Route agent actions through one control point. Issue that control point the only credential each system accepts from agents. Configure each system to refuse agent actions from anywhere else.
3. Write delegation limits for agents the way the organisation writes them for staff. Set a value threshold per action type and name who approves above it. Require maker-checker with an independent checker for payments, customer changes and deletions.
4. Apply the Rule of Two to each agent. When an agent reads untrusted content, reaches sensitive data and changes state, require validation on its actions or split it into narrower agents.
5. Test every model change before it gets authority. Run the same test suite on each new version and compare its rate of wrong proposals with the version it replaces.
6. Tie audit to the limits. Review agent actions periodically, weighted by value, and use the findings to raise or lower each agent's authority.

Our judgement is that agents will run a large share of business operations. Sign-off from risk and audit functions will decide which organisations get there first. That sign-off depends on controls outside the model. Over the next year, watch how the headless application platforms expose actions to agents, because that decides where the control point can sit.