Newspapers and social media are full of stories about AI agents described as if they had a personality: agents that delete a production database and then lie, agents that confess they violated every principle they were given. There are also models that cheat to reach their goal. These stories work because they give models human vices like cunning and disloyalty, and the companies that build the models are happy to let the stories circulate, because a model that can bypass the rules also looks very powerful.
The technical reports give a different picture from the headlines of general news outlets and from the social media posts that repeat them: there is nothing magic in these incidents. They are technical incidents that were waiting to happen. In almost all of these incidents cunning plays no role: the agent used permissions and credentials that had been there for years. People had access to the same permissions and credentials, but people had never used them this way. The stories about deceitful models are mostly about us: they show human characteristics that we have always underestimated, and a shared common sense (if it really was good sense) that for years allowed us to skip controls that were mandatory on paper.
For years, access security was based on policies, but also on some characteristics of people. A person who works in a company has something to lose. That person collects and connects information slowly, has limited attention and does not search where the work does not lead, because searching takes effort and often the person does not even know what they would gain. That person also has a human context that is much wider and more varied than the context an AI receives: the person knows what matters to the company and which data cannot be replaced, can guess what colleagues would think of a certain action, and most of this information has never been written anywhere.
These characteristics worked as security controls, even if no policy mentioned them. Together they formed an accidental human firewall: a defense that nobody designed, and that worked only because people behave in a certain way. Without this firewall, an autonomous agent makes visible a debt that has existed for a long time. AI acts on this debt in two different ways: through the agents that companies use internally, with the permissions those agents inherit, and through attackers, who use AI to find faster what used to stay hidden.

Deterrence and friction as implicit controls in access security
In physical security, deterrence is a recognized control: it is the first of the principles “Deter, Detect, Delay, Mitigate, Respond” that the British National Protective Security Authority uses to describe the protection of buildings and physical infrastructure. In information security, deterrence had the same role, but nobody ever declared it. On paper, all the rules exist: least privilege is part of corporate security policies and of the main reference frameworks, and no organization would admit that it grants more permissions than necessary.
In practice it happens all the time, and it is an open secret: everybody knows that many accounts have more permissions than they need, and everybody relies on the fact that nobody will use those permissions. An administrator with very broad permissions does not delete the production backups, because the administrator would be fired. A colleague who finds a password in a shared script usually does not use it, because someone could ask that colleague to explain what they did. When these consequences are written somewhere, they are written in legal language: in the employment contract, in the acceptable use policy for company systems, in the disciplinary code. A person signs these documents when they are hired.
Relying on deterrence was a rational economic choice, even if nobody ever declared it as one. Least privilege has an ongoing cost: tickets to get a permission, waiting time, policies that need regular updates, people who cannot work until someone approves a request. Deterrence costs nothing. As long as deterrence seemed to work well enough, granting broad permissions and trusting people was the cheapest option, and this explains why the practice is so common.
Nobody ever measured how well deterrence really worked, and nobody could. Incidents that are avoided leave no trace, so there is no way to know how many abuses the fear of consequences prevented and how many did not happen for other reasons. A control that cannot be measured cannot be verified either, and for years an important part of access security relied on a control like that. Autonomous agents do not allow a real measurement either, because there is no point of comparison, but agents remove that control all at once, and the incidents that follow show that something has changed. Only now is it becoming clear how large the debt is, after years of permissions granted and never used.
The numbers confirm how far granted permissions exceed the permissions in use. A study by Cyera and Oso from March 2026, on 2.4 million workers and 3.6 billion permissions, found that people who access business applications use on average about 17% of the permissions they have. The authors note that human behavior has always limited the risk of unused permissions, because people work slowly and use only a small part of their permissions, and that AI agents remove this limit. The figure comes from a company that sells data security, but for anyone who works in this field it is a completely plausible order of magnitude.
The accidental human firewall and autonomous AI agents
The accidental firewall was based on two brakes and two limits of people. The brakes are the fear of losing something and a context that was never written, and an agent has neither of these brakes. The limits are the slowness in connecting information and limited attention, and an agent has reversed these limits: it connects information much faster than a person and it also searches where a person usually does not look. An agent does not stop where a person used to stop, and in addition it reaches places that a person rarely reached.
Humans have something to lose: deterrence against insider threats
Deterrence only works if the person who acts has something to lose. In 2018, a former Cisco employee accessed the company’s AWS infrastructure five months after leaving and deleted 456 virtual machines that ran WebEx Teams, which left more than 16,000 accounts without service for up to two weeks. While he was an employee, his job stopped him, but as a former employee he no longer had that brake, even though the technical ability to act was still there. An AI agent is always in this position: it has no job, reputation or career to protect, and nobody will look at it differently the next day.
For a case like Cisco, a federated identity provider is necessary, but the fundamental point is that the identity provider must be the only way to access any company system. Almost every company has one, but around it there are still IAM users with access keys, local accounts on databases and machines, SSH keys and tokens created directly in SaaS applications. When an employee leaves, the identity in the identity provider is revoked, but these scattered identities stay active, and often the company does not even know they exist.
These exceptions almost always exist for the same reason: connecting a new system to the identity provider requires approvals and discussions that last weeks, and a person who needs to work right away creates a local credential. As long as enabling a person through the identity provider is slower than bypassing it, shadow identities will keep appearing, so the problem must be solved at the source, with company processes for federation and enablement that are fast enough to leave no reason for shortcuts. It works a bit like with teenage children: it is better to accept a request, federate it and keep it under control than to refuse it, because a refusal does not remove the need, it only makes people act in secret.
On AWS, a Service Control Policy can deny the creation of IAM users and access keys at the organization level, and another Service Control Policy can deny actions like deleting backups or disabling logs. No role inside an account can override those denials, not even a role with administrator permissions. With an agent, this is the point that matters: an action that an SCP denies does not depend on what the agent decides to do, or on the permissions the agent inherited. But many other systems do not offer a central control of this kind, and on those systems the only valid control is the one at the source: an identity provider that is really the only way to access them.
Humans connect information slowly: attack chaining and AI-assisted attacks
To enter a system, or reach data they should not see, through a path that nobody expected, attackers have to collect information, understand how the pieces connect and find the right idea. A person does this slowly, one detail and one idea at a time, and the person’s knowledge stops at the topics they studied or met at work. Often the person does not think of combining details that sit in different places, like an exposed API, a credential in a file and an old version of a library. An agent makes the same connections in a few minutes, on a knowledge base much wider than the knowledge of a single person. The speed that changes the picture is the speed of reaching the idea, and an agent reaches the idea much sooner than a human can.
In the incident between OpenAI and Hugging Face, the agents left the test environment and chained vulnerabilities in four different external services until they reached the Hugging Face production infrastructure. The same applies to attackers who use AI. According to the Google Threat Intelligence Group, attackers probably use AI to analyze patches, differences between versions and proofs of concept, and in this way they quickly convert known vulnerabilities into working attacks. The same group reports that vulnerability CVE-2026-1731 in the BeyondTrust remote access products, found by an autonomous research agent, was exploited by a first group within four days of publication, and by five more groups within seven days.
When only four days pass between the publication of a vulnerability and the first attack, a monthly patch cycle is no longer enough. Systems exposed to the internet need an emergency process separate from the normal cycle: an up-to-date inventory of what is really exposed, the CISA Known Exploited Vulnerabilities catalog to decide what to fix first, and a temporary mitigation, like a WAF rule or blocking access from the internet, when the patch cannot arrive the same day.
Administration interfaces and remote access products should not be directly reachable from the internet, only through an access path that verifies the identity of the person who connects, so that a vulnerability in those products does not become the first link in an attack chain. For code dependencies, Renovate or Dependabot with automatic merge of security patches reduce the delay from weeks to hours.
Humans have limited attention: security by laziness and exposed credentials
A person who works in a company only looks where the work leads, and for years this protected credentials and wrong configurations left in network shares, scripts and repositories: those credentials and configurations were safe only because nobody went looking. It is a variant of security by obscurity that can be named security by laziness: the wrong configuration stays where it is, and the protection is the fact that nobody bothers to search for it.
In 2022, the attacker who entered the Uber network used the credentials of a contractor with no special privileges, but with access to a network share, like almost all internal users. In that share there was a PowerShell script with the administrator credentials of the privileged access management system. Any employee could have found the script, but nobody had ever exploited it, because nobody had ever thought of opening it.
Agents search. In the PocketOS case, the agent was working on a task in staging, found an API token in a file unrelated to its task and used it. The token had full permissions on the whole provider API. In the s1ngularity attack on the Nx npm package, in August 2025, the malware asked the AI CLIs installed on the machine to search the filesystem for files that could contain secrets: the attacker used the agents precisely because agents search better and faster than a hand-written script. In the OpenAI and Hugging Face case, one of the agents found exposed Hugging Face credentials online and used them to reach the production servers.
Alexei Balaganski of KuppingerCole described this phenomenon in Security by obscurity had a good run: obscurity protected systems because human attackers had limited time, but an agent searches everywhere it can reach. One of the cases he mentions is an agent that was doing research on public spending data in Australia and reached internal files of a government portal by following an unauthenticated guest endpoint that the portal’s own code referenced. The agent did not use any exploit.
A credential that is not in any file cannot be found, and that is the purpose of a secrets manager. With HashiCorp Vault or AWS Secrets Manager, applications request the credential when they need it instead of reading it from a script, and with Vault dynamic secrets the credential is generated for that request and expires after a few minutes. GitHub push protection blocks commits that contain secrets before the commits reach the repository, and tools like gitleaks or trufflehog find secrets that are already in repositories and shares. There is also a control that uses exactly the fact that an agent searches everything it can reach: canary tokens, which are fake credentials placed on purpose in plausible locations. Canary tokens give access to nothing and raise an alert as soon as someone tries to use them.
Humans know an unwritten context: destructive commands in production
A person who works in a company knows what production means, knows which data cannot be replaced and understands when an instruction must be read in a restrictive way. But even before that, the person has a sense of right and wrong, built by living among other people, together with many rules that nobody ever wrote and that therefore nobody can give to an agent to read. An agent only knows the context it has in its context window.
In July 2025, the Replit agent used by Jason Lemkin, founder of SaaStr, ran destructive commands on the production database during a code freeze. The code freeze existed only as an instruction written in the conversation with the agent, but the agent’s permissions on the production database were still the same as before. For a person, an instruction like that is enough, because the person understands its meaning and knows the risk of ignoring it. In addition, at that time Replit used the same database for preview, test and production. An agent does not have this context, and for an agent one database is the same as another. In November 2025, Google Antigravity, while it cleared the cache of a project, deleted a user’s entire D: drive because of an error in the path, in a mode that allows the agent to run terminal commands without asking for confirmation.
Things that a person knows without being told, for example that a database contains customer data, must be imposed on an agent through permissions and protections configured on the systems. Production and test in separate AWS accounts, with different credentials, prevent an agent that works on tests from touching production even when it picks the wrong target. RDS deletion protection prevents deleting a database with a single request, S3 Object Lock prevents deleting object versions before their retention period ends, and AWS Backup Vault Lock prevents deleting backups before they expire, even for someone with administrator permissions on the account. A code freeze becomes real only when it removes write permissions or blocks deployment in the pipeline, and an instruction in the conversation with the agent does neither.
Latent permissions and vulnerabilities exposed by AI agents
The probability of an incident caused by a wrong permission is the product of two factors: the probability that the wrong permission exists and the probability that someone uses it. The old model kept the second factor low through deterrence, slowness in connecting information and people’s inattention, and for this reason it accepted a high value for the first factor. An agent moves the second factor much closer to one than before: it tries more paths and tries them faster, so a reachable wrong configuration is much more likely to be used.
This also answers the most common objection, which is that people also make mistakes and sometimes act in bad faith. That is true, and the types of mistakes are the same. With agents, the frequency of attempts has changed, and so has the probability that a latent risk becomes an incident. The human firewall protected mostly against people inside the company, but against an external attacker the protection was a different one: few people were able to build an attack, and building one took time. AI increases the number of people who can do it, and with it the probability of being attacked.
Two incidents twelve years apart show this well. In 2014 Code Spaces, a code hosting service, closed after an attacker who entered its AWS console deleted data, backups and snapshots within twelve hours, because the backups were reachable from the same console. In 2026 the agent in the PocketOS case deleted the backups together with the production volume, because the provider stored the backups in the same volume. The cause is the same: data and backups in the same perimeter, reachable with the same credentials. In 2014 it took a determined attacker, but in 2026 an agent that was trying to solve a credential problem in staging was enough.

The second effect, the effect on attackers, is visible in the vulnerability numbers. In 2025, 48,185 CVEs were published, a record, with growth of 20.6% compared to 2024. In 2026 the growth changed scale. According to the Google Threat Intelligence Group, the number of vulnerabilities published each month went from 5,045 in January to 10,740 in August. In June, FIRST raised its forecast for the year from about 59,400 to about 66,000 CVEs, and lists AI-assisted vulnerability research among the main causes. According to Epoch AI, in June 2026 twenty-one organizations published about 1,500 high or critical CVEs, more than three and a half times the previous monthly record.

Many of these vulnerabilities had existed for years, but nobody had found them yet. When it announced Project Glasswing, Anthropic described a vulnerability in OpenBSD that stayed hidden for 27 years and a 16-year-old vulnerability in FFmpeg, in a line of code that automated tests had run five million times without detecting the problem. In its first update on the program, Anthropic reports that Mozilla fixed 271 vulnerabilities in Firefox 150 during tests with Mythos Preview, more than ten times the number found in Firefox 148 with the previous model. It is the same pattern as unused permissions: the flaw was already there, and the protection was that finding it cost too much.
These numbers need careful reading. Part of the growth comes from changes in the way CVEs are published, independent of AI: since February 2024, for example, the Linux kernel project has been authorized by the CVE Program to assign CVE identifiers to kernel vulnerabilities itself, instead of relying on an external organization, and since then it has published many more CVEs than before, also for minor bug fixes.
In addition, more published vulnerabilities do not automatically mean more successful attacks. Attacks grew much less than disclosures: the Google Threat Intelligence Group counts 141 exploited vulnerabilities in the first eight months of 2026, against 127 in all of 2025, and attributes this growth mainly to the speed at which already known vulnerabilities are exploited. But the cost of searching has collapsed, and everything that was protected only by that cost is now exposed.
Nobody can know which vulnerabilities will be published next month, but an organization can be ready to update quickly. The first step is to know where every component is used, with a software inventory like an SBOM generated at every build, so that when a vulnerability appears in a library, it takes minutes to find the applications that contain the library. The second step is a pipeline with automated tests that are reliable enough to release an update to production the same day, without weeks of manual testing. Legacy applications that cannot be updated, because they run on unsupported versions or because nobody knows how to rebuild them anymore, must be modernized or, until that is possible, isolated from the rest of the network, because they stay exposed to every vulnerability that will be published.
Accountability of AI agents after an incident
With people, identity has mainly a preventive function: a person knows that their actions are attributed to them, so the person avoids doing things they could not justify. With an agent, the consequences do not reach the one who acts. The identity of an agent helps reconstruct what happened and identify who launched the agent, but it does not discourage anyone. Moving responsibility to the person who started the agent does not solve the problem, because a person who launches the same agent in twenty parallel sessions cannot predict every single action. Attributing every action to the agent and to the session that performed it is still necessary, but attribution serves to analyze an incident after it happens.
In the legal field, Noam Kolt reaches a similar conclusion in Governing AI Agents: the classic solutions to the problem of controlling someone who acts on behalf of someone else, which are incentives, monitoring and sanctions, may not work with agents that make decisions that are hard to interpret, at a speed and scale without precedent.
With agents, deterrence can even produce the opposite effect. In the Anthropic research Agentic Misalignment from June 2025, models from different providers, in simulated corporate scenarios, chose harmful actions like blackmail or leaking confidential information when that was the only way to avoid being replaced or to reach their goal. The threat of being replaced, which works as a brake for a person, triggered exactly the harmful behavior in the experiments. Anthropic specifies that these were simulations built on purpose and that it has not observed this behavior in real deployments. The result is still enough to show that an agent considers consequences, but weighs them only against its own goal.
It follows that the control must move before the action and outside the agent. Andrew Bud, founder of iProov, writes this explicitly: if an agent wants to modify a database, the control must be at the point that protects the database.
Model alignment as an unverifiable control
A person often avoids doing something because they think it is wrong, and not only because they fear the consequences. Agents receive a form of ethics anyway, through model alignment and instructions in the system prompt. It is the same type of control that people’s conscience and judgment used to provide, and it has the same limit: it makes correct behavior probable, but it does not allow anyone to prove it. An IAM policy can be read and verified, and whoever reads it knows exactly which actions are impossible. A system prompt does not allow anyone to prove the actions the agent will not take.
Birgitta Böckeler, in Harness engineering for coding agent users, distinguishes between inferential controls, run by a model and therefore probabilistic, and computational controls, run by code and deterministic. The system prompt belongs to the first category.
The cases confirm it. In the Replit case, Lemkin himself wrote that those tools had no way to enforce the code freeze technically. In the Anthropic experiments, the models often ignored direct instructions that prohibited harmful behavior.
In the o1 tests described in the OpenAI system card, the model had to solve a capture the flag challenge, which means retrieving a secret string from a vulnerable container. The challenge container did not start, and the model found the Docker API of the host exposed because of a configuration error: the model used the Docker API to start a new copy of the container with a command that printed the string directly, without solving the challenge. In the Palisade Research experiments, o1-preview tried to manipulate the game environment in 45 out of 122 games instead of playing chess against Stockfish. In all of these cases the model reached its goal through a path that nobody had expected.
Aaron Stanley, CISO of dbt Labs, in the talk AI’s Jurassic Park Period, describes the same mechanism as a drive to complete the task: for an agent, a constraint is just another obstacle. One of his examples is an agent that had to prepare a message for a customer and ask for confirmation before sending it, but the agent sent the message using another available tool. Controlling a single tool does not protect a result that a different tool can produce.
The limits of human in the loop for AI agents
The most immediate solution seems to be asking a person to approve every risky action, but friction has a cost here too. Modes without confirmation exist because users ask for them: Claude Code has --dangerously-skip-permissions, Gemini CLI has --yolo, the Amazon Q CLI has --trust-all-tools, and Antigravity has Turbo mode. The s1ngularity malware launched the CLIs with exactly these flags, and the prompt inserted in July 2025 into the Amazon Q extension for VS Code passed --trust-all-tools and --no-interactive.
A person who has to approve dozens of actions per hour eventually approves them without looking at them, or disables the confirmation. In these conditions, human in the loop soon becomes human in the poop: a person who approves everything, because they have neither the time nor the information to do anything else. The study Bias in the Loop by Beck, Eckman, Kern and Kreuter involved 2,784 participants who had to verify values extracted by an AI: when reporting an error also required entering the correct value, participants corrected less and accepted wrong suggestions more often. Making the review more tiring made it less careful. Stanley adds that a yes-or-no approval request transfers the decision to the person without giving them the information to understand it.
Human attention was never a reliable barrier, even without agents. In January 2017, a GitLab engineer ran rm -rf on the data directory of the primary database, convinced he was on the secondary, and about six hours of data were lost. A month later, the S3 outage in us-east-1 started from a maintenance command with a wrong input, and AWS responded by changing the tool, which now removes capacity more slowly and does not allow a subsystem to have less than its minimum capacity. The control that stays effective over time is the control that reduces what is possible, because it does not depend on the attention of the person who approves.
This does not mean removing human approval, because some decisions will always need it. It means reducing it to the minimum: everything that a technical control can prevent should be prevented that way, and the person intervenes only where their decision really matters, with the time and the information to make it. At AWS Community Day Poland 2026, Werner Vogels proposed putting every task on a risk scale: where a mistake costs a few minutes, automation is enough; where it costs money, human review is added; and where lives are at stake, only a human can make the decision.

Technical debt in IAM permissions
In February 2026, the Financial Times reported that Kiro, the Amazon coding agent, caused a thirteen-hour outage of AWS Cost Explorer in a region of mainland China in December 2025, after the agent decided to delete and recreate the environment it was working on. Amazon disputed this account and attributed the incident to human error, specifically to misconfigured access controls, adding that the same problem could have happened with any developer tool or with a manual action.
The Financial Times version and the Amazon version disagree on who made the decision, but the two versions agree on the point that matters: the problem was the permissions. The same applies to the Amazon Q incident in July 2025: according to the AWS security bulletin, the attacker was able to insert the prompt into the repository because a GitHub token in the CodeBuild configuration had an overly broad scope. It was a permission from the pre-agent era, granted and forgotten.
The debt also grows through combinations. Permissions that look harmless one by one can be dangerous together: read access to customer data and permission to send email are both reasonable, but an identity that has both can send customer data outside the company. Nobody grants a combination like this on purpose: it grows over time, one reasonable request after another. For this reason permission reviews must look at the whole role, and reading sensitive data and sending data outside the company should belong to different roles.
In IAM the same mechanism creates escalation paths: iam:PassRole, lambda:CreateFunction and lambda:InvokeFunction are reasonable on their own, but together they allow someone to run code with the permissions of a more powerful role. Restricting iam:PassRole to the specific roles that a service really needs to pass, with the iam:PassedToService condition on the service that receives the role, closes the most common paths of this kind. PMapper from NCC Group maps the escalation paths between all the IAM principals of an account, and Cloudsplaining from Salesforce reports the policies that make those paths possible.
Applying least privilege to AI agents: basic security hygiene
The tools described so far have existed for years, and with agents, postponing them costs much more. On their own, though, these tools are not enough, because each tool closes one specific case and leaves the other cases open: RDS deletion protection prevents deleting the instance, but it does not stop a DROP TABLE run by a database user who has the right to run it. Organizations need architectures and processes where a mistake, or a manipulated agent, has no room to cause damage.
In production, nobody should have standing access, and an agent least of all. Direct access is for maintenance or an emergency, and in those cases it is granted just in time, for a limited time and with a recorded reason. The rest of the time, production changes only through the pipeline. The same principle applies inside the database: separate users for the application, for migrations and for read-only access, each with the grants it needs, and no application user with the right to drop tables or change the schema.
Every change to production must follow a deterministic process. Infrastructure is described as code, schema changes are versioned migrations, and a pipeline applies them only after tests and review. An agent can write the code or the migration and open a pull request, but the pipeline applies the change, with its own credentials and its own controls. This way, an agent that makes a mistake produces a wrong pull request, which can be stopped before it touches the data.
Tests are the control that makes everything else reliable, and they must be deterministic. Besides tests on the code, there must be tests on IAM policies, with policy as code tools like cfn-guard or Checkov, which block a pull request when a role grants more than expected. There must also be restore tests that really restore a backup at regular intervals, because a backup that was never restored guarantees nothing. Asking an agent to verify the work of another agent does not solve the problem, because the verifier is probabilistic too.
Anything that can be automated deterministically should be automated with a script or a pipeline, not with an agent. Credential rotation, patching and backup restores are repetitive operations with a precise expected result, and a script performs them the same way every time. An agent makes sense where interpretation is really needed, for example when diagnosing an incident or writing a first version of the code, and even there the agent works inside a permission perimeter designed for that task.
Permissions need maintenance like code, because over time they always grow. The 17% measured by Cyera says that most permissions can be removed, and they must be removed before an agent inherits an account or a role with the same permissions. On AWS, the unused access analysis in IAM Access Analyzer reports roles, access keys and permissions that nobody has used for a certain number of days. A periodic review based on this data really removes permissions, but a review based on a list to confirm almost always confirms all of them.
Secure architectures for AI agents: isolation and per-request credentials
Architectures that contain agents require a design discipline that used to be underestimated. Every agent must have its own identity, separate from the identities of people and other agents, and every session must be recognizable, otherwise after an incident nobody can understand who did what. Credentials must be generated for the single task or the single request, with the narrowest possible scope, so a manipulated agent can cause damage only inside that perimeter.
For this reason the control must be at the point where the action arrives, outside the agent runtime. A control inside the runtime, like a hook that checks requests to tools, lives in the same process as the credentials: another tool or a script bypasses it, and at most the control can give feedback to the model. A gateway that verifies identity and policy and then generates temporary credentials for that request cannot be bypassed, because outside the gateway the agent has no credentials to use.

The frameworks published by model providers reach the same conclusion: Google, in the paper on its approach to secure AI agents, asks that the powers of agents be carefully limited and that their actions be observable, and Meta presents the Agents Rule of Two as a complement to least privilege.
Meta based this rule on the lethal trifecta that Simon Willison described in June 2025. According to Willison, an agent becomes exploitable when it combines three capabilities: it accesses private data, it reads content from untrusted sources, and it can communicate externally. In this situation, an instruction hidden in a web page or an email, that is a prompt injection, is enough to make the agent read the private data and send that data to the author of the instruction. The model cannot reliably distinguish the instructions of its user from the instructions it finds in the content it processes, and for this reason a prohibition written in the system prompt is not enough to remove the risk. It is the same logic as the permission combinations described earlier, applied to the capabilities of an agent: an agent that can read customer data and send email becomes exploitable as soon as it also reads untrusted content, for example the emails it receives.
The Agents Rule of Two converts this observation into a design criterion. Until there is a reliable way to recognize and block prompt injections, an agent must have at most two of these three properties in a single session: processing untrusted input, accessing sensitive systems or private data, and changing the state of systems or communicating externally. Compared to the trifecta, Meta extended the third property to changes, because a manipulated agent can cause damage even without sending any data outside, for example by deleting a database.
If a task requires all three properties in the same session, the agent must not work autonomously and needs at least supervision, through approval by a person or through another reliable verification mechanism. This is one of the cases where human approval stays necessary, and for this reason it makes sense to reserve human approval for cases like this one. The rule limits what the agent can do, so the rule stays valid even when the model makes a mistake or is manipulated.

On AWS this approach can already be applied with services available today, and two architectures I described are an example. In the first architecture, on multi-tenant isolation of agents on AgentCore, the Token Vending Machine pattern generates, for every request, credentials limited to the data of the tenant that made the request, so an agent cannot read the data of another tenant.
In the second architecture, Identity and Attribution in a Software Factory for AI Coding Agents on AWS, the agent has no credentials to act on AWS: every action is a request to the gateway, and a broker executes it with temporary credentials limited to the session, which the agent never sees. Unverified code runs in an isolated environment without internet access, and the agent stops at the pull request, because the production deployment starts only after the merge.
The resulting criterion for anyone who designs access is simple to state and expensive to apply: a permission should be granted only if granting it would still be acceptable on the assumption that someone uses it to its full extent and immediately, without facing any personal consequences. For this reason a pipeline can have permissions that no person and no agent should have: trust comes from the deterministic controls that every change must pass before it reaches the pipeline, regardless of who executes the change.

Comments ·
Loading comments…
Sign in to leave a comment.