What a personal AI agent should never be allowed to do

You have given a personal agent access to your email, your calendar, your files and maybe your bank. It works well. Then one morning it reads something, decides it needs handling, and acts without asking.

What can it do that you cannot take back?

That is the question to design around. Most advice about agent safety talks about data and privacy. Both matter. But a leaked file is a different kind of problem from an action that cannot be reversed, and the second kind deserves a hard limit rather than a warning.

This post sorts actions by reversibility and gives you a list you can copy.

Sort every action into three buckets

Forget categories like "destructive" or "sensitive" for a moment. Ask how hard the action is to undo.

Reversible. Creating a draft. Adding a label. Writing a calendar event. Saving a note. If the agent gets it wrong you fix it in ten seconds. Allow these freely.

Expensive to reverse. Cancelling a subscription. Moving money between your own accounts. Archiving a thousand emails. You can recover, but it takes work and you may lose something on the way. Gate these behind approval.

Impossible to reverse. Sending a message to another person. Paying someone. Permanently deleting a file. Changing your recovery email. Deny these outright.

The third bucket is short, which is good news. You do not need a long rulebook. You need a small list of things that are never available, and the rest can be handled with judgement.

Start with account recovery, not money

Most people write the money rules first. Put account recovery above money instead.

If an agent moves $500 to the wrong place, you have a bad afternoon, a support ticket and often a reversal. If an agent changes the recovery email on your primary account, you may lose access to everything that account unlocks. Email, cloud storage, your password manager, every service that resets through that address.

There is no support ticket for that. You are locked out of your own life, and the thing you would normally use to prove who you are is the thing you lost.

So the top of the deny list is anything that changes who can get in:

• Changing a password
• Changing or removing two-factor authentication
• Changing a recovery email address or phone number
• Adding or approving an OAuth application
• Adding an SSH key, a deploy key or an API token
• Inviting a collaborator or sharing a folder with a new person

Those last three are the quiet ones. They do not look destructive. Nothing is deleted and no money moves. But each one creates a new way in that persists after the agent has finished, and you will not notice it unless you go looking.

Anything that reaches another person

The second group is actions whose effect lands in someone else's head. Once a message arrives, no amount of cleverness takes it back.

Sending an email. Replying to a thread. Posting to social media. Sending a Slack or WhatsApp message. Placing a phone call. Accepting or declining an invitation on your behalf.

Reply-all deserves a special mention. An agent that replies to a thread can reach forty people who did not expect to hear from you, in a tone you did not choose, about something you have not read.

The fix is almost always drafts. Let the agent write and let yourself send. You keep all of the useful work and none of the risk, and it takes one tap. The email case has enough detail to deserve its own post, so there is one: giving a personal agent access to your email without regret.

Money, and the part people get backwards

Paying a new person is the one to deny. Paying someone you have paid before, under a limit, is a reasonable candidate for approval.

The distinction matters because fraud follows new payees. A rule that allows transfers to accounts already in your history is much safer than a rule based on amount alone. A small payment to an unknown account is a worse signal than a large payment to your landlord.

Better still, do not give a personal agent payment access at all. Open banking separates reading your accounts from moving money, and you can grant one without the other. That is the subject of a personal finance agent that cannot move money.

Also deny purchases, subscriptions and anything that accepts terms. An agent clicking through a checkout is entering an agreement in your name, and the question of whether that agreement binds you is not one you want to find out about.

Deletes, and the cloud trash trap

Deleting looks recoverable because most services have a trash folder. Check that assumption before you rely on it.

Trash usually empties on a schedule, often thirty days. API deletes sometimes skip the trash entirely and remove the item immediately. And a sync client will happily propagate a delete to every device you own before you notice, which turns one mistake into a complete loss.

Deny permanent deletes. Allow archive, move and label instead. An agent that tidies by moving things is doing the same job with an undo button attached.

Why instructions are not limits

You can write all of this into a system prompt. It helps, and you should. But a prompt is a request, and the agent is reading other text all day.

Your inbox is writable by strangers. So is any shared document, any web page the agent reads, any calendar invite. If an agent can act and can read attacker-supplied text, someone can write instructions into the text it reads. That is prompt injection, and no amount of careful wording in your own prompt prevents it.

A limit has to live somewhere the model cannot reach. If the capability was never granted, no instruction can call it. That is the difference between asking an agent not to send email and running an agent that has no send method available.

What this looks like in practice

Two layers. First, only add the methods you want. An agent built from an explicit list of calls cannot invent a new one, and this is where most of your safety comes from. See API access control for AI agents for the general shape.

npx swytchcode
swy login
swy init

swy get gmail
swy auth connect gmail

# Add what you want. Nothing else exists to the agent.
swy discover "list messages" --project gmail
swy discover "create a draft" --project gmail
swy add method <list-messages-id>
swy add method <create-draft-id>

# No send method added. The agent cannot send, whatever it is told.

Second, write the hard limits down as rules. This catches the day you add a method without thinking it through, and it gives you a file you can read back later.

[
  {
    "id": "never-touch-account-recovery",
    "target": ["*.update-password", "*.update-mfa", "*.update-recovery-email"],
    "action": {
      "type": "DENY",
      "message": "Account recovery settings are not available to agents"
    }
  },
  {
    "id": "no-new-payees",
    "target": ["<payment canonical_id>"],
    "when": { "field": "payee_is_known", "operator": "==", "value": false },
    "action": {
      "type": "DENY",
      "message": "Payments to new payees are never automated"
    }
  },
  {
    "id": "known-payees-still-ask",
    "target": ["<payment canonical_id>"],
    "when": { "field": "amount", "operator": ">", "value": 100 },
    "action": {
      "type": "REQUIRES_APPROVAL",
      "message": "Agent wants to pay a known payee over 100"
    },
    "approval_timeout": "2h"
  }
]

Approvals arrive on Slack or Telegram and fail closed when they time out, so an unanswered request does nothing rather than proceeding. Setup is in the human approval docs and the rule schema is in the policy reference. Keeping the file in version control means you can see when a limit changed and why, which is the argument in policy-as-code for AI agents.

Credentials belong in a local encrypted store, not in a config file or an environment variable. They should be attached to the request when it runs, after the model has already decided what to do. Then a leaked conversation cannot contain a working key. More on that in keep API keys out of the LLM.

A starting list you can copy

Deny: password changes, two-factor changes, recovery email and phone changes, new OAuth grants, new SSH or API keys, new collaborators, sending messages as you, posting publicly, paying a new payee, purchases and subscriptions, permanent deletes, cancelling a service.

Approve: payments to known payees over a limit, moving money between your own accounts, bulk changes across many items, anything touching more than a handful of records at once.

Allow: reads, drafts, labels, archive, calendar events, notes, summaries, search.

That allow list is enough for most of what people want from a personal agent. Triage, summarise, prepare, remind, find. The useful work sits almost entirely in the reversible bucket, which is why giving up the dangerous actions costs you less than it sounds.

FAQs

Is an approval prompt enough on its own?

Not for the impossible-to-reverse list. Approval fatigue is real, and a prompt you see forty times a day becomes a button you press without reading. Keep the prompts rare so they still carry weight, and deny the things you would never say yes to anyway.

Can I just tell the agent not to do these things?

Tell it as well, by all means. But an instruction competes with every other piece of text the agent reads, including text written by people who are not you. A capability that was never added cannot be talked into existence.

What about read access? Is that safe?

Safe from irreversible damage, yes. Harmless, no. Read access to your email means read access to every verification code and password reset you receive. Scope what the agent can read, not only what it can change. The security checklist covers the questions worth asking first.

How do I know what the agent actually did?

Keep a log of every call with its arguments. swy audit network gives you that, and swy audit policy shows what the rules stopped. Reading the second one occasionally is how you find out that a limit has been doing real work. See how to log every API call an AI agent makes.

Does this change if the agent runs on my own machine?

The limits stay the same. Running locally changes where your credentials sit and what leaves your machine, which is worth understanding separately. See running a personal agent on your own machine.

Related reading

• A security checklist before connecting any AI assistant to company data
• Policy-as-code for AI agents
• What a credential vault needs to do for AI agents
• Human-in-the-loop approval for AI agent tool calls
• Browse all 300+ API integrations