Giving a personal agent access to your email without regret

Email is the first thing people connect to a personal agent, and with good reason. Your inbox holds the context for most of your week, and triaging it is dull work that a machine does well.

It is also the riskiest connection you will make, for two reasons that have nothing to do with privacy.

Your inbox is where every other account you own gets reset. And your inbox is writable by strangers.

Both facts shape how you should set this up. The good news is that one decision handles most of it, and you make that decision in about four seconds on the consent screen.

The scope you pick is the product you built

Gmail splits access into separate OAuth scopes. People click through this screen quickly, which is a shame, because the choice here matters more than anything you will write in code afterwards.

The combination worth starting with is gmail.readonly plus gmail.compose. The agent reads everything it needs and writes drafts. You press send.

That setup gives you roughly ninety percent of the value of an email assistant and removes the entire category of mistakes that reach other people. It is also easy to explain to yourself at three in the morning, which counts for something.

Avoid mail.google.com unless you have a specific reason. It is the scope that permits permanent deletion, and a deleted message that skipped the trash is gone. Most tutorials ask for it out of habit.

Why an agent that can send is a different product

A reading agent that makes a mistake wastes your time. You read a bad summary, you shrug, you open the thread yourself.

A sending agent that makes a mistake talks to your colleagues, your clients and your family in your name. You find out afterwards, from them.

Those are not two settings of the same tool. They have different failure modes, and the second one needs a level of confidence most setups have not earned yet. Reply-all makes it worse, because a single reply can reach everyone on a long thread, including people you forgot were on it.

If you do want sending, add it later, after a few weeks of reading the drafts it was producing. By then you will know whether it writes like you.

Your inbox is your password reset channel

This is the part people miss when they think read-only means harmless.

Read access to your mailbox includes every verification code, every magic link and every password reset email you receive. An agent with that access can, in principle, see the credentials for your bank, your domain registrar and your cloud provider as they arrive.

It probably will not look. But those messages pass through its context, and from there they can end up in a model provider's logs, in a conversation history, or in a summary saved to a file somewhere. The risk is not malice. It is copies.

Two things help. Narrow what the agent reads, using a query rather than the whole mailbox. And exclude the senders that carry security mail.

# Instead of reading everything, read a slice.
# Gmail search syntax works in the API's q parameter.

is:unread newer_than:2d -from:security@ -from:[email protected]
  -subject:(code OR verify OR verification OR "reset your password")
  -label:banking

This is not airtight. A determined test will find a reset email that does not match your filter. But it moves the common case out of reach, and it costs you nothing in usefulness, because an agent triaging your week has no need for verification codes.

A stronger option, if your provider supports it, is to keep security mail in a separate account that the agent is not connected to at all.

Your inbox is writable by strangers

Anyone who knows your address can put text in front of your agent. That is what email is for.

So if your agent reads email and can also take actions, you have built something that executes instructions supplied by the public. Someone can email you a message whose body says to forward your recent invoices to an outside address, and a capable agent reading that message may simply do it.

This is prompt injection, and it is the reason the read-only plus drafts pattern is more than caution. An agent that cannot send cannot be talked into sending. The capability is absent, so the instruction has nowhere to land.

Three habits reduce the rest of the exposure:

1. Keep the agent's other connections narrow. An email-reading agent that also holds your cloud storage token is a larger target than one that only reads email.
2. Treat message bodies as data, not instructions, and say so in your system prompt. It is not a guarantee, but it raises the bar.
3. Review the audit log now and then. An injection attempt that failed still shows up as a blocked call, which tells you someone is trying.

Setting it up

npx swytchcode
swy login
swy init

swy get gmail
swy auth connect gmail        # grant readonly + compose on the consent screen
swy auth status

swy discover "list messages matching a query" --project gmail
swy discover "get a message" --project gmail
swy discover "create a draft" --project gmail
swy list methods gmail

swy info <canonical_id>
swy add method <canonical_id>  # list, get, create-draft
                              # no send, no trash, no delete

The scope and the method list should agree. Granting gmail.modify and then leaving send out of the allowlist works, but you are relying on one layer where you could have two. Ask for the narrow scope as well, so a mistake in either place is caught by the other.

The OAuth token goes into a local encrypted store, refreshes on its own, and is attached when a call runs rather than being passed through the model. No token in a config file, and no token in your conversation history. See the authentication docs and what a credential vault needs to do for AI agents.

If you do add sending later

Gate it, and gate it on the things that actually go wrong.

[
  {
    "id": "no-new-recipients",
    "target": ["<gmail send canonical_id>"],
    "when": { "field": "recipient_is_known", "operator": "==", "value": false },
    "action": {
      "type": "REQUIRES_APPROVAL",
      "message": "Agent wants to email someone you have not emailed before"
    },
    "approval_timeout": "4h"
  },
  {
    "id": "no-mass-replies",
    "target": ["<gmail send canonical_id>"],
    "when": { "field": "recipient_count", "operator": ">", "value": 3 },
    "action": {
      "type": "DENY",
      "message": "Replies to more than three recipients stay manual"
    }
  }
]

A new recipient is the signal that matters, because an injected instruction almost always points somewhere you have never written to. The recipient cap handles reply-all. Held requests go to Slack or Telegram and fail closed, so an unanswered prompt sends nothing. The rule schema is in the policy reference, and policy-as-code for AI agents explains why these belong in a file rather than a dashboard.

What to build first

Four things that work well with read and draft access only.

1. A morning triage note. What arrived overnight, what needs a reply today, what can wait. One message, read in thirty seconds, instead of forty unread threads.
2. Drafted replies on the easy ones. Scheduling, acknowledgements, short answers you have given before. Open the draft, read it, send it. The work is in the reading, and the agent has done that.
3. Commitment tracking. Find the places you promised something and have not delivered. This is the one people find most useful and it needs no write access at all.
4. Waiting-on list. Threads where you asked a question and nobody answered. Pair it with a drafted nudge and you have a follow-up habit without the admin.

For a worked example with code, see how to build a Gmail triage agent. For the setup on a specific assistant, connecting Gmail to Grok walks through the consent flow end to end.

FAQs

Which Gmail scope should I grant an AI agent?

gmail.readonly and gmail.compose to start. Read plus drafts covers most of what you want. Add gmail.modify only when you want the agent labelling and archiving, and treat sending as a separate decision made later.

Can an agent read my two-factor codes?

If it has read access to the mailbox those codes arrive in, yes. Filter them out of the query, or keep security mail in an account the agent cannot see. Codes delivered by an authenticator app rather than email are not exposed this way, which is one more reason to move off email-based two-factor.

Is prompt injection through email a real risk or a theoretical one?

Real, and email is the easiest delivery method there is, because sending you a message requires no access to anything. The practical defence is to limit what the agent can do rather than to try to detect every malicious phrasing.

Does the agent send my email content to a model provider?

Whatever it puts in the prompt, yes, if you use a hosted model. That is worth knowing and worth narrowing with a tighter query. Running a personal agent on your own machine sets out which parts can stay local and which cannot.

What about Outlook or Fastmail?

The same shape applies. Microsoft Graph splits mail permissions into read and send in much the same way, and IMAP providers usually let you issue an app password with limited rights. Look for the narrowest permission that still does the job, then build the allowlist to match.

How do I revoke access if I change my mind?

Remove the connection in your Google account's security settings, which invalidates the token immediately, and delete the stored credential locally. Do both. A revoked token in a local store is harmless, but leaving it there makes your setup harder to reason about later.

Related reading

• What a personal AI agent should never be allowed to do
• How to connect Gmail to Grok bot in 2 minutes
• A security checklist before connecting any AI assistant to company data
• Keep API keys out of the LLM: credential brokering for AI agents
• Browse all 300+ API integrations