We asked our Hermes agent if it was leaking API keys. It printed one into the chat.

We asked a Hermes agent, running a small free model, "Am I leaking any API keys?" It ignored the security skill we'd installed, wrote its own scan, found the agent's own access token and printed it in full into the chat, where it was saved.

Every app you connect to your Hermes agent hands it a key: Stripe, GitHub, Notion, your AI provider. Those keys end up in its memory, its chat history, its skills, its settings and its old backups. Some of them are sent to your AI provider with every message you type.

API Key Leak Check is a free, open-source skill that checks your Hermes agent in seconds, tells you in plain words what's exposed, and changes nothing without your yes.

Where an agent keeps your keys

When you connect an agent to Stripe, GitHub or Notion, the key has to live somewhere. In Hermes it can end up in more places than you'd think:

• Memory. Hermes puts MEMORY.md, USER.md and SOUL.md into every prompt. A key saved there isn't just a file on your disk: it's sent to your AI provider with every message you send. In our tests, a planted test token in MEMORY.md showed up in the model's thinking before it had run a single command.
• Chat history. Paste a key into a chat once, and it's saved in Hermes's chat database, including recent chats that haven't been written to the main file yet.
• Skills. Some skills have a key written straight into them. Some print keys into the chat when they run. Many read your app keys from the environment, so the agent holds them all.
• Settings. .env and config.yaml keep keys as plain text. That's normal for the agent's own keys, but a problem when the file can be opened by other accounts on the computer.
• Old backups. Hermes keeps snapshots. On our own Mac, the check found an old snapshot copy of .env we'd long forgotten about.
• Terminal startup files. A key in .zshrc or .bashrc reaches every program you start, the agent included.

On a test server, besides the keys we planted, the check found unnamed app keys in the agent's settings and three skills handling keys directly.

What the check does

Install it:

hermes skills install swytchcodehq/api-key-leak-check/api-key-leak-check

Then ask your agent: "Is my setup leaking any keys?" You get an answer written for a person, not a security team:

Here's a real run on one of our own computers, with the skill and app names hidden:

SECURITY CHECK: ACTION NEEDED

Some of your Hermes skills can print your keys into the chat, where they get saved in your history.

What we found:
1. [Hermes] 1 skill can show your keys in the chat, where they get saved: [skill name].
2. [Hermes] 2 old backup files still hold copies of your keys.
3. [Hermes] 9 skills use your [app names] keys directly: [skill names].

What you can do:
1. Clean up: remove the stray copies of your keys. Your agent can do this for you, one step at a time.
2. Optional: let Swytchcode hold your app keys ([app names]), so your agent never has them. Your agent can set it up for you.

Say "show me the details" to see where each one is.

Nothing was changed. No key was shown or sent anywhere.

Ask for the details and it tells you which file and line. Ask it to clean up and it says exactly what will change, then waits for your yes.

It can remove a stray key, lock a file or delete an old backup, one item at a time. It never touches the keys your agent needs to run, and it never makes a backup copy of a key, because a backup is just another leak.

Why you can trust a tool that reads your keys

A tool that looks for keys looks a lot like a tool that steals them. So we made it easy to check:

• It only reads. The check changes nothing. Fixes happen only after you say yes, through a separate script.
• It can't reach the internet. It uses Python's standard library only. A test checks that the scripts import nothing that can open a network connection or start another program.
• It never shows a key. Only the first 4 characters and the length, in every output.
• It's tested. 62 tests on macOS, Linux and Windows run on every change. They build a fake agent, plant fake keys, and check that each one is found and none is printed.
• It's open source. Two short Python scripts, MIT licensed: github.com/swytchcodehq/api-key-leak-check.

Hermes blocked us at first, and that's a good sign

Hermes scans every skill you install from GitHub, and blocks anything that looks dangerous. Our first version was blocked.

A credential checker trips every alarm a stealer would: key patterns, .env files, lists of agent config files. We didn't hide our scripts from the scanner, which is what a malicious skill would do. We reworded what it flagged, honestly, until it passed. Hermes now rates the skill safe.

Hermes is right to be strict. Its own code treats every community skill with caution, after hundreds of malicious skills turned up on a public skill registry earlier this year. Read what a skill does before you install it, including this one.

Use a capable model

The leak at the start of this post happened with a small free model. With a GPT-5 class model, the agent loaded the skill, ran one command and showed the result exactly as written.

A skill is instructions. It can't force a model to follow them. To make it more likely:

• Start the chat with the skill loaded: hermes chat -s api-key-leak-check
• Or ask for it by name: "Use the api-key-leak-check skill. Am I leaking any keys?"

The safest way, with any model, is to run the check yourself. No agent, no chat, no chance of a key ending up in either:

git clone https://github.com/swytchcodehq/api-key-leak-check
python3 api-key-leak-check/skills/api-key-leak-check/scripts/audit.py

Try it

hermes skills install swytchcodehq/api-key-leak-check/api-key-leak-check
hermes chat -s api-key-leak-check

Then ask: "Is my setup leaking any keys?"

It works with OpenClaw too, and with npx skills add swytchcodehq/api-key-leak-check.

We make Swytchcode, which holds your app keys so your agent no longer has them. The check offers it once, as an option, and does nothing unless you say yes. We have a generous free plan if you'd like to try it.

The check itself is free and works the same either way.