Watch agent API calls change when the prompt changes

Tuesday's agent read tickets and drafted notes. Wednesday's prompt edit tells it to resolve the ticket when the customer sounds done. The dashboard still says "API calls, about the same." Support sees a pile of closed tickets. The total count hid a shift from a read method to a write method. The prompt was the release. Nothing in the chart was watching per operation.

This post is the monitoring pass for that shift. API observability for AI agents covers what to record and where the gaps are. Here you set a baseline per method and you treat a prompt change as the event that invalidates it.

Set the baseline on each method

Pick the methods in tooling.json. For each one, write down three numbers from a quiet week: how many calls, what share failed or were blocked, and which arguments showed up. A blocked call exited 6 and never left the machine. An approval wait exited 7. Those are different from a provider 400, and the baseline should not add them into one "error" bucket.

swy audit stats
swy list methods

swy audit stats is the local summary. It does not require a sign-in. It is not the hosted log. Hosted storage stays off until you turn it on. When it is on, retention is 7 days on Free, 30 days on Pro, and 90 days on Business. Do not draw a 90-day baseline on a Free workspace. You do not have the days.

The audit and telemetry docs describe what the local commands show. If a method is missing from the stats, it did not run. That zero is part of the baseline. A prompt that starts calling it is a change you want to see.

What a prompt edit does to the mix

The model picks the method. The wording is what it picks from. You can ship a prompt that mentions "resolve" and watch the write method move from zero calls a day to most of the traffic, while the read method drops. The sum of the two can stay flat. A chart of total calls will call that normal.

Tie the baseline to the prompt revision the way you tie a dashboard to a deploy. When the prompt file changes, note the git sha next to the week's numbers. The next day's job is a comparison, method by method:

• A method that was near zero and is now busy
• A write whose argument keys were not in last week's sample
• A block rate that fell because someone edited the policy in the same pull request

The third one is easy to miss. A prompt and a policy often land together. If you only watch provider errors, a policy that stopped blocking looks like healthy traffic. Log every API call is how you separate blocked, held, and sent.

Alerts that match this failure

Alert on the method, not on the workspace total. A practical set:

• Write methods: alert when the daily count is more than double the baseline, or when it leaves zero
• Argument keys: alert when a new key appears on a write
• Block rate: alert when it drops to zero on a method that used to block

Skip an alert on "total calls plus or minus 10 percent." That is the Wednesday chart that stayed green while tickets closed.

Dry runs do not belong in the baseline. swy exec --dry-run does not send and does not count as a live command. A CI job that dry-runs on every pull request would drown the numbers if you mixed it in. Count live commands: requests that went to the provider. API-key calls count, including ones from a scheduled job.

What you do when the mix moves

If the write spiked and the prompt sha changed, roll the prompt back or remove the method from tooling.json. Removing the method stops the call even if the prompt still asks. That is the faster rollback when you do not yet trust the new wording.

If the mix moved and the prompt did not change, look at the policy file and at the provider. A schema change on their side shows up as failures, not as a new method. A policy edit shows up as a different block rate. Those are two different pages in the log.

Re-baseline only after you accept the new mix. Otherwise Thursday's spike becomes Friday's normal and you lose the only signal you had.

Write the three numbers per method before the next prompt edit. Compare them the morning after. Roll back the prompt or the method when a write leaves zero without a review.

Related reading

• Best API observability tools for agent traffic. When you need a tool past the local audit summary.
• Policy as code for agent calls. The file that changes block rate without a prompt edit.
• Human approval for a tool call. What an exit code 7 means in the baseline.
• API governance tools for AI agents. Why the allowlist is the list you baseline, not the vendor catalog.

FAQ

Is swy audit stats the same as the hosted log?

No. swy audit stats is local and works without a sign-in. The hosted log is off until you enable storage. Retention then follows the plan: 7 days on Free, 30 on Pro, 90 on Business.

Should the baseline include dry runs from CI?

No. A dry run does not send and does not count as a live command. Baseline the calls that reached a provider. Otherwise the pull request job sets the normal.

What if I only have a total count from the provider?

You will miss a shift from read to write when the total holds still. Ask for the method ID in the log before you trust the chart. A provider total is a backup signal, not the baseline.

How long should the quiet week be?

As long as your retention. A Free workspace with storage on can see 7 days. Do not average a month you cannot query. If storage is still off, use the local audit on the machine that ran the agent and say so next to the numbers.