Can AI Do API Testing? What Works and What Doesn't

AI can do API testing. It can draft cases, vary inputs, and point at a response that does not match a schema faster than a person starting from a blank file. It cannot supply the contract. If the model guesses the endpoint, guesses the body, and then writes the assertion, the test checks the guess against itself. A green run means the invention was consistent. It does not mean the API behaves.

That split is the whole subject. Schema-driven generation works. Guessed assertions do not. The rest of this page is which jobs belong on which side, and how a contract test stays attached to the operation your client actually calls.

What people mean by API testing with AI

The searches collapse several jobs into one phrase.

Generating requests. Given an operation, produce valid and invalid inputs: missing fields, wrong types, boundary numbers, empty strings, ids that do not exist.

Generating assertions. Given a response, decide whether it is correct.

Exploring. Send traffic at a service the model has not been given a spec for, and see what comes back.

Maintaining a suite. Update tests when the API changes, or explain a failure.

The first job is where current models are useful, on the condition that the operation is real. The second job is useful only when the expected shape comes from the spec or from a fixture you captured, not from the model's recollection of the provider. The third job finds undocumented routes and also invents routes. The fourth job is editing. It inherits whatever was wrong with the suite it is editing.

Why Your AI Agent Calls the Wrong API is the production version of the third job. A test suite built the same way fails in the same way, quietly, because a test that calls a fictional route and expects a fictional error can still be written so that it passes.

What works

Hand the model the operation, not the provider's name. The operation has a method, a path, parameters, and a response schema. From that, a model can produce a useful set:

• One request with every required field set to a plausible value.
• One request per required field, with that field removed.
• One request per typed field, with the wrong JSON type.
• Boundary values where the schema states a minimum, maximum, format, or enum.
• An id that will not resolve, when the schema says the field is an identifier.

A second step, which should not be the same unchecked generation, compares the HTTP status and the body to the response schema. Required response fields present. Types match. An error response uses the error schema, not an HTML page with status 200. Unknown fields can be allowed or flagged, but the rule should be written down, because many APIs add fields without a version bump.

This is ordinary contract testing. The model is a generator of cases. The schema is the oracle. You can read every case. You can delete the ones that encode a business rule the schema does not contain. That last point matters. "A refund of 500 is five dollars" is true for Stripe and is not something a generic JSON schema always states in a way the model will notice. Amount units, idempotency, and auth are business rules. Put them in the test on purpose. Do not hope the model remembers them. Those three are why connecting Stripe is not a generic HTTP exercise.

Recording a known-good response and diffing later runs is the other thing that works. The model can write the diff and summarise it. The fixture, not the model, decides what "good" was. When the fixture is updated, a person updates it, against a changelog or a captured call, not against the model's preference for a cleaner body.

What does not work

Guessed assertions. The model writes expect(body.status).toBe('succeeded') because that is what a payment object often looks like. The API returns paid, or returns the status on a nested charge, or uses an enum the model has not seen since training. The test fails on a correct response, or, worse, the model "fixes" the test to match whatever the last call returned, including a bug.

Happy-path only. Generated suites love the example in the OpenAPI description. They never send the null, the zero, the duplicate idempotency key, or the second call after a timeout. The suite is green and the production incident is a double refund. Retry behaviour has to be a test you add. How Swytchcode Retries Failed API Calls is the behaviour worth asserting: the second attempt reuses the key.

Status-code theatre. 200 is not a pass. A 200 with an error string in the body is a failure. A lookup that finds nothing can be a valid result, not a flake, and a timeout is the case you retry. Teach the test the difference between "not found," "not allowed," and "try again." When the call goes through Swytchcode, the runtime already classifies retryable failures and reuses the same exec. That is the setup in Go-UPC API for AI Agents.

Testing the model's client instead of the API. A suite that mocks the HTTP layer and asserts the model built the request it expected to build is a unit test of a prompt. It never contacts the service. It will stay green when the service removes the field. Contract tests have to hit the implementation you ship against, or a simulator that speaks that implementation's schema.

Self-grading. Asking the model "did this response look right?" after it chose the request is not a test. It is a second opinion from the same source. Use it as a hint when triaging a diff. Do not use it as the assertion.

Contract tests from the spec you actually call

The spec and the client drift apart. A test generated from a public README, a blog post, or the model's training data checks an API you might not be calling. Editors do this in production already. Cursor making outdated API calls is that drift in the editor. A test suite copied from the same memory locks the drift in. The suite passes against the old shape, or it skips, and the new client fails in front of a user.

Fetch the operation into the project and generate tests from that artifact.

swy get <provider>
swy info <provider.method>
swy add method <provider.method>

swy info is the schema for the method you added. Generate cases from that output. When you regenerate, diff against the previous info output so a renamed field or a removed enum shows up as a contract change, not as a mysterious test failure three weeks later. The catalog is the API directory. If the method is not there, do not invent the path in the test and do not invent it in the client.

Auth in tests is the same rule as auth in the agent. The suite needs a credential for the sandbox. It does not need the secret in the prompt that asks the model to write the test. A model that writes Authorization: Bearer sk_live_... into a committed file has turned a test generator into a leak. Connect the credential outside the suite and let the call attach it. That arrangement is credential brokering. Sandbox versus live is not a comment in the test name. It is the mode of the project, because test and live keys look alike and the assertions will not catch a live charge that "passed."

Policy belongs in the contract too, if the client is an agent. A test that only checks the happy HTTP call will not notice that the agent can call a second, more dangerous method, or that an amount over the threshold went out without approval. Those are tests of the execution path: which methods are allowed, which calls were denied. Policy-as-Code for AI Agents is the rule file. swy audit policy is how you see the denied calls the functional test never made.

Versioning and what a passing suite is allowed to mean

APIs move. A date in the path, a header that selects a version, an enum that gains a value. A generated suite that pins nothing will follow whatever the model thinks is current, which is the wrong-API problem with assertions attached.

Pin the version in the client and in the test. When the version changes, change both in the same edit, and read the diff of the schema. A test that was checking a removed field should fail until you delete that assertion on purpose. A test that the model rewrites to keep the suite green has stopped being a test.

Compliance checks are narrower than people hope. A model can look for "does this response include a field we are not supposed to log" if you give it the list of fields. It cannot decide your retention rules. Give it the list.

A passing AI-generated suite means: these cases, generated from this schema, got responses that matched these assertions, against this environment. It does not mean the API is correct, the agent will call only these methods, or a retry is safe. Add those as their own cases or stop claiming them.

A small suite that is worth running

For one write operation you actually ship:

1. The valid request, asserted against the response schema, not against a snapshot the model invented.
2. Each required field removed, expecting the error the spec describes.
3. The same write sent twice with the same idempotency key, expecting one effect.
4. The same write sent twice with two keys, expecting two effects, in a sandbox, so you know the server is not deduping on the body alone.
5. A call the policy should deny, expecting the denial and no HTTP request, or an HTTP request that never left because the runtime blocked it.

That fifth case is the one generated suites skip, and it is the one that tells you the agent is contained. Exit codes when a call is blocked or waiting are in "swy exec" Explained. If the tool layer is the Anthropic Agent SDK, the retry in case 3 is not guaranteed by the SDK. It is guaranteed by the key. See Making Anthropic Agent SDK Tool Calls Idempotent.

FAQ

Can AI do API testing?

Yes, as a generator of cases from a real schema, and as a writer of glue around assertions you can read. No, as the source of the contract. If the model invents the endpoint and the expected body, the test does not tell you about the API.

What is the difference between schema-driven tests and guessed assertions?

Schema-driven tests start from the operation's parameters and response schema and check the live response against that schema. Guessed assertions start from what the model remembers a similar API doing. The first fails when the service breaks the contract. The second fails when the model's memory was wrong, or passes when both the client and the test share the same mistake.

Should I trust a test that only checks the status code?

No. Check the body against the response schema. A 200 can carry an error payload. A 404 can be the correct result for a missing resource. The expected status belongs to the case, and it should come from the spec.

How do I stop generated tests from going stale?

Generate them from the schema your client uses, and diff that schema when you regenerate. Pin the API version in the client and the suite together. Do not let the model update assertions just to make a run pass.

Can AI test whether an agent called the right API?

Only if the right API is an artifact you can name: a method id, a path, a version. Compare the call the agent made with that artifact. A model grading another model's choice, with no schema, repeats the original error. The failure mode is Why Your AI Agent Calls the Wrong API.

Where should API credentials live in an AI-written suite?

Outside the prompt and outside the committed test. The suite runs against sandbox credentials attached at execution time. A generated file that contains a live secret is a leak, not a test.