ElevenLabs API for Voice Agents
It is xi-api-key, not Bearer. Character quotas are a hard wall rather than a slowdown, and audio does not fit a tool call's return value. Here is the shape that works.
DevRel Engineer
Three things about ElevenLabs will cost you an afternoon each if nobody tells you, and the first one is thirty seconds of reading.
One: the auth header is not Bearer
ElevenLabs uses its own header: xi-api-key: ELEVENLABS_API_KEY. Not Authorization: Bearer. The base is https://api.elevenlabs.io/v1.
Every model and every autocomplete will write Bearer, because Bearer is what nearly everything else uses, and the resulting error reads like a bad key rather than a bad header. If you are debugging auth here, check the header name before you regenerate anything.
One more auth detail worth knowing: ElevenLabs supports IP allowlisting, and a request from a non-allowlisted IP is rejected with a 403. If your agent runs on infrastructure with rotating egress IPs, that will look like intermittent auth failure. There are also single-use tokens with a limited lifetime, which are the right tool for handing a browser client temporary access rather than shipping your key to the front end.
Two: the quota is a wall, not a slowdown
Most APIs degrade when you push them. You get 429s, you back off, you carry on. Character-metered speech does not work like that. You have a credit allowance, and when it is gone, generation stops. There is no slower tier to fall back to.
For a voice agent that is a product outage, not a performance issue. Mid-conversation, your agent simply cannot speak. Which means character budget is a thing to design around rather than monitor after the fact.
The useful feature here: each API key can carry its own custom credit limit. That is an unusually good primitive for agents. Give the experimental agent a key with a small allowance and the production voice path a key with the real budget, and a runaway loop in the experiment cannot eat the credits your customers depend on. Most APIs make you build that separation yourself.
Then keep your own count. Track characters per call in the audit log so you know the burn rate before the wall arrives; API observability for AI agents covers what to capture.
Three: audio does not fit in a tool call's return value
A tool call returns a value to a model. Speech is a stream of bytes, often several hundred kilobytes, and sometimes it needs to start playing before it has finished generating. Those two shapes do not fit together.
Do not hand audio back through the model. Base64 in a tool result wastes an enormous number of tokens, the model has no use for the bytes, and in a streaming voice flow the latency is unacceptable anyway.
The pattern that works: the agent's tool call decides what to say and returns an identifier or a URL. Your own code, outside the model, fetches or streams the audio to wherever it needs to go. The model stays in the business of choosing words and never touches the payload.
Setting it up
npx swytchcode
swy login
swy init --mode=sandbox
swy get elevenlabs
swy auth connect elevenlabs # paste the key; the xi-api-key header
# format is handled for you
swy auth status
swy discover "text to speech" --project elevenlabs
swy discover "list voices" --project elevenlabs
swy list methods elevenlabs
swy info <canonical_id>
swy add method <canonical_id> # synthesis + voice reads. Not voice deletes.Leave the destructive methods out. Deleting a cloned voice your team spent a week tuning is not recoverable through the API, and a voice library is exactly the kind of shared resource an agent should be able to read and not touch. If the method is not in the allowlist, nothing can reach it: not a prompt, not an injected instruction, not a confused tool call.
The key lives outside your project, encrypted at rest, injected at call time, so it never enters the model's context and the correct header is assembled for you. See the authentication docs. Keep --project elevenlabs on discover calls, and run swy get before swy info.
The policy that protects your credits
{
"id": "cap-synthesis-length",
"target": ["<elevenlabs text-to-speech canonical_id>"],
"when": { "field": "text", "operator": "length>", "value": 1500 },
"action": {
"type": "REQUIRES_APPROVAL",
"message": "Agent wants to synthesise more than 1500 characters in one call"
},
"approval_timeout": "2h"
}Characters are the billing unit, so a character cap is a spend cap. Use whichever field the method exposes, which swy info will show. A second rule pinning which voice IDs may be used is worth adding too, because an agent that picks a different voice mid-conversation sounds like a bug to the person listening. Schema in the policy reference, approval behaviour in the human approval docs, and the reasoning for keeping rules in git in policy-as-code for AI agents.
Retries, where a duplicate is only money
Synthesis has no external side effect, so a retry does not corrupt anything. It does spend characters twice, and in a live call it also costs you the one thing you cannot buy back, which is time while a human waits in silence.
Rate-limit errors are retried with exponential backoff automatically; see the retries guide. For voice specifically, add a short overall deadline: if synthesis has not started returning audio within a second or two, it is better to fall back to a pre-recorded holding phrase than to keep waiting. Cache aggressively as well, since the fixed phrases in any voice product get spoken thousands of times and should be generated once. Background on when keys matter in API idempotency: the complete guide.
From an MCP client
Useful for content work rather than live calls: you are writing copy and want to hear it, or generating narration for a demo. Connect through the MCP server and the character cap and voice allowlist apply there too. Setup in the MCP guide, reference in the MCP docs, conventions for a coding agent in skills.md. For a live phone agent you want a purpose-built orchestrator instead, which is what the Retell AI guide covers.
What you can build
- Narration for generated content. Script, synthesise, and attach the audio to a video or a doc. Batch work where latency does not matter, which is the easiest place to start. Pairs with Luma if you are generating the visuals too.
- Accessible versions of written output. Turn a daily digest or a report into audio automatically. Low risk, genuinely appreciated, and the character cost is predictable because the input length is.
- Multilingual voice output. Translate first with a glossary so product names survive, then synthesise in the target language. See the DeepL guide for the translation half.
- A prompt-to-voice prototype. Wire synthesis behind a tool call so a text agent can speak, with a small-credit key so experimentation cannot touch the production budget.
FAQ
My key works locally and fails in production. Why?
Check IP allowlisting first. A non-allowlisted source IP gets a 403, and if your production egress IP rotates, the failure looks intermittent and random rather than like a configuration problem.
Should the agent ever hold the API key directly?
No, and for a browser or mobile client use the single-use token flow rather than shipping a long-lived key to the front end. For the agent itself, keep the key out of its context entirely; the reasoning is in credential brokering for AI agents.
How do I keep a voice agent from running out of credits mid-call?
Separate keys with separate limits so non-production work cannot drain the live budget, a character cap in policy so no single call is enormous, a cache for fixed phrases, and an alert on burn rate rather than on the balance hitting zero. By the time the balance is the signal, you are already down.
Can I use a self-hosted TTS engine instead?
Yes, and the integration story is the same: any REST API with an OpenAPI spec or a Postman collection can be added as a custom provider and gets the same allowlist, policy, retry and audit treatment. That is the usual route when voice data cannot leave your environment.
Related reading
More content
SendGrid API for AI Agents
One boolean in the SendGrid payload mails people who unsubscribed. bypass_list_management is the field your agent must never be able to set.
Chaitrali KakdeAddepar API for Wealth Management Agents
Addepar has no read-only key tier, so a key carries its user's permissions. If you want an agent that can only read portfolios, you have to build that constraint yourself.
Chaitrali KakdeFreeAgent API for Accounting Agents
FreeAgent lets you read accounting period locks before you write. An agent that skips that check will eventually post an entry into a period someone has already filed.
Chaitrali Kakde