Using the Hugging Face API from an AI Agent

When someone says "give the agent Hugging Face access," they could mean three quite different things, and the difference decides how much damage a confused agent can do.

• Hub reads. Looking up models, datasets, metadata, model cards, downloading weights. Research and retrieval.
• Inference. Actually running a model on some input and getting a prediction back. This is the one most agents want: a classification step, an embedding, a reranker.
• Hub writes. Creating repos, pushing files, editing a model card. Almost never what an agent needs, and the one people accidentally enable.

That last point is worth sitting with. A write token handed to an agent means the agent can modify your public model card. Not because it wants to, but because a tool call went sideways or a document it read contained instructions. The fix is boring and effective: never give it a write token.

The three token types, and which one to use

Hugging Face gives you three roles, and the naming hides how different they are.

• read grants read access to every repo you can read, including private ones and your organisations'. It is the easy choice and it is broader than it sounds: one leaked read token exposes everything your account can see.
• write is everything read can do, plus writing to anything you can write to. For training jobs and model pushes. Not for agents.
• fine-grained scopes access to specific resources, such as one model or the models in one organisation. Hugging Face's own docs describe this as the type that is "useful in production environments," and that is the right read: it is the only one where a leak has a bounded blast radius.

So the rule for an agent is: one fine-grained token, scoped to the specific model or org it needs, per agent. Not a shared read token, and definitely not your personal write token that also happens to work.

Hugging Face also recommends one token per app or usage, so you can revoke the token for a misbehaving agent without taking down your CI, your notebook and your inference server at the same time. If you are wiring this into a pipeline, look at Trusted Publishers too, which exchanges a CI provider's OIDC token for a short-lived Hub token so there is no long-lived secret sitting in your settings at all. That is the same idea as credential brokering, applied upstream.

Setting it up

npx swytchcode
swy login
swy init --mode=sandbox

swy get huggingface
swy auth connect huggingface     # paste the FINE-GRAINED token here
swy auth status

# inference and hub reads are different methods; find both
swy discover "run inference on a model" --project huggingface
swy discover "get model info" --project huggingface
swy list methods huggingface

swy info <canonical_id>
swy add method <canonical_id>     # allow inference + reads, nothing that writes

Pass --project huggingface on every discover, because "run inference" matches a dozen model providers in the catalog and you do not want to accidentally wire up a different one. And run swy get before swy info, or info will claim a real method does not exist.

The token lands in a local store outside your project, encrypted at rest, and gets injected into the Authorization header when a call runs. There is no HF_TOKEN environment variable in your project and nothing for the model to read. That matters here specifically because the Hugging Face Python libraries are designed to pick a token up from the environment, which is convenient in a notebook and a liability in an agent process. See the authentication docs.

Calling it from your own code is one function:

from swytchcode import exec

# canonical_id from `swy list methods huggingface`
res = exec("<hf inference canonical_id>", {
    "params": {"model": "your-org/your-classifier"},
    "body": {"inputs": ticket_text}
})

# the method was allowlisted; the token was injected; nothing to leak
label = res[0]["label"]

The failure mode nobody plans for: a cold model

Shared inference is not guaranteed to be instant. A model that has not been called recently may need loading, and the first request can be slow or come back telling you it is not ready yet. Your agent sees something that is not a clean prediction and decides inference is broken.

Handle it as a retryable condition with backoff rather than a failure, and if the latency matters to a user who is waiting, use a dedicated endpoint rather than shared infrastructure. Rate-limit responses are retried with exponential backoff automatically, which covers the related case where you have simply gone too fast. Details in how Swytchcode retries failed API calls and the retries guide.

Watch for the soft failure too. An inference response can be a perfectly valid 200 containing an error object or an empty result, which standard monitoring records as success and your agent treats as a real answer. Response validation catches it; the general pattern is in 200 OK but it failed.

Idempotency, briefly, because inference is unusual here

An inference call is a POST, but it has no side effect: run it twice and you get a prediction twice. So the duplicate risk is cost and quota, not data corruption. You do not need an idempotency key to stay correct; you do want caching to stay cheap.

Cache by model plus a hash of the input. Agents re-ask the same question constantly, especially inside a retry loop or a multi-step plan that revisits the same document. One cache in front of inference usually cuts call volume substantially. The wider discussion of when keys matter and when they do not is in API idempotency: the complete guide.

Policies for a model API

Two rules carry their weight here. Pin which models the agent may call, so it cannot wander off to some random community checkpoint, and cap batch size so a loop cannot run inference over your entire dataset.

{
  "id": "approved-models-only",
  "target": ["<hf inference canonical_id>"],
  "when": { "field": "model", "operator": "!=", "value": "your-org/your-classifier" },
  "action": {
    "type": "DENY",
    "message": "Only the approved classifier may be called from this agent"
  }
}

A deny rather than an approval request, because "which model are we allowed to use" is a decision your team makes once, not something to interrupt someone about at runtime. Check the available operators in the policy reference and keep the file in version control so adding a model is a reviewed change. That review step is more valuable than it looks: swapping the model behind a production classifier is exactly the kind of change that should not happen quietly. See policy-as-code for AI agents.

There is a data question too. If your agent sends customer text to a shared inference endpoint, that is a data transfer your privacy review will care about. The audit log of exactly which inputs were sent is what answers it, and a fine-grained token plus a dedicated endpoint is what keeps the answer comfortable.

In Claude Code or Cursor

Hub reads are a genuinely pleasant MCP tool: you are picking a model, you want to compare three model cards and their licences without leaving the editor. Inference over MCP is fine too for exploratory work. Connect through the MCP server and the allowlist travels with it, so an editor session cannot reach a write method you never added. Setup is in the MCP guide, the reference in the MCP docs, and skills.md if you want a coding agent to do the wiring.

What you can build

1. A cheap classification step inside a bigger agent. Not everything needs a frontier model. Routing a support ticket to a queue, detecting language, or scoring sentiment is a small specialised model's job, and it is faster and cheaper than asking a large model to do it. Pairs well with a FreshDesk or Intercom triage agent.
2. Embeddings for your own retrieval. Generate embeddings for internal documents on a schedule and store them yourself. Read-only against Hugging Face, and you keep the vectors.
3. A model-selection research agent. Given a task, search the Hub, read the model cards, and produce a shortlist with licences, sizes and stated limitations. Genuinely useful, and licence checking is the part humans skip.
4. A dependency watcher. Check the models you depend on for new revisions, licence changes or deprecation notices, and report. The agent reads; a human decides whether to move.

FAQ

Can the agent download and run a model locally instead?

It can, and that moves the problem rather than solving it: you now have weights on disk and a process that can execute arbitrary model code. For agent work, calling a hosted endpoint with a narrowly scoped token is usually the easier thing to reason about and to get through review.

Do I need a dedicated inference endpoint?

If a human is waiting on the response, probably yes, because shared infrastructure has variable latency and cold starts. For batch work on a schedule, shared is fine and cheaper.

What is the single most important setting?

The token type. A fine-grained token scoped to one model, created specifically for this agent, means a leak or a confused tool call cannot touch anything else you own. Everything else in this post is a refinement on that. Related thinking in best secrets management tools for AI agents.

How do I track what it is costing?

swy audit network gives you every inference call that actually ran, with arguments, which is the number that maps to your usage. If it is higher than you expected, you are missing a cache rather than a rate limit.

Related reading

• Hugging Face integration page
• Keep API Keys Out of the LLM: Credential Brokering for AI Agents
• Securely Connecting the Claude API to Your Agent
• Luma API for AI Agents
• Browse all 300+ API integrations