How to Triage GitHub Issues and Pull Requests With Jev

Issue triage is the work nobody schedules. A repository gets twelve issues in a week, nine of them need a label and an owner, two are duplicates, one is a security report that should never have been filed in public, and all of it sits there until a maintainer has an hour.

It is also close to an ideal job for a decision model, because every part of it is a classification with a small answer space. Which area of the codebase. How urgent. Is it actually a bug or a question. Is this a duplicate of something open.

What it is not is a job for a bot that comments. We will come back to that, but the short version is that a label is reversible and a comment is a notification to everyone subscribed to the thread, and your triage bot will be wrong sometimes.

If you want the pattern against an inbox or a chat channel instead, the Gmail triage build and the Slack router are the same shape. Swytchcode's own GitHub triage bot is in the swytchcode-examples repo as openclaw-swytchcode, which is worth reading alongside this.

The quirk that breaks naive triage

Start here, because it will cost you an afternoon otherwise.

In GitHub's REST API, every pull request is also an issue. The issues endpoint returns both. A PR comes back with the same shape as an issue, with a number, a title, a body, labels, and the only reliable way to tell them apart is that a PR carries a pull_request key and an issue does not.

So a triage loop that lists issues, asks a decision model "is this a bug or a feature request", and applies a label will cheerfully label your pull requests as bugs. Ours did. Three PRs got type:bug and one got needs-reproduction, which is a confusing thing for a contributor to receive on a patch they just submitted.

const isPullRequest = (item) => Boolean(item.pull_request);
const issuesOnly = items.filter(i => !isPullRequest(i));

That one line is the difference between a triage bot and an apology. If you want to triage PRs as well, do it deliberately and with different questions, because "which area of the codebase" makes sense for both while "is this reproducible" does not.

Setup

No .env file and no token in your code. Credentials go into Swytchcode's own store outside the project, and the execution mode is a project setting answered once.

npx swytchcode
swy login
swy init --mode=sandbox

swy get github
swy get jev

swy auth connect github   # OAuth, opens a browser
swy auth connect jev      # prompts for your API key
swy auth status

swy init writes the editor and execution mode into .swytchcode/tooling.json. Sandbox versus production never appears as an environment variable.

The methods, verified

Only methods you add can run. For a bot with write access to a public repository that is not a nice-to-have, it is the main reason to put an execution layer under it at all.

swy add method <jev-method-id>                # find it with: swy list methods jev
swy add method github.issue.get.1             # list open issues
swy add method github.issue.labels.create     # add labels to an issue
swy add method github.issue.labels.get        # read an issue's current labels

Two notes on this table, both learned the hard way.

github.comment.create is not what you want. It creates a comment on a gist, not on an issue. We assumed from the name and were wrong. If you do want your bot to comment, find the right method yourself rather than copying one out of a blog post:

swy discover "add a comment to an issue" --library github
swy list methods github
swy info github.issue.labels.create

There are several label methods and they behave differently. labels.create adds to what is there. labels.update replaces the set. labels.delete removes everything. A triage bot almost always wants create, because a maintainer may have already applied a label you should not clobber.

Reading the issues

import { exec } from "@swytchcode/runtime";

const OWNER = "your-org";
const REPO = "your-repo";

export async function openIssues(limit = 50) {
  const items = await exec("github.issue.get.1", {
    params: { owner: OWNER, repo: REPO, state: "open", per_page: limit, sort: "created", direction: "desc" }
  });
  return items.filter(i => !i.pull_request);
}

Then the filter for what is worth spending a decision on:

const BOT_AUTHORS = new Set(["dependabot[bot]", "renovate[bot]", "github-actions[bot]"]);

function isTriageable(issue) {
  if (BOT_AUTHORS.has(issue.user?.login)) return false;      // dependency bumps triage themselves
  if (issue.labels?.length) return false;                     // a human or a previous run got here first
  if (!issue.body || issue.body.trim().length < 30) return false;
  return true;
}

issue.labels?.length doing double duty is the same trick as using a reaction in Slack: GitHub has no "triaged" flag, so your label is the marker. Once an issue has any label, the bot leaves it alone. That means a maintainer labelling something by hand also removes it from the bot's queue, which is the behaviour you want, and it survives your process restarting because the state lives in GitHub.

The downside is that an issue a human labelled good first issue will never get an area label from the bot. We decided that was the right trade: a human touched it, so the bot should stop.

The questions

const AREAS = {
  "area:cli":      "The swy command line tool: commands, flags, output, installation.",
  "area:runtime":  "The language SDKs and the exec call: imports, errors, types, retries.",
  "area:policies": "Policy rules, approval flows, and anything in policies.json.",
  "area:docs":     "Documentation that is wrong, missing, or unclear. Not a code problem.",
  "area:unclear":  "Not enough information to tell which part of the project this concerns."
};

const PRIORITY = [
  "Nice to have. A suggestion, a question, or a cosmetic issue.",
  "Normal. A real problem with a workaround, or a well-scoped improvement.",
  "Important. Affects a common path and has no good workaround.",
  "Urgent. Broken for everyone, data loss, or a failing release."
];

export async function triage(issue) {
  const { answers } = await exec("<jev-method-id>", {
    body: {
      model: "jev-latest",
      state: `${issue.title}\n\n${issue.body.slice(0, 4000)}`,
      questions: {
        area:        { type: "choice", instructions: "Which part of the project does this issue concern?", criteria: AREAS },
        priority:    { type: "score",  instructions: "How urgent is this issue for the maintainers?", criteria: PRIORITY },
        is_bug:      { type: "noul",   instructions: "This describes something behaving incorrectly, rather than asking a question or requesting a feature." },
        has_repro:   { type: "noul",   instructions: "The issue contains concrete steps, code, or commands that would let someone reproduce the problem." },
        is_security: { type: "noul",   instructions: "The issue describes a vulnerability, an exposed credential, or another security problem." }
      }
    }
  });
  return answers;
}

A few deliberate choices in there.

area:unclear is the escape option. Without it, an issue that says "doesn't work, please fix" has to be assigned to the CLI or the runtime or the docs, and it will be, with a confidence around 0.6 that looks fine in a log. Every choice question wants an escape option.

state is truncated to 4,000 characters of body. Issue bodies can include an entire stack trace, a full log, and three screenshots' worth of markdown. Jev's limits are 64k tokens for state plus questions, so a long issue will not fail outright, but you pay linearly for tokens and the signal is almost always in the first screenful. Our pricing post goes through why trimming state is the only real cost lever.

is_security is not there to label anything. It is there to stop the bot.

The plan

const CONFIDENT = 0.82;

export function planFor(a) {
  // 1. Security reports: never labelled, never categorised, straight to a human.
  if (a.is_security.noul > 0.4) {
    return { action: "alert", reason: "possible security report" };
  }

  // 2. No idea which area: leave it for a person rather than guessing in public.
  if (a.area.choice === "area:unclear" || a.area.confidence < CONFIDENT) {
    return { action: "skip", reason: `area confidence ${a.area.confidence.toFixed(2)}` };
  }

  const labels = [a.area.choice];

  if (a.is_bug.noul > 0.85) {
    labels.push("type:bug");
    if (a.has_repro.noul < 0.3) labels.push("needs-reproduction");
  } else if (a.is_bug.noul < 0.2) {
    labels.push("type:question");
  }

  if (a.priority.score >= 2.6) labels.push("priority:high");

  return { action: "label", labels, priority: a.priority.score };
}

Rule one has a threshold of 0.4, which is deliberately low, and no confidence term at all. For most questions you want to be fairly sure before acting. For "is this a security report", you want to stop on a suspicion, because the cost of wrongly pausing is that a maintainer reads an issue thirty seconds later, and the cost of being wrong the other way is a public label drawing attention to an unpatched vulnerability. Asymmetric costs deserve asymmetric thresholds, which confidence thresholds covers properly.

Rule two skips rather than labelling area:unclear. We tried applying the unclear label and it turned out to be worse than nothing: maintainers filter on labels, an area:unclear label looks like triage happened, and the issue ends up less visible than if the bot had left it alone.

Note also that is_bug has a gap in the middle. Above 0.85 it is a bug, below 0.2 it is a question, and between those it gets the area label and nothing else. A two-sided threshold with a deliberate dead zone is usually better than a single cutoff at 0.5, because the middle of a noul range is exactly where the model is telling you it does not know.

Applying the labels

export async function applyLabels(issue, labels) {
  await exec("github.issue.labels.create", {
    params: { owner: OWNER, repo: REPO, issue_number: issue.number },
    body: { labels }
  });
}

labels.create adds rather than replaces, so this cannot remove something a maintainer put there. If you reach for labels.update instead, read it as "replace every label on this issue with the ones I am sending", which is almost never what a triage bot should do.

Checking a method's inputs with swy info before running it. For the GitHub label methods this is worth doing specifically, because create, update and delete differ in whether they preserve what a human already applied.

The loop:

export async function runOnce() {
  const issues = (await openIssues(50)).filter(isTriageable);
  const log = [];

  for (const issue of issues) {
    const answers = await triage(issue);
    const plan = planFor(answers);

    if (plan.action === "label") await applyLabels(issue, plan.labels);
    if (plan.action === "alert") await notifyMaintainers(issue, plan.reason);
    // "skip" does nothing, on purpose

    log.push({ number: issue.number, ...plan, confidence: answers.area.confidence });
  }
  return log;
}

notifyMaintainers deliberately does not post to the issue. Ours posts into a private Slack channel with slack_web.chat.postmessage.create, because the entire point of rule one is to get human eyes on something without drawing public attention to it.

Why labels and not comments

The temptation with a triage bot is to have it explain itself in a comment. "Thanks for the report. This looks like a CLI issue with high priority." It feels helpful and it reads well in a demo.

Three reasons we stopped doing it.

A comment is a notification. Every subscriber to the repository gets it. A label is silent and shows up where maintainers already look.

A comment cannot be taken back. You can edit it, and the edit history stays. When your bot is wrong at a 12% rate, that is a lot of small public mistakes accumulating in a repository that contributors read.

Labels are already the interface. Maintainers filter by label. An issue labelled area:cli and priority:high is in the right list. A comment saying the same thing is prose someone has to read and then act on manually.

So the recommendation is the same as the Slack router's: do the reversible, quiet thing first, measure it for a week against a repository you already watch, and only then consider whether anything louder earns its place.

The policy underneath

planFor is your code, so a refactor can change it. A policy sits outside the code and is checked before the call goes out.

{
  "id": "no-label-wipes",
  "target": ["github.issue.labels.update", "github.issue.labels.delete"],
  "action": { "type": "DENY", "message": "Triage may add labels but never replace or clear them" }
}

swy policy add
swy audit policy
swy audit network

That is a DENY rather than an approval gate, because there is no version of the triage bot that should be clearing a maintainer's labels, so there is nothing to approve. Combined with only having added four methods in the first place, the bot's total write capability against your repository is "add labels to an issue", which is a sentence you can put in a pull request description when you propose running it.

How to set up policy guardrails covers the rule shapes, and how to build a GitHub AI agent covers the broader integration.

A real run

Fifty open issues from a mid-sized repository, with the security path enabled:

50 open items fetched
  -7 pull requests filtered out
  -18 already had labels
  -3 bot-authored
  = 22 triaged

  14 labelled
   5 skipped (area confidence below 0.82)
   2 skipped (area:unclear)
   1 alerted (possible security report)

The alert was a real one: an issue containing a pasted log with an API key still in it. It got no label, no comment, and a Slack message to the maintainers, who deleted the log and rotated the key. That single case is why rule one has a low threshold and sits above everything else.

Of the 14 labelled, a maintainer later changed two area labels. A 14% correction rate on a reversible action is fine. The same rate on public comments would not have been.

Things we got wrong

We labelled pull requests as bugs. The issues endpoint returns PRs, and we did not know. Three PRs got type:bug and one got needs-reproduction. Filter on the pull_request key.

We assumed github.comment.create commented on issues. It creates a gist comment. The name is reasonable and the behaviour is not what we expected, which is a good argument for running swy info before using any ID you have not used before.

We used labels.update first. It replaced the label set, which wiped a good first issue label a maintainer had added that morning. labels.create adds. The difference is one word in the method name and a conversation you do not want to have.

We labelled area:unclear. It looked like diligence and functioned as camouflage: the issue appeared triaged, so it dropped out of the untriaged filter and got less attention than if we had done nothing. Skipping is better than labelling your own uncertainty.

We set the security threshold at 0.8. Far too high for that question. We lowered it to 0.4 after an issue with a leaked token in a log came back at 0.57 and got a cheerful area:cli label. Thresholds should follow the cost of being wrong, and that one is lopsided.

FAQ

Why does my issue triage keep labelling pull requests?

Because GitHub's issues endpoint returns pull requests as well as issues. They have nearly the same shape. Filter out anything with a pull_request key before you triage, or handle PRs deliberately with their own questions.

What is the canonical ID for listing GitHub issues?

github.issue.get.1, described in the registry as "List open issues in a repository". For pull requests specifically it is github.pull.get. Confirm with swy list methods github rather than trusting a remembered ID.

How do I add a label without removing existing ones?

github.issue.labels.create adds to the existing set. github.issue.labels.update replaces the whole set and github.issue.labels.delete clears it. A triage bot almost always wants create.

Can the bot comment on issues instead of labelling them?

Technically yes, but we would not. Comments notify every subscriber, cannot be withdrawn, and duplicate information that labels already carry in a form maintainers filter on. Note that github.comment.create is for gists, so find the right method with swy discover if you go this way.

How does it avoid re-triaging the same issue?

It skips any issue that already has labels, which means its own label doubles as the triaged marker. A maintainer labelling something by hand also takes it out of the queue, which is usually what you want.

Should it triage dependabot pull requests?

No. Filter bot authors out. Dependency bumps carry their own labels and conventions, and spending a decision on them is noise in your logs.

What stops it going wrong on a security report?

A dedicated noul question with a deliberately low threshold, checked before anything else and with no confidence term. On a suspicion it applies no label at all and notifies maintainers privately. The asymmetry is the point: a false alarm costs thirty seconds, a missed one is public.

How much does this cost to run?

Very little. At roughly 1,200 tokens per issue, 22 triaged issues is about $0.001. The execution side is metered separately by action; the pricing post breaks both down.

Is there working code I can read?

Yes. openclaw-swytchcode in the swytchcode-examples repo is a GitHub triage bot built on this pattern.

Wrapping up

Issue triage is a good first agent because the answer space is small, the actions are reversible, and the cost of a mistake is a maintainer changing a label. Get the PR filter right, label rather than comment, use an escape option on the area question, and give the security check its own low threshold ahead of everything else.

The part that does not come from the model is what the bot is allowed to do. Four added methods and one DENY policy mean its entire write capability against your repository is adding labels, and that is a claim you can make to the person who has to approve running it.

The Jev integration is live at swytchcode.com/apis/jev and the GitHub integration page covers the action side. npx swytchcode installs the CLI, and swytchcode.com/skills.md is the agent-readable setup file for handing to a coding agent.