Auto fix CI failures and PR comments with AI

Being able to auto fix CI failures with AI takes one of the most repetitive chores off your plate. You push, and ten minutes later GitHub Actions is red. You open the log, scroll through two hundred lines, find a type error or a stale snapshot, fix it and push again. PR comments are the same story: small notes like “rename this variable” or “missing a null check here” pile up, and each one pulls you out of what you were doing.

This post explains how to hand that work to an AI agent: what triggers it, the guardrails it runs under, how it delivers, and what to look at before it reaches main.

Why fixing by hand is expensive

Fixing a red build is rarely hard. The context switch is what costs you:

• You’re in the middle of something else. The red CI notification pulls you off it.
• Reading logs takes time. The real error is buried in hundreds of lines, and the first red line is often not the real error.
• You have to reproduce it locally. Finding the CI command, running it and sorting out environment differences is its own job.
• Comments arrive in pieces. A review with six comments means reading all six and answering each one.

What these jobs share is a known set of steps. Work with known steps is work you can hand to an agent.

Triggers: when does the agent start?

In AgentVera, automations live under Details → Automation → Automations. A rule starts work on its own when an event happens. Two triggers matter here.

CI turned red

This trigger watches the providers you added in the CI/CD tab: GitHub Actions, GitLab CI and Jenkins. Leave Branches empty to watch all branches, or list them, like main, release/*.

Some runs are skipped on purpose. Runs for pull requests from forks and GitLab merge request runs don’t fire the rule. A fork’s branch and log belong to someone outside your repo, so the rule only reacts to branches you can push a fix to.

PR got a comment

New review and conversation comments on open pull requests are read every 2 minutes, with filters:

• Only comments from the repo’s owner, members or collaborators count. A drive-by comment won’t start an agent.
• Bot comments don’t count. AgentVera’s own replies are marked and never picked up again.
• If an agent is already working on that PR, new comments go to it; no second agent is opened.
• If the daily limit is reached, comments aren’t lost; they’re read again later.

PR comment and issue triggers need the repo’s origin on GitHub and a GitHub sign-in on your computer.

Events are watched by the desktop app, so AgentVera needs to be open on your computer for rules to run.

Presets

When a project has no rules yet, AgentVera offers presets. Two of them do exactly this job:

The other two do issues labeled “ajan” and update dependencies every Monday.

What the agent does

When a rule fires, the agent starts in a new worktree on a new branch that begins with ajan/. The branch is based on the event’s branch: the red run’s branch or the PR’s branch. Your working folder is left alone. Git-ignored .env files are copied into the worktree so tests can run, but they are never pushed.

The agent follows the steps in the rule’s Playbook. For CI, the default playbook is “Fix a CI failure”:

1. Find the first real error in the log.
2. Run the CI command locally and reproduce the error.
3. Fix the root cause; if the problem is in the CI config (workflow, version, environment variable), fix that.
4. Rerun the CI commands locally and see them pass.

For PR comments, the default is “Address PR comments”: read every comment and the code it points to, apply the ones that are right, explain your reasoning where you disagree, run the tests, and in the reply list what was done for each comment.

You can change playbooks. Save your own to this computer or to the repo; repo playbooks are written under .agentvera/playbooks/, shared with the team through git, and replace the built-in playbook of the same name.

Use Extra instructions to narrow the rule. For example: “Only fix tests and type errors; don’t change the API contract.”

In the Agent field, pick Claude Code or Codex.

Guardrails: what stops the agent?

An agent that runs on its own has to be limited, not just good. In AgentVera the limits are:

• A daily limit. Every rule has a Max per day value from 1 to 50, 5 by default, counted over the last 24 hours. Runs you start by hand don’t count against it.
• A per-branch limit. At most 2 automatic CI fixes are tried per branch per day, and only one fix per branch runs or waits at a time. If a fix breaks CI again, the agent doesn’t spin forever.
• Outside text is not instructions. CI logs, PR comments and issue text are handed to the agent as untrusted outside text that only describes the work. The agent is told not to share secrets, not to run unrelated commands and not to send data anywhere else.
• The agent doesn’t push. It only commits. Pushing and opening the PR are AgentVera’s job, done after the agent hands in.
• A secret file scan. Before pushing, every commit and the final state are scanned for secret files: .env files, keys like .pem and .key, id_rsa, .npmrc, .netrc, credentials.json and the like. Sample files such as .env.example are fine. If one is found, nothing is pushed and the run fails.
• A stuck agent stops. If the agent asks for help, goes silent or closes, the run moves to “needs attention” and you get a notification.

Delivery: where does the work go?

What happens when the agent finishes is set by On delivery:

One rule never changes: AgentVera never pushes directly to the default branch. If the agent made no changes, the run finishes with a “no changes” note.

For PR comments, the work is pushed to the PR branch and AgentVera posts a reply on the PR describing what the agent did for each comment.

Picking the merge queue means the fix is tested once more on top of the latest main. I cover how the queue works in the merge queue post.

Review before it reaches main

An automatic fix is not an automatic accept. There are a few ways to review the agent’s work:

• The pull request itself. Failures on main already come back as PRs. Read the diff, watch CI go green, merge it like any other PR.
• Approval in the merge queue. Land passing branches automatically is off by default, so work that passes its check waits for you.
• Leave on the branch. For the first weeks of a new rule, choosing Leave on the branch and reviewing by hand is the most cautious option.
• The Runs tab. See each run’s status, its agent and the PR it opened.

For your own agents, AgentVera also has automatic code review: when an agent’s turn ends, a Claude model you pick reviews the code and reports findings with severity and file:line. More in the AI code review post.

When you review, look closely at:

• Was the test fixed, or the code? If the agent changed an expected value to make a test pass, that’s a warning sign. You can forbid it in Extra instructions.
• Did the CI config change? A workflow edit can be legitimate, or it can be a skipped step.
• Did it stay in scope? A type error shouldn’t come back as a ten-file refactor.

One-click fixes from the CI/CD center

You may not want a rule for every red run. The CI/CD tab lists the newest 25 runs per provider; you can rerun a run, rerun only the failed jobs (not on Jenkins) or cancel it. On a red run, Have an agent fix it opens an agent with the run’s details and the log tails of its failed jobs. This time you’re right there with the agent.

Common mistakes

• Watching main with “Push to the same branch” and expecting a direct push. AgentVera doesn’t push to the default branch; the fix still comes as a PR.
• Maxing out the daily limit. The limit is a safety net. Keep it low until the rule earns your trust.
• Leaving Extra instructions empty. Telling the agent what not to touch narrows the results noticeably.
• Handing flaky tests to the agent. For a test that fails now and then, the agent may change things that don’t need changing. Deal with flaky tests yourself first.
• Keeping secrets in the repo. The scan stops the push, but the real fix is never putting secrets in the repo.

A sample day

Morning

Someone’s late-night commit to main broke CI. The “Fix CI when it breaks” rule caught it: the agent found the type error in the log, reproduced it locally and fixed it. Since it was the default branch, the work came back as a PR. You read the diff over coffee and merge it.

Noon

A teammate leaves four comments on one of your PRs. Within two minutes the “Handle PR comments” rule kicks in. The agent applies three, disagrees with one and explains why. AgentVera pushes to the PR branch and leaves a reply on the PR. All you do is read the one it disagreed with and decide.

Afternoon

CI keeps failing on a feature branch. The agent tries twice, and tests stay red both times. There is no third attempt; you see the status in the Runs tab. The log shows the problem is a test environment variable, so you fix it yourself.

Checklist

• Is your CI provider added in the CI/CD tab?
• For GitHub repos, are you signed in to GitHub on this computer?
• Is the Branches field right? Empty means all branches.
• Do the Extra instructions say what the agent must not touch?
• Is Max per day set conservatively?
• Does On delivery match how you like to review?
• Are your secrets kept out of the repo?

Wrapping up

Handing CI failures and PR comments to an AI agent gives back the most interrupted part of your day. What matters isn’t that the agent is fast but that it’s bounded: in AgentVera it works in its own worktree, doesn’t push on its own, never writes directly to the default branch, doesn’t push work that contains secret files, and gets a limited number of tries per branch. The final call stays with you, in the PR or the merge queue.

Read more:

• Automations
• CI/CD center
• Code review
• AI agent workflow pipeline
• Parallel coding agents

To try AgentVera, start at the home page.