Running an AI Software Factory: Agents That Ship Production Code

At Navero we are a small team with a large roadmap. Over the past year we rebuilt our development workflow around AI agents that take a ticket from the tracker and deliver a reviewed, QA-verified pull request.

The Problem

Coding agents are good at writing code and bad at knowing when they are wrong. Left unsupervised, they:

• Declare success without verifying the feature actually works in a browser
• Review their own code with the same blind spots that wrote it
• Block silently on a question nobody sees, stalling work for hours
• Drift away from the ticket's acceptance criteria

Pointing an agent at a ticket is easy. Trusting the output enough to merge it is the hard part.

The Solution: A Pipeline With Gates

We modeled the workflow like a factory line. Every stage ends in a verifiable gate, and no stage trusts the stage before it.

Linear ticket
   |
   v
Implementation agent --> deterministic gates: lint, typecheck,
   |                     ~1,000 unit tests, e2e suite
   v
QA agent -----------> drives the real app in a browser (Playwright MCP),
   |                  logs in through the real OIDC flow, verifies the
   |                  acceptance criteria, commits an e2e spec as proof
   v
Reviewer agent -----> fresh context, never saw the code being written,
   |                  returns PASS or FAIL with line-anchored comments
   v
Human merge --------> the only stage allowed to touch production signals

Four design decisions did most of the work:

Fresh context for review. The reviewer agent never shares a session with the writer. An agent reviewing its own code inherits the assumptions that produced the bug; a clean-context reviewer does not.

Deterministic gates over model opinion. Tests, linters, and type checkers emit pass or fail. Wherever possible the pipeline branches on that truth rather than on a model's self-assessment.

Escalate silence, not every question. Agents pause for human input at defined checkpoints. A Slack notification fires only when a session has been blocked past a grace window — so humans hear about stuck work, not every routine question.

The board moves itself. Cards transition from git events through the tracker's GitHub integration, and a scheduled board audit runs as a GitHub Action that names an owner for every anomaly it finds. Nobody drags cards by hand.

Key Lessons

An agent's claim of success is not evidence. The QA agent exists because "the tests pass" and "the feature works" are different statements. Observed browser behavior beats inference every time.

Keep humans on the irreversible edges. Deploy triggers and preview-environment labels are human signals in our system. Agents are explicitly forbidden from setting them, because they control live infrastructure spend.

Determinism is the antidote to drift. The more gates that branch on real signals — exit codes, committed specs, CI checks — the less room an agent has to talk itself into "done."

Results

• Over 900 pull requests merged through the pipeline, hundreds of them agent-authored and human-merged
• Every agent-built UI feature lands with a committed Playwright spec proving its acceptance criteria
• Board transitions are fully automated from git events, audited daily — zero manual card moves