DevOps for AI Coding Agents: Measure Verification Capacity Before You Rebuild Your Platform

Coding agents can increase the number of proposed changes faster than most teams can review, test, deploy, and verify them.

That does not automatically create a problem.

It becomes a problem when the rate at which changes enter the delivery system exceeds the rate at which the organisation can produce enough evidence to release them safely.

This distinction matters because many teams will respond to agent adoption by adding more CI capacity, more reviewers, or more approval rules. Those measures can help, but only when they address the actual constraint.

Before redesigning your platform, measure whether verification has become the bottleneck.

The relevant change is the arrival rate of proposed work#

The quality of agent-generated code receives most of the attention. Delivery systems are affected by a simpler mechanism.

Agents reduce the time required to produce a plausible change.

A developer who previously completed one implementation at a time may now:

• Delegate several small tasks concurrently.
• Generate alternative implementations.
• Open more pull requests.
• Respond to review comments faster.
• Attempt work that previously remained in the backlog.

Even when the average quality of each change remains constant, the volume entering review and CI can increase.

The delivery system now has to evaluate more changes with roughly the same:

• Senior review capacity.
• Test infrastructure.
• Staging environments.
• Release coordination.
• Operational attention.
• Understanding of dependencies.

The constraint has moved only when this additional demand starts increasing waiting time, rework, or production risk.

Some teams will experience this quickly. Others may not experience it for years.

Which teams should care now#

This problem is most relevant to teams that already have:

• Several developers or agents producing changes concurrently.
• Shared services or cross-repository dependencies.
• Slow or unreliable CI.
• Scarce senior reviewers.
• Manual deployment approvals.
• Long-lived staging environments.
• Releases containing several unrelated changes.
• Significant production or compliance risk.

A 10-person product team deploying a modular monolith several times per day may be able to absorb a large increase in code production without changing its platform.

A 100-person scale-up with shared services, contract dependencies, slow integration tests, and several approval stages may see the review queue grow after a modest increase in pull-request volume.

The difference is not enthusiasm for AI. It is the amount of spare capacity in the delivery system.

Define verification capacity before trying to increase it#

Verification capacity is the rate at which a team can produce enough evidence to move a change safely to its next stage.

That evidence may include:

• Human review.
• Automated tests.
• Security checks.
• Contract compatibility.
• Deployment approvals.
• Canary results.
• Production metrics.
• A tested rollback path.

The important phrase is enough evidence.

A documentation change does not require the same process as an authorization change. Running the maximum possible verification on every pull request limits throughput without necessarily reducing risk.

A simple model is sufficient:

backlog growth = change arrival rate - verification completion rate

Let:

• λ represent changes submitted per day.
• μ represent changes verified per day.

When λ remains below μ, the system has spare capacity.

When λ remains above μ, work accumulates in review, CI, staging, or release queues.

For example, suppose a team can reliably verify 25 changes per day.

Before agent adoption:

changes submitted = 20
changes verified = 25
spare capacity = 5

After adoption, the team submits 35 changes per day but still verifies 25.

changes submitted = 35
changes verified = 25
daily backlog growth = 10

The example is hypothetical, but the mechanism is not.

Within a week, 50 additional changes are waiting somewhere in the delivery process. Reviews become shorter, branches become stale, CI runs repeatedly, and teams begin grouping changes into larger releases.

The productivity gain has not disappeared. It has accumulated as work in progress.

Measure the bottleneck for four weeks#

Do not begin with a new internal platform.

Collect a four-week baseline using data from your source-control and CI systems.

Track:

1. Change arrival rate
   How many pull requests become ready for review each day?
2. Verification completion rate
   How many reach the point at which they are approved and deployable?
3. Queue age
   How long do changes wait for review, CI, an environment, or approval?
4. Rework rate
   How often does a change return to implementation after review or failed verification?
5. Verification cost
   How much reviewer time and CI time does each class of change consume?
6. Delivery outcome
   Did higher change volume improve deployment frequency or customer lead time, or did it only increase open pull requests?

Separate the data by change type where possible:

• Documentation.
• Tests.
• Local application logic.
• Shared libraries.
• API contracts.
• Database migrations.
• Infrastructure.
• Security-sensitive code.

This prevents low-risk generated changes from hiding saturation in the parts of the system that require experienced review.

A useful threshold is not universal. The team should intervene when increasing arrival rate causes a sustained increase in queue age, verification cost, or escaped failures.

Existing controls still work#

Most teams do not need a new category of infrastructure immediately.

The current controls remain useful.

Human review#

Human review is effective when architectural context, product behaviour, or operational history matters.

It becomes a bottleneck when every change requires the same scarce reviewers.

The solution is not to remove review. It is to reserve qualified reviewers for changes that require their judgement.

Full CI suites#

Running every test is reasonable when the suite is fast, reliable, and affordable.

It becomes inefficient when a small change triggers hours of unrelated integration tests.

More compute can reduce waiting time. It does not improve test relevance.

Ownership rules and service catalogues#

Static ownership and dependency information is cheap to understand and operate.

It works well for known, stable boundaries.

It fails when runtime dependencies, shared databases, events, or common libraries create coupling that the catalogue does not represent.

Feature flags and canaries#

Progressive delivery reduces the number of customers exposed to a failure.

It works when teams know what to monitor and can reverse the change quickly.

A canary without a promotion rule is merely a smaller production deployment.

Observability#

Metrics, logs, and traces help determine what happened after release.

They are less useful when operators must manually reconstruct which deployment affected which path and which signals should have changed.

The practical opportunity is to connect these controls around a shared description of the change.

Add a verification plan to each pull request#

The smallest useful improvement is a machine-readable verification plan.

It should describe:

• What kind of change this is.
• Which services may be affected.
• Which checks should run.
• Which approval policy applies.
• How the change should be released.
• Which production signals should be inspected.
• How the change can be reversed.

For example:

summary: Add optional discount codes to checkout requests

change_types:
  - public-contract
  - user-visible-behaviour

affected_services:
  - checkout-api
  - payments-api

required_checks:
  - checkout-unit
  - checkout-integration
  - contract-compatibility

risk_tier: 4

rollout:
  strategy: canary
  initial_percentage: 10

expected_signals:
  - checkout_success_rate
  - checkout_p95_latency

rollback:
  method: disable-feature-flag
  target: checkout.discount_codes

The plan is not proof that the change is safe.

It makes the assumptions explicit.

A reviewer can identify a missing dependency. CI can check whether the required tests ran. The deployment system can choose the rollout policy. The release owner can inspect the expected signals rather than opening a generic dashboard.

For agent-generated changes, the agent can propose the plan. It should not be the only authority approving it.

Generate the first plan from hard-coded rules#

Do not wait for a complete dependency graph.

Start with repository paths and the failure modes the team already understands.

{
  "rules": [
    {
      "glob": "services/checkout/**",
      "services": ["checkout-api"],
      "tests": [
        "checkout-unit",
        "checkout-integration"
      ],
      "risk": 2
    },
    {
      "glob": "contracts/checkout/**",
      "services": [
        "checkout-api",
        "payments-api"
      ],
      "tests": [
        "contract-compatibility"
      ],
      "risk": 4
    },
    {
      "glob": "db/migrations/**",
      "services": [
        "checkout-api",
        "payments-api",
        "ledger"
      ],
      "tests": [
        "migration-dry-run",
        "database-integration"
      ],
      "risk": 5
    }
  ]
}

A CI script can compare the branch with its base commit, match the changed files against these rules, and generate a verification plan.

The implementation can be simple:

import fnmatch
import json
import subprocess
from pathlib import Path

base_sha = subprocess.check_output(
    ["git", "merge-base", "HEAD", "origin/main"],
    text=True,
).strip()

changed_files = subprocess.check_output(
    ["git", "diff", "--name-only", f"{base_sha}...HEAD"],
    text=True,
).splitlines()

rules = json.loads(
    Path(".delivery/rules.json").read_text()
)["rules"]

services = set()
tests = {"lint", "unit"}
risk = 1

for filename in changed_files:
    for rule in rules:
        if fnmatch.fnmatch(filename, rule["glob"]):
            services.update(rule.get("services", []))
            tests.update(rule.get("tests", []))
            risk = max(risk, rule.get("risk", 1))

plan = {
    "changedFiles": changed_files,
    "affectedServices": sorted(services),
    "requiredChecks": sorted(tests),
    "riskTier": risk
}

Path("verification-plan.json").write_text(
    json.dumps(plan, indent=2) + "\n"
)

This will not discover every dependency.

It still creates value because the organisation has made its current assumptions inspectable and version-controlled.

Run the workflow in observation mode before making it a merge gate.

For each change, compare the generated plan with:

• The tests reviewers requested.
• The teams eventually consulted.
• The systems affected in staging or production.
• The signals inspected after deployment.
• The cause of any rollback or incident.

Update the rules from this evidence.

Use risk tiers to protect scarce capacity#

A risk tier is a policy category, not a prediction of failure probability.

Its purpose is to vary verification cost according to consequence and uncertainty.

This protects senior review time and expensive test infrastructure for the changes that need them.

It also makes the organisation’s risk policy visible.

Without tiers, teams often rely on informal escalation. Experienced engineers notice dangerous changes and intervene. That works until change volume rises or the relevant engineer is unavailable.

Keep agent-generated changes small#

Verification becomes harder as a pull request combines more independent behaviour.

An agent may be able to:

• Change an API.
• Update its consumers.
• Add a migration.
• Reorganise a package.
• Rewrite tests.

That does not make the combined pull request easier to verify.

Large changes increase:

• The number of possible failure causes.
• The affected dependency set.
• The amount of context required for review.
• The chance that unrelated work blocks the release.
• The difficulty of rollback.

The useful unit of agentic delivery is the smallest independently verifiable change, not the largest task the agent can complete.

This is usually a better first control than a sophisticated AI reviewer.

Know where the simple system fails#

A path-based system has clear limitations.

It will miss:

• Runtime-only dependencies.
• Shared database coupling.
• Dynamically selected consumers.
• Customer-specific paths.
• Critical behaviour hidden in common libraries.
• External systems.
• Important changes that modify only a few lines.

It can also produce false confidence.

An empty dependency list may mean:

• No dependency exists.
• No dependency matched the rules.
• No reliable dependency data was available.

These are different states.

Represent uncertainty directly:

dependency_confidence: low
reason: No runtime dependency evidence is available for checkout-api

A delivery system should not convert missing information into a green result.

What a stronger system requires#

A more complete verification system becomes justified when the manual rules require frequent maintenance, important dependencies remain invisible, or verification queues continue growing despite risk-based policies.

The stronger system should combine:

• Declared change intent.
• Repository and build graphs.
• API and event contracts.
• Infrastructure configuration.
• Ownership data.
• Runtime traces.
• Deployment history.
• Incidents and rollbacks.

These inputs should produce an evidence plan, not an unexplained risk score.

The output should answer:

• What is likely to be affected?
• How confident are we?
• Which checks are relevant?
• Who needs to review the change?
• How should it be exposed?
• Which signals determine promotion or rollback?

Runtime data is especially useful because static architecture rarely captures the entire system.

It should still be treated as evidence rather than truth. Traces may be sampled. Low-volume paths may not appear. Scheduled and customer-specific behaviour may remain invisible.

The long-term model should combine several incomplete sources and expose the resulting confidence.

Measure whether the change improved delivery#

After introducing verification plans and risk tiers, compare the next four weeks with the baseline.

Look for:

• Reduced review waiting time.
• Reduced CI time for low-risk changes.
• Stable or lower change failure rate.
• Fewer unnecessary senior approvals.
• More accurate identification of affected services.
• Faster rollback decisions.
• More deployments reaching customers.
• Lower verification cost per successful deployment.

Do not declare success because the team merged more pull requests.

The change is useful only when it improves the rate at which independently valuable work reaches customers without increasing operational risk or long-term maintenance cost.

The practical decision#

Most teams do not need to rebuild their delivery platform because they have started using coding agents.

They should first determine whether agent adoption has created a measurable verification constraint.

The sequence is:

1. Measure change arrival, verification throughput, queue age, and rework.
2. Separate low-risk volume from changes requiring scarce expertise.
3. Add a verification plan to each pull request.
4. Generate the first plan from simple, version-controlled rules.
5. Use risk tiers to select tests, reviewers, and rollout policy.
6. Improve the model using incidents, rollbacks, and runtime evidence.
7. Build a richer platform only when the simpler workflow stops being sufficient.

The near-term goal is not autonomous delivery.

It is to prevent faster implementation from creating larger queues, weaker reviews, and slower customer delivery.

Coding agents increase the amount of work your delivery system can receive.

Verification capacity determines how much of that work becomes safe, useful software.