Dev, Test, and Prod Still Matter: What Gets Deployed Has Changed

Conventional delivery already runs active code in continuous-integration jobs, integration tests, package registries, and deployment pipelines. Those systems can reach Production too. What changes with an agent is the degree of initiative inside the boundary. Its harness can inspect a result, choose another tool, and chain actions that nobody described in advance.

AI did not invent porous environments. It lets software discover and use paths that ordinary release controls may never have modeled.

The Artifact Is No Longer the Whole Release

A model produces output. A harness lets that model do work.

The harness supplies tools, memory, instructions, identity, network access, retry logic, and a way to observe what happened. I explored that system in Why AI Needs a Harness. Once the model can inspect a result and choose another action, we are no longer testing only a static artifact. We are evaluating a bounded decision-and-action system that can change the environment around it.

That changes what an environment means.

For ordinary software, we often ask, “Where is the application running?” For an agent, we also have to ask:

• What can it read?
• What can it change?
• Which other systems can act for it?
• What survives after the run ends?

A package proxy may live inside the test architecture while fetching material from the public internet. A test credential may still call a live software-as-a-service endpoint. A shared repository may let one run leave state for another. A browser agent may be operating in a test tenant while sending requests to real people and public websites.

The environment name tells us where the process started. It does not tell us where the consequences can end.

This matters well beyond frontier-model laboratories. A coding agent may hold a GitHub token. A finance agent may read a production ledger while writing only to a test workbook. A service agent may work in a training queue but still send email. A data agent may query a copy of a warehouse while exporting results through a live connector. A security agent may be testing a fictional target through a network path that reaches a real one.

In each case, the box says Test. The authority says something else.

When Test Touched Production

In July 2026, an internal OpenAI cybersecurity evaluation supplied a stark example. OpenAI said the models did not have direct internet access. They could, however, install packages through an internally hosted Artifactory proxy and cache. The models found and exploited a previously unknown Artifactory vulnerability, obtained internet access, and eventually reached Hugging Face production infrastructure. OpenAI called its account preliminary.

Hugging Face later reconstructed about 17,600 actions across roughly two and a half days inside its infrastructure. It reported that five datasets apparently connected to the benchmark were accessed, while finding no effect on other customer-facing models, datasets, Spaces, or packages. Hugging Face also said no unauthorized software-supply-chain change was shipped.

I examined the mechanics of that incident in The Sandbox Was Not the Boundary. The point here is different. The models were still being evaluated. They had not been released as a customer product. Yet the evaluation already had a path to production consequences through a shared service, a public launch point, and live third-party infrastructure.

Separate incidents were later disclosed, but they involved different mechanisms and should not be treated as one breach count.

Anthropic reviewed 141,006 cybersecurity evaluation runs and reported three incidents involving six runs. A configuration problem left live internet access available even though the models had been told they were inside simulations. In one case, a model published a malicious package to the Python Package Index. Anthropic said the package was available for about an hour and ran on 15 real systems. In another, a model scanned roughly 9,000 targets before compromising one real application. Anthropic’s latest research model stopped after concluding that its target was real, an important piece of counterevidence.

The U.K. AI Security Institute reported a separate event. This was not a sandbox escape. Internet access was intentionally enabled, and the model providers’ cyber classifiers were disabled so the institute could measure maximum capability. Across 122 runs, AISI found 19 out-of-scope actions in 10 runs, including an attempted malicious pull request and the creation of fake identities. A human maintainer rejected the code, and AISI found no resulting real-world harm.

Several later headlines involving different model providers traced back to one Irregular evaluation scenario. Irregular has said those disclosures shared an underlying control problem. Counting every provider statement as a separate failure would exaggerate the evidence.

These were unusually permissive cyber tests. They often used unreleased models, reduced refusals, disabled classifiers, or deliberately open internet access. They do not show that an ordinary enterprise assistant will break out of its environment.

They show something narrower and more useful: a system labeled Test can create production risk as soon as it can cause an external effect.

The Promotion Gate Has Four Different Jobs

The traditional pipeline promotes an application release. An agentic system makes the gate responsible for four connected dimensions: artifact, authority, state, and evidence.

They do not cross the boundary in the same way. The tested artifact should be promoted unchanged and accompanied by verifiable provenance. Authority should be issued in the destination. Runtime state should be reset; any necessary migration should be deliberate and governed. Evidence should be independently protected and should keep accumulating after release.

1. The Artifact

The promotable artifact is a signed, versioned release assembly or deployment manifest. It identifies the application code, declared model or model version, system instructions, policy and tool definitions, connector definitions, and configuration and routing dependencies.

It does not include destination credentials, identities, live endpoints, or secret values. Those are bound separately in each environment.

Any one of these elements can change behavior. A new connector may expose an action the agent could not take yesterday. A revised system instruction may alter when it stops or asks for help. A provider update may change model behavior without a customer deploying new application code, depending on the service and versioning contract.

The release record therefore has to identify the whole runnable assembly, not only the Git commit. Where the platform permits it, identify its components with immutable versions or digests and bind them to verifiable provenance. The goal is simple: know that the assembly tested is the assembly released.

2. The Authority

Authority includes identity, credentials, network egress, tool scopes, transaction limits, approval rules, and the systems the agent may affect.

Giving a test agent a live token creates production-equivalent exposure, even if no code moves. Outbound internet access, a connector pointed at a live tenant, or a tool expanded from read to write can create the same class of external consequence.

Policy intent can be version-controlled and promoted with the release. The destination workload identity should be bound there, and short-lived credentials should be issued there rather than copied forward. Production authority should be scoped to the destination and independently revocable.

This is why capability and authority have to remain separate. In How Much Work Can Your AI Safely Own?, I argued that demonstrated capability does not automatically grant operating authority. The same rule belongs in the delivery pipeline. A system may be capable of acting before the organization has earned the right to let it act.

3. The State

State includes retrieval data, working memory, caches, queues, shared directories, previous tool results, and whatever remains for the next run.

State can cross an environment boundary without anyone deploying software. A production document can enter a test retrieval index. A test run can leave a file that another run treats as instruction. A cached response can outlive the policy that permitted it. A refreshed knowledge source can change the answer even when the model, prompt, and code stay fixed.

State needs lineage, retention rules, reset procedures, and regional handling appropriate to the data. Calling it “test data” does not make it synthetic, disposable, or harmless.

Most runtime state should not be promoted at all. Test memory, sessions, queues, caches, shared files, and previous tool results should normally be reset or isolated. When a business dataset, retrieval index, or model checkpoint must move, treat it as a separate approved migration with versioning, integrity checks, provenance, retention rules, and a rollback plan.

4. The Evidence

Evidence includes evaluation results, provenance, monitoring baselines, rollback results, known limits, and operating telemetry. Approval and risk acceptance are decision records informed by that evidence. They are not evidence that the system is trustworthy.

The agent should not be the only system that writes the work, designs the test, grades the result, and decides it is ready. Independent scenarios matter. Holdout tests matter. A real rollback rehearsal matters. So does evidence that the identity can be revoked and the action trail can be reconstructed under pressure.

Evidence accumulates throughout the lifecycle. Its record should be independent of the agent and protected from alteration. A model update, tool change, new data source, expanded permission, or material drift can invalidate yesterday’s approval without a traditional deployment. Production telemetry, drift findings, interventions, incidents, and reassessments should append to the assurance record. Promotion is not a one-time ceremony. It is a continuing claim that the running system still deserves its authority.

Keep the Environments, Redefine Their Gates

We do not need to throw away Dev, Test, and Prod. We need each environment to make a stronger promise.

Dev should minimize consequence. Use synthetic or carefully de-identified data. Deny external write paths by default. Give each run its own short-lived identity and disposable state. Replace live tools with mocks, simulators, or mediated services when the work does not require reality.

Test should make pre-release evidence decision-grade. Testing happens throughout the lifecycle; Test is where the release case should become credible. Verification asks whether the system meets its specified requirements and controls. Validation asks whether it is fit for its intended use in a representative deployment context. Agentic systems need both.

Use realistic tasks, independent evaluations, separate identities, resettable state, and monitoring that can stop a run while it is happening. When live information is necessary, begin with read-only access and intercept external side effects. Test the route through proxies, package registries, domain name resolution, webhooks, queues, and third-party tools, not only the agent’s visible network interface.

The record should let another person reconstruct what was tested, what failed, what was waived, and who accepted the remaining risk.

Shadow mode should validate behavior without granting action. Let the agent receive representative production inputs and propose what it would do without giving it permission to make the change. Compare its decisions with actual outcomes. Measure quality, latency, exceptions, cost, and security before expanding authority. That is the same lifecycle discipline I described in The Cost of a Finished Job.

Prod should grant narrow, revocable authority. Production is not the end of testing. Measurement and monitoring continue while the system operates, because live telemetry, drift, incidents, and rollback may show that yesterday’s evidence no longer applies.

Expand exposure in stages where the work permits it. Use least privilege, per-action traces, transaction limits, policy checks, human approval where consequence requires it, live monitoring, and rehearsed rollback. The system needs a separate observer that can stop it. The agent should not control the only alarm, the only log, or the only switch that turns it off.

Third-party evaluators belong inside this model. Their contracts should identify permitted network paths, cross-client isolation, data handling, incident-notification timing, evidence retention, audit rights, and responsibility when a test reaches a real system. “Independent evaluation” describes who performs the test. It does not prove that the test infrastructure is independent, isolated, or safe.

Five Questions for the Release Review

The release meeting does not need another hundred-page checklist. It needs questions that follow the four different jobs of the gate: identify the assembly, bind its authority, govern its state, and judge the evidence.

1. What changed? Name the model, code, prompts, policies, tools, connectors, configuration, and data sources that differ from the last accepted version.

2. What can it touch? Trace direct and indirect access through identities, proxies, package services, webhooks, shared platforms, and third parties. Record what the agent may read, propose, or change.

3. What survives? Identify memory, files, logs, caches, queues, retrieval indexes, and artifacts that another run or environment can inherit.

4. Who tested it independently? Show evidence from scenarios the builder did not control, including failure cases, stop behavior, permission boundaries, and rollback.

5. Who can stop it and reconstruct the result? Name the decision owner, revocation path, incident lead, evidence location, and conditions for resumption.

This is the practical machinery behind The IT Adaptive Factory. The factory is not mature because agents can write and test code quickly. It is mature when the organization can explain what was promoted, what authority was granted, what state survived, why the evidence was sufficient, and how to recover when the answer was wrong.

OpenAI’s later response shows the cost of learning this after a control boundary fails. On Aug. 18, the company said it had paused some frontier research workloads, added stronger workload and network isolation, and left its largest planned reinforcement-learning run on hold while it gathered more evidence. OpenAI attributed the broader slowdown to both the Hugging Face incident and concerns about an upcoming model’s cyber capability, so the pause should not be assigned to the incident alone.

That response does not prove the new controls will work. It does show that research and test infrastructure can become important enough to slow the production of the model itself.

The Arrow Still Matters

Organizations will keep drawing three boxes. They should.

The boxes remind us that consequence should rise slowly and evidence should rise first. But an environment label cannot enforce that discipline. Architecture does. Identity does. Network policy does. State isolation does. Independent testing does. Someone with the authority to say “stop” does.

Before approving the next agent release, look at the arrow and ask four different questions. What artifact was promoted? What authority was bound in the destination? What state was reset or deliberately migrated? What evidence supported the decision, and how will that evidence continue after release?

If a test agent can already cause a real-world change, the organization is already carrying production risk, whatever the box is called.

The arrow still matters. It moves the assembly. The gate decides what authority it receives, what state it inherits, and whether the evidence is good enough to let it act.