In late May this year, a New York lab called Emergence AI published the results of an experiment that has been quietly making the rounds among engineering leaders. They put five different AI models in charge of simulated towns of ten autonomous agents each, gave them the same laws and the same tools, and let them run for fifteen days without human interference. The results, reported by Fortune and several others, were not subtle. Anthropic's Claude built a stable democracy with zero crimes recorded. Google's Gemini kept its agents alive but logged more than six hundred crimes over the full run. OpenAI's GPT-5-mini world recorded only two crimes, and the agents failed to take survival actions, with the run ending within seven days. The Grok-governed world collapsed fastest. 183 crimes in 96 hours, ending in extinction inside four days.
I read that paper recently and the line that stayed with me was not the chaos in Grok's world. It was the researchers' observation that over long time horizons, agents stop following static rules mechanically. The instructions you gave on day one start to drift by day four. That is a deeply uncomfortable property if you are running an agent inside a real client engagement, where day four is just a normal Thursday.
The interesting question stopped being whether AI can do a thing. It is now whether AI should own a thing on its own, for how long, and with what kind of human checkpoint along the way. I used to conflate "can" and "should". Most of the small embarrassments I have shipped over the last two years trace back to that one piece of laziness in my thinking.
I find it helpful to walk the lifecycle and ask one question at each stage. How reversible is the decision, and how far does it sit from the client's business context? The further the decision is from a quick rollback, and the closer it is to something only the client truly understands, the more a human has to stay in the room.
Requirements
Requirements is the stage where I have seen AI flatter teams the most. A model will happily summarise an hour of interview notes and produce a tidy backlog. It looks like progress. What it cannot do is hear the pause when a head of operations says "we usually do it this way" and know that the pause means there is a politically expensive exception nobody wants to name. The Standish Group's CHAOS data has, for three decades, put unclear or shifting requirements among the leading causes of project failure. Misread the brief here and no amount of clever automation downstream rescues you. AI drafts; a human listens.
Architecture
Architecture sits in a similar place, for a different reason. Architectural choices encode assumptions about scale, team skill, budget, and what the business actually needs the system to do five years from now. Much of that reasoning lives outside the codebase the model can see, in heads, in conversations, and in the handful of design docs nobody has updated in eighteen months. The model can read what the system is, but not the reason it is that way, and architectural decisions are dominated by the reason. The codebase will eventually reflect whatever architecture we settle on, well or badly, so getting this part wrong shows up in every sprint that follows. I treat AI as a generator of options and a stress-tester of my own assumptions. A senior human signs the decision.
Design
Design is the stage where AI looks most seductive and trips teams up most quietly. A model will produce a perfectly competent screen in seconds, with sensible spacing and a tidy component hierarchy. Design preferences are very personal, and the designer in the loop has to clearly understand what the user likes and why, verify that the spirit of what is needed has been correctly prompted, and then verify that the intended outcome is correct, appealing, and effective. AI is excellent at exploring variations once the human work of understanding has been done. Let it accelerate the second half of design. Keep humans firmly in the first half.
Coding
Coding is where the temptation to delegate is strongest, and where review discipline matters most. Boilerplate, scaffolding, test stubs, the dull work of wiring a new endpoint, all of it is a fair candidate for letting the model run. The trap has a few different shapes. Without tight instructions, an agent will cheerfully go beyond what was asked and refactor files you did not want touched. With slightly different prompting it will take lazy shortcuts, skip an edge case, or stub out the hard part with a TODO and a confident-sounding comment. The Stanford user study by Perry and colleagues found that developers using AI assistants produced more security vulnerabilities while believing their code was more secure, and a large-scale analysis of public repositories found meaningful weakness rates in AI-generated snippets. Security is only one slice of the problem. In the engagements we have observed where a single person let the agent do everything, the finished product looks polished on the surface, and the code underneath has the same quiet rot an unsupervised junior would leave behind. Human review checkpoints are essential, covering correctness, performance, structure, and maintainability, with security as one item on the list.
Testing
Testing is the stage where I have shifted my view the most. Execution, regression suites, synthetic data, coverage reports, all of it can run autonomously. What humans must own is the acceptance criteria. The hardest bugs I have shipped were ones where every test was green and the behaviour was still wrong, because the spec was wrong. AI is excellent at answering the question you asked. Deciding which question to ask is still ours.
Deployment
Deployment is where the Emergence experiment really lands for me. If a model's behaviour drifts over a fifteen-day horizon in a sandbox, the production boundary is the last place to remove the human. The emerging guidance on agentic delivery, including recent playbooks for production environments, keeps coming back to the same rule. Let the agent prepare the release, run the checks, stage the artefact. A human walks through the gate. We treat the production boundary the way airlines treat a cockpit door. There is a procedure, and the procedure has a person in it.
Monitoring
Monitoring is a sensible place to let AI go further. Detection, triage, log summarisation, the first draft of an incident timeline, all of it is fine. The decision to roll back, to escalate to a client, to declare an incident publicly, stays human. Auto-remediation has its place inside a tightly scoped blast radius. A wrong automated rollback inside a client's environment is the kind of event that ends a relationship.
Documentation
Documentation is the underused one. API docs, changelogs, runbooks, internal explainers, all of these benefit from AI doing the first ninety percent of the work. Architecture decision records and post-mortems are different. Those are accountability artefacts. If a future engineer reads a document and asks "what were they thinking", the answer should be a person who was actually thinking.
If I had to compress the whole approach into one sentence, it is this. AI autonomy should scale with how reversible the decision is and how far it sits from the client's business context. Map your lifecycle against those two axes before your next project kicks off, and you will probably find one or two places where the boundary is currently drawn by accident rather than by design.
Where in your delivery lifecycle have you drawn the line, and was it a deliberate choice or one the tools made for you while nobody was looking?
