Welcome back to Building with AI. Most of what I read this week came back to one question: when an AI tool gives you a recommendation, how much of the final decision is still yours?
So much of the work that used to take real effort now arrives in a few seconds with a confident explanation attached. It can be useful, but I don't think we should hand over the judgment along with the legwork. Deciding what matters, weighing the trade-offs and owning the outcome is the part of the job that's ours, and a new experiment from Anthropic shows how easily it slips away, even from experts.
The Story
When AI weighs in, who's really making the call?
On October 27, Anthropic published an experiment with 50 professional analysts, each using Claude Code to advise a farming cooperative on whether to expand an advisory program. Along the way they had to make four judgment calls, each with two reasonable options. Without telling them, the researchers set each analyst's AI agent to favor one option on each call, decided by coin flip, while still making an honest case and never misleading.
Years of experience and self-rated expertise didn't predict who deferred, and the analysts who followed the AI most closely were just as confident it was their own judgment. Anthropic is careful about the limits: it was one task, and the AI's leanings were strengthened on purpose. The analysts using the agent also finished faster and made far fewer serious errors than the few who worked without it. Its conclusion is that organizations need to decide deliberately which decisions need a person's own informed judgment, and give people the time to form one.
A Google and University of Toronto study, Toolflation, shows the same pattern when AI agents choose on our behalf. When "seller" agents rewrote product listings with the single goal of being picked more often, they gradually claimed more than their tools could deliver. Under one of the five models tested, the listings blurred together until the "buyer" agents' success rate on Amazon data tasks fell from 89% to about 30%, close to random.
What I take from this is that there is no substitute for human discernment. When every decision is reduced to probability and rules, the value of relationships and discernment gets lost. MIT's Center for Information Systems Research offers a practical way to decide where people need to stay in charge. Its AI Decision Matrix sorts decisions by two questions: how ambiguous is the answer, and how costly is it to get wrong? Routine, low-risk calls are good candidates for automation, while ambiguous, high-stakes ones need a person leading. At One New Zealand, which the researchers studied, no AI agent goes live without a named business owner who is accountable for its results.
Sources: Anthropic · arXiv · MIT Sloan
The Rundown
"A finding, not a failing"
Amazon published a memo to its software engineers on October 27 discouraging "tokenmaxxing," using as much AI as possible to look productive. Measures like token usage and AI adoption rates are "deeply flawed," it says, because they can drive up costs and complexity when they become targets. What stood out to me is this line: "If your work does not benefit from current AI tooling, that is a finding, not a failing." Amazon now calls the role it wants the "Renaissance Developer," someone who blends AI tools with their own judgment and domain expertise.
Source: Amazon
Gartner's trends point to agent identity
Gartner's Top Strategic Technology Trends for 2027 predicts that by 2028, 25% of enterprise security breaches will trace back to AI agent abuse, because agents are being given access through old credential practices. Its answer is to give every agent its own identity and permissions, tied to the people accountable for it. The problem is already showing up: in a Cloud Security Alliance survey commissioned by security firm Zenity, 53% of organizations said AI agents had gone beyond their intended permissions, and 47% had an AI agent-related security incident in the past year. Another trend on Gartner's list, disposable applications built to be retired when their job is done, picks up on the build-or-buy question I wrote about last week.
Sources: Gartner · Cloud Security Alliance
A startup taking on ServiceNow
Serval raised $150 million at a $3 billion valuation on October 29, its third round in 12 months, led by Sequoia. Its agent lets employees in IT, HR, finance and legal describe a workflow in plain language and turn it into code-based automation, with admins controlling permissions and approvals. It's one of the clearer examples of software that lets non-engineers build, with guardrails built in rather than added later.
Source: Serval
A cloud built for running AI, not training it
General Compute, founded in 2025, raised a $100 million Series A on October 27, at an $800 million valuation according to DCD. Instead of relying only on Nvidia GPUs, it runs specialized inference chips from Cerebras, SambaNova and Positron AI and routes each workload to the hardware that suits it, with its first major deployments going live early next year.
Sources: General Compute · DCD
The Human Side
The thinking is still ours to do
David Tost's essay How To Become Insanely Focused argues that for knowledge workers, focus comes down to a decision about what you refuse to look at for the next couple of hours. Our work has no hard edge, so nothing forces us to stop and think unless we choose to.
I see this in my own work. I often feel as though I'm on the back foot when it comes to getting the most effective use out of my AI tools, because of the constant stream of tools and notifications competing for my attention, which makes it hard to set aside focused time for deep thinking. Our technology is conditioning us to be always on, believing we are multitasking and being highly productive when, in fact, we are working against ourselves.
AI can generate answers faster than any of us, but people are the ones who create. We're the ones who decide what's worth thinking about and who care about the people affected by a decision, and we can only do that well if we've set aside the time to actually think it through. Anthropic's researchers make a related point: judgment depends on skill, and people who stop making these decisions may lose the ability to make them.
For leaders, I think that makes attention something to look after, our own and our teams'. If AI is going to serve human flourishing at work, some of the time it saves has to go back into thinking, and into the people around us. Here's to finishing one impactful task a day over starting, and never finishing, a hundred less important ones.
Sources: The Second Brain · Anthropic
Try This Week
List three decisions your team now makes with help from AI, and put each one through MIT's two questions: how ambiguous is the answer, and what happens if it's wrong? For anything that comes out both ambiguous and costly, agree that the person responsible writes down their own view before asking the AI, and that every AI agent you run has a named owner.
One for your IT team
On October 29, a tampered copy of Anthropic's Claude Code extension sat in Cursor's extension marketplace for about 11 hours and tried to install a remote-access backdoor, Cursor said. Anthropic's own systems weren't breached. If your developers use Cursor, have them check for versions 2.1.312 or 2.1.313, update to 2.1.314, and follow Cursor's cleanup steps.
Source: Cursor
That's it for this week. When was the last time an AI tool changed your mind about something, and did you notice at the time? Hit reply and let me know.
References
- Anthropic, How AI agents shape experts' judgment calls
- Tang et al. (Google, Google DeepMind and University of Toronto), Toolflation: Seller Agents Distort Tool Markets Through Spontaneous Generative Engine Optimization
- MIT Sloan, A framework for determining when AI can make decisions
- Amazon, An update from Amazon engineering leadership on the role of the developer in the age of AI
- Gartner, Gartner Identifies the Top Strategic Technology Trends for 2027
- Cloud Security Alliance, More Than Half of Organizations Experience AI Agent Scope Violations
- Serval, We raised $150M to make everyone a builder
- General Compute, General Compute Raises $100M Series A
- DCD, Neocloud General Compute raises $100m in Series A funding round
- David Tost, The Second Brain, How To Become Insanely Focused (As A Knowledge Worker)
- Cursor, Security notice: malicious version of Anthropic.claude-code identified in the Cursor VS Code Extension Marketplace
