Why AI Agents Feel Scammy, Despite the Impressive Demos
Hint: AI Agents Are Sometimes Not the Right Tool for the Job
Lately, it’s become trendy to predict that software engineering will mostly be done by AI agents (*) working autonomously in the background, while human developers mainly manage and direct these agents (a view that Mark Zuckerberg and others support). This is a departure from the “co-pilot” mode of AI interaction (e.g. using ChatGPT, Cursor, etc), which requires humans to actively steer the AI output through small chat interactions.
This seductive vision of AI minions working 24/7 in the background feels inevitable, given the ongoing progress in reasoning LLMs (e.g. O3) and continued user growth for AI assistants. Clearly, people love working with AI assistants. But will they love handing the reigns to AI for important tasks, especially in enterprise settings?
In this post, I will argue that the industry is severely underestimating the difficulty of transitioning from the “co-pilot” regime to “autonomous AI agent” regime. The core issue is that greater autonomy doesn't necessarily mean better outcomes —especially when it undermines reliability and explainability down the line. In many cases, humans prefer to stay involved in key intermediate steps to maintain control and understanding.
Basically, here’s the challenge: for many complex tasks, detaching humans from the creative process with AI agents can be dangerous and costly. What do I mean by that?
If we create a big gap between human workers and work, and fill it AI agents, this creates a “knowledge vacuum” which deprives humans of rich contextual information about the final output. This increases the chance of workers forming a half baked knowledge of their company’s output. The price will be paid when the workers need to explain or troubleshoot the work down the line, or improve and evolve the output.
Of course, AI agents can provide metrics, thought traces, telemetry and so on to teach the human about the work, but I foresee this won’t be as easy as creating some dashboards or notifications.
It’s about making tradeoffs between reliability and organizational learning, versus upfront, “optical” productivity gains. Paradoxically, if the agent isn’t creating a sufficient gap, then it’s not as useful - birthing a conundrum.
This also implies that most “AI agent” adoption will be biased to happen inside predefined AI workflows or low-risk use cases, and they will be sold as “AI agents”. If a team wants productivity through automation, they will harness agentic behavior inside predefined workflows - while trying to actively “kill” autonomy to increase reliability. That’s human nature.
To expand this idea further, handing off the reins to autonomous AI agents is fundamentally awkward, assuming that humans are ultimately in charge of sign-offs, because:
Workers learn far less if you remove them from the creative process: Autonomous AI agents promise to free developers from fine-grained interactions (like with ChatGPT) and shift them into a review role. But without active participation in the creation process, workers can’t refine their understanding of the problem and requirements. They become less equipped to grasp the nuances of their own work.
Learning, steering the AI, and evaluating AI work are all related: Proponents of autonomous AI agent paradigm assume that workers can review AI outputs reliably, even if they are decoupled from the creative process. This can work for simple or familiar problems, but fails to hold true for ambiguous or complex tasks that require many judgement calls and implicit / explicit tradeoffs, which only get surfaced during the creative process.
Interpretability and explainability suffers with autonomous software engineering: With “autonomous software development”, a greater trust is placed on the AI to “do the right thing”, thereby obfuscating the complex thought process and tradeoffs that preceded the final solution. In the happy path, this is not an issue, but may backfire during troubleshooting or new feature development, when surface-level understanding of AI output no longer cuts it.
Providing a “safe and effective” environment for AI agents is much harder than it sounds: To get started with autonomous AI agents, companies must first do the grunt work of 1) curating all company data for AI agents’ consumption, and 2) providing the agents with programmatic access to take actions (e.g. Anthropic’s computer use or Cohere’s North are attempts at facilitating this “agent onboarding” process). The difficulty of this work is severely underestimated. For example, what does “relevant company data” even mean? Also, this legwork is not properly executed at the vast majority of F500 companies currently.
The point here isn’t to doom post on AI agents wholesale, but inform prospect AI agent buyers about the fallacies in the “hire an AI co-worker” narrative, which is how many startups are framing their products (probably in an effort to justify ridiculous pricing based on anchoring).
In the rest of this post, I will expand on these challenges of adopting autonomous AI agents. I’ll also make some predictions on how the AI agent market will shape up.
(*) By “AI agent”, I mean a system that leverages LLMs to make autonomous decisions to accomplish user-specified goals, with the ability to interact with external systems (tools). A cynical ELI5 would be: a script calling LLM APIs inside a while-loop, with some data integrations and API calls. See Anthropic’s definitions of AI agents and AI workflows, which I endorse.
Why Autonomous AI Agents Aren’t a Panacea for Knowledge Work
To see why autonomous AI agents are fundamentally disadvantaged versus “co-pilot mode”, let’s notice what happens during a typical ChatGPT session (or Cursor, Claude, etc), illustrated in the diagram above:
small interactive loops (HIL) produce learning (rewards): often, workers start out with a description of desired output, but the original prompt rarely contains everything required to reproduce the final output. The prompt itself is dynamically changing, as the user “learns” by observing and evaluating the intermediate AI outputs. The final prompt itself ends up being a “reflection” of the human worker’s learning about the task at hand. This refined learning is not available if AI agents are making all the intermediate executive decisions.
HIL allow for course correction: when working with AI, there are a lot of dead-ends. But dead ends also produce insights about the problem constraints. With AI agents running the show, you don’t get exposed to counterfactuals (e.g. what doesn’t work, what alternatives have we considered).
HIL allow for injection of unwritten tribal knowledge: much of valuable operating knowledge is often trapped inside workers’ head, and not available for RAG (retrieval augmented generation). We call that “tribal knowledge”, “pet theories”, “experience”, etc. This is not digitized and therefore unavailable to AI agents by definition, but often they are necessary to get things done, especially in large, old organizations running ancient systems. Autonomous AI agents have no chance at absorbing this into its context.
HILs allow for natural explainability and easier troubleshooting: since the human was evaluating intermediate steps, by definition, explainability comes out of the box, as well as trace (the thread itself). When there’s an issue down the line, one could go back to the thread to audit any faulty assumptions, etc.
A more realistic sequence of AI collaboration looks like below.
Typically, solving a problem requires multiple threads that happen over time, each with different number of intermediate steps (chats). That’s why ChatGPT and Claude both have “projects” to group related threads. Each thread contributes to the user’s understanding of what works and what doesn’t (darker green means higher quality AI output).
When the first thread (thread 1) outlives its usefulness or context window limits get in the way, we transport our learning and bake into a fresh prompt in a new thread (thread 2), which allows the AI to start from a more favorable initial condition. We then spend some time reflecting, until we develop enough comfort to move forward.
In effect, co-pilot process allows for complex steering of AI toward the end goal, as well as multiple “escape opportunities” when we are stuck in local minimas. But autonomous AI agents completely mask this level of granularity - which may or may not be important - depending on the importance of the task. Performance suffers in theory because humans aren’t able to provide “timely” feedback.
Passing the Buck to Evals
So what happens if we move away from the co-pilot mode to autonomous mode? Unless the task itself is either simple or easy to evaluate, I argue that any optical productivity gains from AI agents come at the cost of “passing the buck” to the evaluator (the human). Unfortunately, evaluation becomes harder when the human is removed from the creative process. This is especially so for long-running AI agents that perform complicated tasks.
The longer an AI agent runs on its own, it naturally makes more micro decisions based on more assumptions. This reasoning path is largely masked, mainly because it requires too much bandwidth to evaluate both the final output and the thought process simultaneously. This leads to potentially missing out on some key cognitive errors committed by the agent:
incorrect assumptions (orange): AI agent assumes something that isn’t really true (e.g. AI decided it needs to overengineer something which doesn’t gel with existing codebase).
unknowable judgement calls (blue): AI agent assumes that it’s ok to do something that’s not explicitly forbidden, but undesirable (e.g. AI agent picks an outdated SDK version which wasn’t strictly forbidden)
unobservable hidden decisions (black): AI agent makes some hidden decisions that aren’t directly observable, but may cost down the line (e.g. AI agent decides it’s okay to not check for locale when displaying text, but this becomes a problem when deploying to Right-To-Left text countries, etc)
Examples I provided may seem innocuous but it should be easy for you to think of what assumptions would hurt your codebase or work. It’s difficult to parse these decisions from outside-in, if the human worker is just a passenger in the creation process.
Note, all these observations apply to other knowledge work. If you are writing some long report with the help of ChatGPT, don’t just copy paste - it might be assuming some things that aren’t really true. This has nothing to do with the LLM’s reasoning ability, since garbage in, garbage out holds true. If the bar for evaluations is low, well, good for you.
Where Autonomous AI Agents Belong
Thus, the real challenge of building autonomous AI agents is to then come up with the right “cadence” to “loop in” the human, such that they maximize human’s understanding of how the sausage is made, while also saving time. I argue this will remain challenging for a while, but is something I’m actively working on / thinking about (work with me).
What’s challenging is that this “cadence” is highly personal. Some people grasp info faster and learn more from less info, while others need hand holding. Thus, this is not a trivial problem.
In the meantime, I think AI agents shouldn’t try to be autonomous AI agents. Luckily there are already some proven, useful patterns. For example, agents are perfect for prototyping or exploratory use cases (e.g. prototyping, doing resource gathering, etc), and for serving as “decisioning gate” components to automation workflows (e.g. tier-1 and tier-2 customer support). These use cases collectively are large opportunities on their own, and the best fit for AI agent startups, rather than trying to misrepresent their value by comparing themselves to human workers on a job description basis (and failing).
Below diagram illustrates a simple framework for deciding among co-pilot mode, autonomous AI agent, and AI workflows. Generally speaking, leave the complex and highly ambiguous tasks to humans. It almost seems like common sense, but clearly we need a full Substack post to offset the recent AI Agent hype.
Note, if your job entirely consists of following a standard operating procedure that’s spelled out 100% (which admittedly a lot of white collar workers fall into) in a document, then the case of “autonomous AI agents” replacing you may be stronger.
I will save my detailed views on good AI agent use cases for a future post.
Predictions for the AI Agent Market (and Jobs)
There’s a “tug of war” between humans’ desire to be involved in the creative process, and the desire to save time. For enterprise use cases, I believe the former will win in most cases, since reliability and trust are more important than speed. That said, I am bullish on general “agentic behavior” straightjacketed into procedural programs (”AI workflows”), or the use of exploratory AI agents as plugins inside SaaS or coding IDEs doing well-understood tasks.
Also, I do think that AI agents - while they don’t replace humans directly - will still chew off a lot of small grunt work, and overall act as ongoing headwind for tech job market.
I’ll conclude this post with some predictions I have for the AI agent market in 2025:
AI agents will continue to confuse buyers due to poor messaging (please forward this post to anyone who’s confused): the tech industry seems hell-bent on selling any software that uses LLMs as “AI agents”, even if it does little autonomously and basically indistinguishable from a script or workflow. I expect this to continue because positioning as AI agents allows you to charge more via creating an anchoring on human labor costs. However, I think the truth will also eventually prevail, and customers will progressively become allergic to AI agent messaging.
Companies will always prefer AI workflows over autonomous AI agents: autonomous behavior is not a “feature” - it’s more like an “alternative”. If autonomy can be eliminated, companies will be biased to eliminate it to increase the reliability of the program. Thus, it makes no sense to put “autonomy” in the value proposition.
AI agent startups will struggle to compete with point solutions, at least enterprise: There are several AI BDR, AI SDR startups, etc, that position themselves in the “AI workforce” category, but I am unconvinced that buyers will pay enterprise prices for a glorified Slack bot. Existing CRM or marketing automation companies are theoretically in a much better position to offer these AI agents as a complement.
B2C AI agent market will be very tough, and if anyone wins it, it’s Apple + OpenAI (iOS), or Google: I don’t think niching down works here, because agents are only useful if they are generally useful.