AI Agent Scaffolding: Four Failures and What Prevents Them

Key Takeaways

• The problem: AI agents without proper scaffolding fail silently, misroute work, and make uncontrolled decisions — without anyone noticing until the damage is done.
• Why it matters: The harness, not the model, determines whether an agent is safe, reliable, and worth deploying.
• What you’ll learn: Four real failure patterns from a production agent system, and the specific harness components that prevent each one.

One of my agents was supposed to generate a workflow. Instead, it called the language model directly and produced a set of image prompts. Technically valid output. Completely wrong task. The API charged anyway. Nobody knew until I pulled the traces an hour later.

The model did not hallucinate. It executed exactly what it was capable of doing. The system had no control over which capability it was allowed to use. Nothing stopped it.

That is not a model problem. That is an AI agent scaffolding problem — and specifically, a missing harness.

Agent = LLM + Harness

An agent is not just a model. It is a model connected to an operating system.

I call this layer the agentic harness — the AI agent scaffolding that sits between the model’s reasoning and the real world. Anthropic describes these as systems where “LLMs dynamically direct their own processes and tool usage” — the scaffolding is precisely what makes that direction safe, controlled, and recoverable.¹

Think of the language model as the brain: it understands language, reasons about problems, proposes a next step. The agentic harness provides everything else — hands (tools and APIs), memory (context and session state), eyes (search results, files, tool outputs), rules (permissions, policies, approval gates), a workflow engine (planning, execution, retries, stopping conditions), and a flight recorder (logs, traces, metrics, outcomes).

The model decides what might need to happen. The harness determines how that decision is executed, monitored, and controlled.

Remove the harness, and you have a brain without a body — capable of reasoning, incapable of acting reliably.

Four Failures, Four Missing Components

I run a three-agent content pipeline. It has failed in four distinct ways. Each one taught me something a better harness would have prevented.

1. The Silent Cascade

Agent 2 stalled. No error surfaced. No signal went downstream. Agent 3 had no inputs, reported “not enough inputs to act upon,” and waited — indefinitely.

I did not know Agent 2 had failed. Agent 3 did not know either. The pipeline sat in limbo while I assumed it was running.

When an orchestration loop has no failure detection, a broken agent becomes an invisible one. The fix is not smarter agents. It is a harness that propagates failure signals so the system can decide: retry, escalate, or stop cleanly.

Missing component: Orchestration loop with failure detection and propagation.

2. The Timezone Trap

My agents communicated through a shared folder — predefined filenames, each agent reading what the previous one wrote. Clean in theory.

The CRON jobs were scheduled in Indian Standard Time. The execution environment ran on UTC. Six hours of drift meant jobs fired at the wrong intervals. Agents read files from the previous run — stale data — producing feedback that did not match the current cycle. Engagement windows were missed. The feedback loop between agents was misshapen.

The problem was not the timezone. It was the absence of a state contract: no agreed execution environment, no handshake to confirm inputs were fresh. A harness enforces this contract. File polling does not.

Missing component: Execution environment contract and state management.

3. The Dumb Executor

One agent depended on an email notification from a CRON job. When the job died, the email never arrived. The agent waited for a signal that was never coming.

A CRON job is a timer. It fires at a scheduled time regardless of whether the context makes sense. It has no intent, no awareness of what came before, and no recovery when things go wrong. It cannot tell downstream agents why it failed or how long they should wait.

A workflow engine is different. It carries intent — it knows the goal, tracks which step it is on, and knows whether the previous step succeeded. When something fails, it decides: retry with backoff, escalate, or terminate cleanly. Replacing CRON with a workflow engine is not a configuration change. It is an architectural one.

Missing component: Workflow engine — intent-aware, not just time-aware.

4. The Capable Agent That Did the Wrong Thing

Back to the opening. The agent had access to two capabilities: execute the provided script, or invoke the language model directly. Without tool scope control, it chose the wrong one.

The cost per call was small — approximately six times more expensive than the correct path. The output was 100% waste. But the dollar amount is not the concern.

The concern is this: none of it appeared in any log until I pulled API traces manually, an hour later. A single misconfigured agent, in a single run, in a small personal system. Extend that to hundreds of agents running thousands of tasks per day — systematic tool misrouting that nobody detects until a manual audit is a governance failure, not a cost line.

The model performed correctly. The system had no guardrails. Tool scope control is a specific form of guardrail — one that governs what an agent is allowed to do, not just what it is allowed to say. If you are new to the broader guardrails landscape, five production failure modes every AI deployment hits first is a useful companion read.

Missing component: Tool scope control per agent.

The Harness Is the Multiplier

Andrew Ng’s team measured GPT-3.5 on a standard coding benchmark: 48.1% accuracy with a single prompt. The same model, with an iterative agent loop: 95.1%. The model did not change. The scaffolding did. His conclusion: “The improvement from adding an iterative agent workflow dwarfs what you get from a model upgrade.”²

Agent effectiveness is not a function of model capability alone. It is a product of the entire system:

Agent effectiveness = model capability + tool quality + context quality + orchestration + memory design + safety controls + evaluation

A stronger model improves reasoning. It does not fix unscoped tools, stale context, silent failures, or invisible misrouting. The same model will appear far more capable — or far more dangerous — depending entirely on the harness around it.

---

Three Questions Worth Sitting With

1. Which of these four failure patterns exists in an agent system you are responsible for right now?
2. If an agent in your organization called the wrong API today — misrouted, misscoped, undetected — how long before anyone would know?
3. What is the difference between a CRON job that triggers your agent and a harness that runs it?

---

In the next post: I build a config-driven harness from scratch — stateless, observable, and designed to prevent exactly these failures. Building a Config-Driven Agentic Harness →