The 2 AM Bug That Isn’t a Bug
I have a three-tool AI pipeline. Tool A fetches a user profile. Tool B fetches recent activity. Tool C drafts a personalized email. Every tool has unit tests. They pass. Integration tests pass too.
Then I run the agent end to end, and about one run in ten, it fails.
Not with a crash. With a drift.
Tool A returns a user bio that happens to mention React Native. The LLM latches onto that phrase and invents a parameter for mobile activity when calling Tool B. Tool B returns an empty array. Tool C writes an email about a product that doesn’t exist.
Every tool worked correctly. The pipeline is broken.
I spent three hours debugging this like a normal software problem: tightening the prompt, adding validators, locking down the schema. Treating the system like a chain of logic.
That was the mistake.
The Fallacy
In neuroscience, the “grandmother neuron” is the idea that somewhere in your brain there’s one specific neuron that fires for one specific concept: your grandmother. It’s a seductive idea because it’s clean. One part, one job, one meaning.
Neuroscience moved past this decades ago. The brain doesn’t work that way. Concepts emerge from patterns across distributed networks, not from single dedicated cells.
We haven’t learned that lesson in AI engineering.
We build a tool called fetch_user_activity and act as if it has one fixed meaning. In isolation, it does. But once that tool’s output enters an LLM context window, it stops being just a function. It becomes part of a semantic field, shaped by nearby text, prior tool calls, naming choices, and everything else competing for the model’s attention.
The mistake isn’t building deterministic tools. The mistake is assuming they stay deterministic once a model starts reasoning over them.
Clocks and Clouds
Karl Popper had a useful distinction: clocks and clouds. Clocks are mechanical and predictable. You can take them apart and understand every gear. Clouds are irregular and context-dependent.
Software engineers are excellent at building clocks. A REST endpoint is a clock. A database transaction is a clock. We know how to test these because they behave the same way every time.
An LLM is a cloud.
Not random, but not mechanically predictable either. Small changes in wording, tool ordering, or prior outputs can shift the path it takes. So what did we do when these systems arrived? We tried to force clouds to behave like clocks. Rigid JSON. Strict schemas. The assumption that enough structure would make reasoning deterministic.
Structure helps. But it doesn’t solve the core problem, because the instability isn’t in the formatting layer. It’s in the reasoning layer.
That’s why model-facing systems need more than strict schemas. They need semantic clarity: tool descriptions that explain intent, not just parameters. Responses that preserve meaning, not only structure.
Variation Isn’t Always Noise
In conventional software, there’s one correct path and any deviation is a bug. That works for clocks.
It works less well for LLMs.
When the same prompt produces slightly different reasoning across runs, we treat it as unreliability. We lower the temperature, add constraints, tighten the corridor. Sometimes that’s right. But sometimes the variation is telling us the system is navigating a genuinely complex space, not tracing a single fixed route.
This becomes dangerous in tool chains. If your system depends on one exact wording, one exact parameter name, one exact sequence of intermediate states, you haven’t built a robust agent. You’ve built a narrow corridor and hoped the model never brushes the wall.
The better goal isn’t eliminating variation. It’s building interfaces that stay legible even when the model’s trajectory shifts.
The Butterfly Effect in Tool Chains
A tiny detail appears in one tool response. A slightly different phrase enters the context. Suddenly the model starts making calls that looked impossible in your tests.
This is the butterfly effect in LLM systems. Traditional APIs don’t work this way. If you pass the wrong ID, you get the wrong user, but the failure is local and the response format stays stable.
With LLMs, failures are non-local. One semantic nudge early in the chain can alter what the model pays attention to for every subsequent step.
That’s why brittle tool contracts fail so easily. If the model must hit one exact key name, one exact parameter pattern, one exact interpretation, a tiny context shift breaks everything.
Human-readable tool descriptions help because they preserve intent. “Provide the numeric user identifier” is more robust than a bare field name. The model can recover meaning even when local wording drifts.
Adding Tools Changes Everything
Here’s a reductionist habit that burns people:
I tested Tool A. I tested Tool B. So a system with both should behave predictably.
In an LLM environment, adding a tool doesn’t just add a capability. It changes the decision surface of the whole system. New words appear in the context. New actions become thinkable. Attention shifts.
I’ve seen an agent use insert_database_record correctly hundreds of times. Then I add a search_web tool, and suddenly it starts filling database fields with URLs instead of cleaned values.
The web-search tool changed what the model sees as relevant, available, and natural to output. One more tool doesn’t add one more branch. It reshapes the probability landscape for every decision.
This is why connecting dozens of tools to one agent isn’t like handing it a clean list of independent APIs. You’re changing the environment in which every choice gets made. Tool descriptions need to be clear, differentiated, and semantically grounded. Raw JSON schemas alone won’t keep things stable.
Reductionism Still Has Its Place
None of this means deterministic engineering is useless. It means it has a boundary.
A database write should be deterministic. A payment flow should be deterministic. Your backend tools should still be tested like clocks.
The failure happens when you assume the logic of the clock continues unchanged at the model boundary. It doesn’t.
Keep your backend precise and structured. But once that capability is exposed to an LLM, the model-facing layer needs to account for semantic interpretation: clearer descriptions, better discoverability, better differentiation between tools, and response formats that preserve intent, not just structure.
Narrative doesn’t replace engineering. Semantic framing helps deterministic systems survive contact with probabilistic reasoning.
The One-Sentence Version
Your tools are deterministic. Your agent is not. Design the boundary accordingly.