The first wave of generative AI was a question and an answer. Somebody typed a prompt, a model replied, and a person read the reply and decided what to do with it. Whatever went wrong, a human was standing between the model and the consequence.
The wave now arriving is different, and the industry’s word for it is orchestration. A model reads a request, breaks it into steps, calls a tool (a search, a calculator, a database query) or another model for each one, checks the results, and retries what failed. Some of these systems run for hours. Increasingly the orchestrator, the software coordinating all of this, also chooses which model handles each step, trading capability against cost. It splits a large project into subtasks, sometimes several levels deep, and runs them across parallel sessions (each a separate conversation with a model, working on its own piece), then waits for each to return and assembles what they produce. Most of the work happens without anyone reading the intermediate output.
That is mostly being discussed as software engineering plus prompt craft, the skill of wording instructions for a model, and both matter. But the moment one step’s output decides what the next step does, the system has stopped being a pipeline, a fixed sequence of steps. It is a feedback loop. Feedback loops have a discipline of their own, older than computing, and it is worth bringing to these systems on purpose rather than rediscovering it one incident at a time.
I trained as a biomedical engineer. The core coursework was signals and systems, the study of how a system responds to what goes into it, and control theory, the study of how feedback keeps a system where you want it. The body is among the most elaborate control systems anyone studies. Blood pressure is held steady by sensors in the neck and chest that report to the brainstem, which adjusts heart rate within a beat or two and how tightly the blood vessels constrict over several seconds. Stand up too quickly and the brief light-headedness you feel is that loop catching up. I have spent much of this year building software with AI agents (models that carry out multi-step work on their own, using tools) that are performing an increasingly large share of the work, and the failures I keep meeting are ones that coursework named decades ago.
What changes when the loop closes
Take an example from the industry I currently work in. A manufacturer wires up an agent to turn dealer emails into quotes. One model reads the email and the attached drawing and drafts a quote. A second model checks the draft against the pricing rules. If the check fails, the draft goes back to the first model with the reasons, and it revises. The loop repeats until the check passes or a counter runs out.
In code it is a simple loop that repeats until a condition is met, and it looks just that harmless. In control terms it is a closed-loop controller. The first model is the plant, the thing being controlled. The second is the sensor. The pricing rules are the setpoint, the target the loop steers toward. The orchestration logic, which decides what to send back and how hard to push, is the controller. And the time each round trip takes is delay.
Naming the parts is worth the trouble, because each part tends to fail in its own way. People who have spent careers on chemical plants, aircraft and implanted devices have catalogued those failures carefully.
You cannot control better than you can measure
One of the first things a controls course teaches is that a loop is only as good as its sensor. A thermostat with a thermometer reading three degrees low will hold the room three degrees warm, reliably, forever. The loop is working exactly as designed. It is regulating to what it can see.
Orchestrated AI has the same property, and it is the one most often missed. If the checking model can confirm that a quote is complete, well formed and internally consistent but cannot tell whether the price is right, the loop does not produce right prices. It produces quotes that pass the check. Every retry pushes the draft toward what the sensor rewards, and the result converges, confidently and with a clean audit trail, on well-formatted wrong answers.
In most of the AI systems I have reviewed, the checker is the least examined part, and it decides almost everything about the result. Before asking how capable the generating model is, ask what the checking step can actually see and what it cannot. Then ask how anyone would find out when it is wrong.
Shared mistakes do not average out
A common way to raise reliability is to run several models on the same task and take a vote, or have one model review another’s work. That works on independent noise. If five measurements each wander randomly, their average wanders less, which is why instruments sample many times and average.
The same holds for models. Asking one model the same question several times and taking the majority answer does help when its mistakes vary from run to run, and research on reasoning tasks bears that out. It does nothing for error that the answers share. Five copies of the same model, given the same prompt, share the same blind spots. When one confidently misreads a drawing, the others are likely to misread it the same way, and the vote comes back unanimous. When the errors are shared, a unanimous vote carries little more information than one answer, while looking like far more.
I have watched this happen with my own tools. Several agents worked in parallel on separate parts of one software change, and each reported no conflicts with the others. The verdict was unanimous, and it was wrong for the same reason in every case: each was right about what it could see, and none of them could see the others’ work. The conflicts were there.
Engineers who work on safety systems call this common-cause failure and handle it with diversity: redundant sensors built on different principles, so they do not fail together. The orchestration equivalent is a check that does not share the drafting model’s weaknesses, such as a fixed rule that always gives the same answer for the same input, a replay against known history, a different kind of model, or a person. One independent check is often worth more than a committee of copies.
Gain, delay and stability
Every controller decides how hard to correct. That is its gain, and too much of it makes a loop oscillate. Turn a shower’s temperature knob too far each time it feels wrong and you swing between scalding and cold.
Retry loops have gain too. A loop that sends back a full critique and asks for a complete rewrite is correcting hard. The rewrite fixes the flagged line and disturbs three others, the next check flags those, and the system bounces between two wrong answers without settling. Anyone who has watched an agent alternate between two broken versions of the same piece of code has seen a sustained oscillation, the kind control engineers call a limit cycle. The usual fix is an iteration cap, a limit on the number of attempts. That limits the damage but does nothing for stability, and a loop that hits its cap has not settled on an answer. It has run out of attempts, and the output is whatever it happened to be holding at the time. The control fix is lower gain: smaller, bounded corrections, changes confined to what the check actually flagged, and a check that narrows as the draft improves.
One of the most famous films in engineering shows what runaway oscillation looks like at full scale. If you have not seen it, or not for a while, the footage runs about two and a half minutes and is worth watching before reading on. In November 1940, four months after it opened, the Tacoma Narrows Bridge in Washington State twisted itself apart in a steady wind of around 40 miles an hour.
The bridge had misbehaved from the start. Its deck bounced up and down in moderate winds from the time it was being built, which is how it came to be called Galloping Gertie (Washington State’s history of the bridge tells the story). Resonance may well have played a part in that bounce. Wind flowing past a solid deck sheds swirls of air at a steady rhythm, an effect engineers call vortex shedding, and when that rhythm matches one of the deck’s natural frequencies, the rate at which it tends to sway, each swirl pushes in time with the motion, like a parent pushing a child on a swing. Engineers still debate how much of the bounce that explains; the original federal investigation instead put it down to gusty, turbulent wind.
The bounce, though, is not what brought the bridge down. On the final morning the motion switched from bouncing to twisting, and the twisting tore the deck apart. Physics textbooks long blamed the collapse on resonance as well. Engineers who studied it concluded otherwise, most clearly Yusuf Billah and Robert Scanlan in a 1991 paper written to correct the textbooks. The wind was steady, and the bridge’s own twisting changed how the wind caught the deck, so each twist drew in the energy for the next. That is aeroelastic flutter, a self-excited oscillation, and it is the better lesson here. Nothing outside the bridge pushed in rhythm. Its own motion fed back into the force acting on it, with too little damping (whatever absorbs motion, the way a car’s shock absorbers do), and the motion grew until the deck failed. A retry loop that answers its own last mistake with a bigger correction follows the same pattern.
Delay makes all of this worse. The longer the gap between an action and the measurement of its effect, the less gain a loop can tolerate before it goes unstable. An agent that works for an hour before anything checks its output is a loop with an hour of delay. By the time the error is seen, a great deal has been built on top of it. That points to a rule for how often to check: check at least as often as the work can drift, and sooner when a mistake would be expensive to unwind.
Software teams know this problem by another name: work in progress. Every change that is built but not yet released is an action whose effect nobody has measured. Little’s Law, a basic result of queueing theory, turns that into arithmetic: average lead time equals the work in flight divided by the rate at which it finishes, so doubling the work in progress at the same finishing rate doubles how long each piece waits for its verdict. That is why DevOps, the practice of joining software development and IT operations into one continuous process, favors small, frequent releases. It is why teams using Kanban boards, which track each piece of work as a card moving across columns, cap how many cards can be in progress at once. And it is why DORA, the DevOps Research and Assessment program, tracks lead time for changes: how long a change takes to go from written to running live for customers. Leaders who have watched a portfolio with too many projects in flight have seen the same thing at a larger scale. An agent workflow with a dozen unchecked steps in flight has the same problem as a team with a dozen half-finished features: the feedback arrives after the next decisions have already been built on top of it.
Small errors also compound along a chain. A step that is right 98 percent of the time sounds dependable. If any single mistake spoils the result, twenty such steps in sequence come out right about two times in three. Manufacturing learned the answer long ago: inspect at each station, not only at the end of the line. Control engineers have a related structure called cascade control, in which a fast inner loop corrects a disturbance close to where it enters, before it can upset the slower outer loop. In orchestration terms, check each step where it happens rather than only inspecting the finished result.
The plant changes underneath you
Classical control assumes that the thing being controlled behaves tomorrow as it did today. Orchestrated AI systems often do not. A team can pin a specific model version, and a pinned version usually stays put, but the convenient names that always mean “the latest model” move to newer models as they ship, older versions are retired on a schedule, and the tools and data around the model change too. A prompt that behaved one way in spring can behave differently in autumn. In control terms the plant is not time-invariant: it does not behave the same way from one day to the next.
Process engineers have a practice for an unfamiliar or drifting plant called system identification. Before tuning a controller, they disturb the plant in a known way, measure how it responds, and fit an equation to what they observe. Afterwards they keep checking whether it still responds that way. The orchestration equivalent is an evaluation suite: a fixed set of cases with known right answers, run whenever the model, the prompt or the tools change, and compared against last time. The right answers in that suite should change only when the requirements do; rewriting them to match whatever a new model produces turns the test into a record of the drift. A team that would never swap a valve on a production line without re-testing the loop will often take a model update with no re-testing at all.
Loops around loops: routing and parallel work
Orchestration no longer stops at a single task. Two more layers are now common, and each adds a loop of its own.
The first is routing across models. Models differ widely in capability and in price per token (tokens are the word fragments that AI vendors meter and bill by), so an orchestrator increasingly reads each request and decides which model should handle it: a small, fast one for pulling routine details out of an email, a larger one for an unusual drawing or a nonstandard discount. Control engineers would recognize a close relative of gain scheduling, where a supervisor reads the conditions a system is operating in and changes how it responds. Strictly, a router swaps the model doing the work rather than retuning a controller, but the dependency is the same: everything rests on reading the conditions correctly, which makes the router a sensor. When it misjudges a hard request as easy, the small model often does not fail loudly. It produces a fluent answer at a fraction of the price, and the error surfaces downstream, if it surfaces at all.
Routing also has to be judged on the whole loop rather than the single call, in both directions: sending everything to the largest model wastes money as surely as sending hard cases to the smallest wastes quality. A model at a tenth of the price that needs four attempts, with a check after each, does not cost a tenth as much. A router tuned only on token spend will minimize token spend, because that is what it can see. The usual safeguard is escalation, where a request that fails its check moves up to a more capable model instead of retrying on the same one, and it works only as well as the check that triggers it.
The second layer is parallel work across sessions. For a large project, a coordinating session splits the work into tasks, often one component per session, starts them in parallel and waits for each to return, much as a program can start several jobs at once and wait for each to finish. Engineers call this a fan-out and a fan-in: the work fans out to many workers and back in to one. It brings four problems, most of them familiar to anyone who has built distributed systems, software that runs across many machines at once.
The first is the brief. Each session typically starts with little more than the task description the coordinator wrote for it, and none of the conversation that produced it. Whatever the coordinator understood but did not write down does not reach the worker, and a vague brief comes back as a confident, well-built component that solves a slightly different problem.
The second is the join. The coordinator waits on the slowest task, so one stalled session holds up the whole result unless something limits how long each task can run. Every task in flight is also work in progress in the sense above: started, unchecked, and waiting for a verdict.
The third is the report. What the coordinator usually receives is each task’s own account of its work, and unless that account carries evidence the coordinator can verify, such as tests that ran and their results, a task’s report of its own success is not an independent check. It is the same sensor problem one level up. The parallel agents I described earlier each reported, honestly, on the only part of the work each could see.
The fourth is the seam. Parallel tasks combine cleanly to the degree that the boundaries between them were decided in advance: which component owns which data, what each interface (the agreed point where two components connect) promises, and who settles a disagreement between them. That is integration architecture, the design of how separate systems connect and share data, under another name. When several parties each hold part of one result, the problems concentrate where the work changes hands. A coordinator that merges whatever comes back, without an agreement on what each task was allowed to touch, has moved the integration problem to the end of the project, where it is usually most expensive to fix.
Where the analogy breaks
None of this means control theory can be applied to AI systems as it stands. Classical control works with systems that respond in proportion to what is done to them and behave the same way from one day to the next. The textbook term is linear and time-invariant, and it is what allows a system’s behavior to be written as an equation and its stability proven on paper. A language model is neither. More advanced branches of control handle systems that are nonlinear or that change over time, but most of them still begin from an equation that describes the system, and nobody has a useful one for a language model. The methods that work without one, such as tuning a controller by trial on the real system, give up most of the proofs. No one is going to draw a Bode plot, the standard chart of how a system responds at each frequency, for a prompt. Most of what the field can prove about stability does not apply.
Researchers have started on the formal side. A 2023 paper by Aman Bhargava and colleagues, What’s the Magic Word? A Control Theory of LLM Prompting (LLM is short for large language model, the kind of AI model this piece is about), treats the prompt as a control input and asks which outputs a short prompt can steer a model toward. That work concerns how a single model responds to its input. This piece is about the loops being built around models, and about the practice more than the proofs.
What carries over is the discipline. Identify the plant by measuring it rather than trusting its description. Know what the sensor can and cannot see before relying on the loop. Look for shared error before trusting agreement. Treat retries as gain and delay as a budget. Assume the plant drifts, and keep re-measuring it. None of that requires the mathematics. It does require the habit of looking at a system and asking whether it is stable, not only whether it is fast.
Who brings this lens
This is not a question of job titles. The people building these systems are mostly software engineers, IT architects and machine-learning specialists, and many of them already think this way. What matters is that someone on the team looks at a design and asks where the loop is, what closes it, and what happens when the sensor lies. Coursework in signals and systems and in control trains that habit directly; it is part of electrical, biomedical, mechanical, aerospace and chemical engineering, and of many computer engineering degrees. Plenty of people with that training already work in software, as I do, and plenty of others picked up the same instincts running large production systems.
The lens is useful from the first sketch, when the question is which loops are worth closing at all. Plenty of tasks are better done open-loop, with no automatic correction and a person reading the output. It matters most in testing. Checking the case where everything goes right tells you little about a loop; the useful test is to break things on purpose, disturb the system in known ways, and watch whether it settles quickly or rings, swinging back and forth for a while first.
The questions are learnable, and software engineers already answer some of them every day. Anyone who has run a large distributed system has seen a retry storm, where many failed requests retry at the same moment and the retries overload the system further. That is positive feedback, feedback that amplifies instead of correcting, and the standard fix is backoff, waiting progressively longer between retries. Others get rediscovered under new names, such as a flaky agent that works only some of the time, an infinite retry, or a model regression, where a new model version gets worse at something the old one did well. What control theory adds is a shared vocabulary for all of them, and a catalogue of failures worked out long before the first incident. The practical step is simple: make sure someone on the team is asking these questions from the start, whatever their title.
What to ask before you build or buy one
If someone proposes an orchestrated AI system for your business, whether a vendor or your own team, these questions will tell you a good deal in a short meeting:
- What checks the output, and what is that check unable to see?
- Is the check independent of the thing it checks, or does it share the same model and the same blind spots?
- What stops the loop other than a counter, and what is done with an output that hit the limit?
- How does quality at the twentieth step compare with quality at the first, and how do you know?
- When the model changes, what gets re-tested, and against what?
- If requests are routed between models, what decides the route, and what happens when a hard case goes to a small model?
- When work runs in parallel, what was each task allowed to change, and what checks the merged result rather than each task’s own report?
- Where is a person in the loop, and is that person a real check or a rubber stamp?
A team that has good answers has thought about control, whether or not it uses the word. A team without answers yet has probably built a feedback system without meaning to, and these questions are a better place to find that out than the first time it oscillates.
The same discipline already exists in legacy modernization, the work of updating long-lived business systems, and it was the subject of the first piece in this series. When pricing rules are extracted from legacy code, the result should not be trusted until it has been replayed against a year of real quotes and every difference has been explained in writing. That replay is an independent sensor, and the reconciliation closes the loop, with a person deciding what each difference means. Orchestrated AI needs the same thing: the oldest idea in this piece, applied to the newest kind of system.