AI agents

AI Agent Orchestration: Managing Complex Workflows with Intelligent Automation

How to choose between a workflow, a single agent and sub-agents, make long runs survive failures, and test the result. With patterns and code.

By the APIStack team··9 min read

Black threads joined at many points to form an irregular web against a pale background

Orchestration is the code that decides which model call or tool runs next, what context it receives, and what happens when a step fails. This article is for developers who have a working prototype and need to decide how much structure to put around it: a fixed workflow, one agent with tools, or an orchestrator with sub-agents.

It covers the patterns that vendor engineering teams document and recommend, a worked example, how to make long runs survive crashes and deployments, and how to evaluate the result.

Workflows and agents are different designs

Anthropic's Building effective agents draws the distinction most teams now use. A workflow orchestrates model calls and tools through code paths you wrote in advance. An agent lets the model direct its own process and tool use, in a loop, until it decides the task is done.

The same article recommends finding the simplest solution that works and adding complexity only when it demonstrably improves outcomes. Agentic systems trade latency and cost for task performance, so the trade has to be worth it. For many applications a single well-built model call with retrieval is enough.

The practical consequence is that control flow belongs in code whenever you can write it down. If the steps of a refund, an onboarding check or a report are known, a workflow gives you predictable cost, a fixed number of calls, and failures you can reproduce. Reserve the agent loop for the part of the task where the steps cannot be listed in advance.

Pick the simplest pattern that fits

The five workflow patterns in Anthropic's article, plus the autonomous agent, cover most production designs. They compose: a router can sit in front of a chain, and one step of a chain can be an agent.

PatternWho decides the next stepUse it whenMain failure mode
Prompt chainingYour code, in a fixed orderThe task splits into known sub-tasksAn early error flows downstream unless a check stops it
RoutingA classifier, then your codeInput types need different handling or different modelsMisclassification sends work down the wrong path
ParallelizationYour code, fanning outSub-tasks are independent, or you want several votesResults that conflict and no rule for merging them
Orchestrator-workersA lead model that plans sub-tasksSub-tasks cannot be predicted from the inputVague delegation, duplicated or missing work
Evaluator-optimizerA second model that critiquesQuality criteria are clear and revision measurably helpsLoops that never converge without an iteration cap
Autonomous agentThe model, every turnOpen-ended tasks in an environment you can sandboxCompounding errors and unbounded cost

A worked example

Take a support task: a customer asks when an order placed today will be dispatched, and dispatch takes three working days. The steps are known, so this is a chain with a check between steps, not an agent. The model extracts structured input, code validates it, an API computes the date, and the model writes the reply from that fact.

The function below takes any llm(prompt) function that returns text, and calls the working-days endpoint behind the public holidays API so the model never does calendar arithmetic itself.

JavaScript
async function dispatchReply(message, llm) {
  const raw = await llm(
    'Return only JSON with keys "country" (two-letter ISO code) and ' +
    `"orderDate" (YYYY-MM-DD) for this message:\n${message}`
  )
  const { country, orderDate } = JSON.parse(raw)
  if (!/^[A-Z]{2}$/.test(country) || !/^\d{4}-\d{2}-\d{2}$/.test(orderDate)) {
    throw new Error('Extraction failed the check; route to a person')
  }

  const url = new URL('https://apistack.online/api/data/public-holidays/v1/add')
  url.search = new URLSearchParams({ country, date: orderDate, days: '3' })
  const res = await fetch(url, { headers: { 'x-api-key': process.env.APISTACK_API_KEY } })
  if (!res.ok) throw new Error(`Working-day lookup failed: ${res.status}`)
  const { data } = await res.json()

  return llm(`Write a two-sentence reply giving the dispatch date. Use only these facts:\n${JSON.stringify(data)}`)
}

Two model calls, one API call, and every failure is an exception at a known line. The same task given to an agent with a calendar tool would usually produce the same answer, with more turns and less certainty about which path it took.

Use one agent before several

When the task does need an agent, the next decision is whether one agent holds the whole context or a lead agent delegates to sub-agents. The published evidence points in two directions, and both are worth reading.

Cognition's Don't build multi-agents argues for a single-threaded agent by default. Its two principles are that agents should share full context, including the traces of earlier actions and not just messages, and that every action carries implicit decisions, so parallel workers that cannot see each other make conflicting ones. Its example is two sub-agents building halves of a game that do not match in style.

Anthropic's account of its multi-agent research system reports the opposite result for a different kind of task. On an internal research evaluation, a lead agent with sub-agents outperformed a single agent by 90.2%. The same post reports the price: agents use about 4 times the tokens of a chat interaction, and multi-agent systems about 15 times. It also states where the design fits poorly: tasks where all agents need the same context, or with many dependencies between agents. It adds that most coding tasks have fewer parallelizable sub-tasks than research.

The two positions are consistent once the task shape is considered. Sub-agents pay off when the work is broad, mostly read-only, and separable, such as searching many sources, because each worker explores in its own context and returns a short result. Anthropic's context engineering guide describes a sub-agent spending tens of thousands of tokens and returning a summary of 1,000 to 2,000. They pay off poorly when workers write things that must agree with each other.

Write the delegation brief

Most orchestrator-worker failures are briefing failures. Anthropic's post lists what each sub-agent needs: an objective, an output format, guidance on tools and sources, and clear task boundaries. Without boundaries, its early system had sub-agents duplicating each other's searches. A brief can be a plain structure the lead model fills in.

JSON
{
  "objective": "Find the statutory public holidays in Germany that differ by state in 2027",
  "output_format": "JSON array of {date, name, states}, no prose",
  "tools": ["list_public_holidays"],
  "boundaries": "Germany only. Do not cover school holidays. Stop after 8 tool calls.",
  "context": "The user is planning delivery cut-off dates for a retailer shipping from Hamburg"
}

Make long runs durable

An agent run that lasts minutes or hours will meet a crashed process, a rate limit or a deployment. Restarting from the first step repeats cost and, worse, repeats side effects such as emails and payments.

Durable execution engines solve this by recording each completed step and replaying the record after a failure. Temporal's documentation describes a workflow execution as resuming by replaying its event history, with outside-world calls placed in activities that are retried automatically and should be idempotent. Anthropic's research system post describes the same need from the agent side: it resumes from where the agent was when an error occurred, and combines the model's adaptability with deterministic safeguards such as retries and regular checkpoints.

The core of the technique fits in a few lines. This helper assumes store is any key-value store with asynchronous get and set; each step runs once per run and its result is reused after a restart.

JavaScript
async function step(store, runId, name, fn) {
  const key = `${runId}:${name}`
  const saved = await store.get(key)
  if (saved !== undefined) return saved
  const result = await fn(key) // pass the key on as an idempotency key
  await store.set(key, result)
  return result
}

A crash between fn and store.set runs the step again, which is why the key is passed to the side-effecting call: the downstream system uses it to ignore the duplicate. Wrap every model call and every tool call in a step, and a restarted run costs only the steps it had not finished.

Deployment needs the same care. A running agent holds state, so replacing all workers at once breaks runs in progress. Anthropic describes shifting traffic gradually from the old version to the new while both stay up, which it calls a rainbow deployment.

Evaluate the outcome, not the path

Orchestration changes are hard to judge by reading a few outputs. Anthropic's guide to agent evaluations recommends starting with 20 to 50 simple tasks drawn from real failures, and grading what the agent produced instead of the sequence of tool calls it used, because agents find valid routes the designer did not expect.

Three practices from that guide apply directly to orchestration work:

  • Check the outcome in the environment. Assert that the record exists in the database, not that the agent said it created it.
  • Run each task several times. The guide's example is a 75% per-trial success rate, which gives about 42% for three successes in a row. A customer-facing workflow needs the stricter measure.
  • Keep a regression suite near 100% and a separate capability suite that starts low, so a change that helps hard cases cannot quietly break easy ones.

Run the suite before and after every structural change: adding a sub-agent, swapping a model in a router, or changing a brief. If the multi-agent version does not beat the single agent on your tasks by enough to cover its token cost, keep the single agent.

Failure modes to watch for

  • No turn or budget limit on a loop. Every agent loop and evaluator loop needs a maximum number of iterations and a cost ceiling.
  • Too many overlapping tools. The context engineering guide names bloated tool sets as a common failure, because the model cannot choose between near-duplicates. The APIStack MCP endpoint takes the small-surface approach, exposing its catalog through three tools (search_apis, get_api_details, call_api).
  • Sub-agents that return everything. A worker that passes its full transcript back defeats the purpose of giving it a separate context.
  • Retries without idempotency. A retried step that sends a second email is an orchestration bug, not a model bug.
  • Framework layers you have not read. Anthropic notes that incorrect assumptions about what a framework does underneath are a common source of error. Know the prompts your framework sends.

Summary

  • Put control flow in code when the steps are known; use an agent loop only for the part that cannot be scripted.
  • Start with a single agent. Add sub-agents for broad, read-heavy, separable work, and measure whether the gain covers roughly 15 times the tokens of a chat.
  • Brief sub-agents with an objective, an output format, allowed tools and boundaries.
  • Checkpoint every model and tool call, pass idempotency keys to side effects, and deploy without killing runs in progress.
  • Judge changes with a small task suite that checks outcomes across repeated trials.

Build with APIStack

One API key for validation, dates and holidays, web checks and reference data, over REST or as MCP tools for your AI agents. The free plan includes 1,000 calls a month.

Keep reading