No Agent Left Behind

We’re hiring! Explore our Product Engineering Guide to learn how we build at Orbital.

Most of what gets written about AI agents is about making them smarter: better planning, better tool use, longer autonomy. That work is real, and we do plenty of it. But the questions that shaped my team’s last six months were quieter ones. What happens when a deploy lands while an agent is twenty minutes into a task? What stops one tenant’s batch job from slowing the platform for everyone else? What happens when a client updates their question list while parallel agents are still answering the old version? How quickly do we notice a failure that arrives silently?

These questions carry real weight at Orbital because our agents review legal documents for real estate transactions, where a lost agent run is a lawyer’s half-finished due-diligence report on a live deal.

They also arrived at pace: over six months, document volume grew roughly 7×, from about 4,000 to about 30,000 documents a week, while agent tasks became longer-running and more autonomous. This post covers three stories from that period. None of them are about model quality; they are about the platform habits that let long-running agents survive deploys, growth, retries and noisy neighbours.

Deploys should be invisible, even to an agent mid-task

We hold a simple conviction: a user should never be able to tell, from inside the product, that we shipped code. First-generation agent platforms fight this by design. Agents run on a task queue with hand-rolled state checkpointing, so recycling a worker discards in-flight state. When agents run for minutes and you deploy many times a day, that flaw stops being theoretical.

We moved the fleet to durable execution on Temporal, and it has earned its place. The headline capability is that workflow state survives any restart, so a deploy no longer costs in-flight work. The compounding benefits matter just as much for an agent platform:

• Completed steps are no longer repeated. Every step is recorded in workflow history, so a resumed run returns recorded results immediately — including expensive LLM calls fanned out in parallel — rather than executing, and paying for, them twice.
• Retry policies, timeouts and heartbeats are first-class primitives, declared per activity.
• Replay is controllable. We decide how in-flight workflows behave across code changes, and we can verify that behaviour in CI.

The same guarantee extends to the work’s inputs. A run pins the version of the question list it started with, so parallel agents all answer against the same version, and a mid-run update won’t produce a report stitched together from two definitions of the work.

The harder question sat above the framework: how do you swap the execution substrate underneath multiple production agents while lawyers are using them? We built a playbook and applied it, agent by agent:

• Shadow runs. Each agent’s new execution path ran in parallel behind real user interactions, with side effects switched off. Production traffic became the test suite, at zero user impact.
• Parity harnesses. An LLM-as-judge comparison of old-path and new-path outputs, plus record-and-replay of production-shaped runs, because the requirement was equivalent answers, not merely a clean exit. The same checks guard promotion to the live path.
• Dice-roll canary. Where double-running was too expensive, as with long-running sandboxed agents, a randomised ramp from 1% of runs to 10% to 100% bounded the blast radius, each stage holding until the numbers earned the next. The first 1% still made me nervous to approve, which is exactly why it started at 1%.
• Same-week deletion. The moment an agent hit 100%, the legacy path was deleted. Nobody was sad to wish it farewell; zombie code paths are where regressions hide.

Two decisions inside that playbook are worth pulling out. Because sensitive payloads can pass through workflow history, encrypting them at rest was a precondition; it shipped before the first agent did. And when replay non-determinism errors appeared (the failure mode where a workflow’s code drifts from its recorded history), we paused the migrations entirely and built replay-safety testing into CI before continuing.

Slowing down to build the guard rail is what made the subsequent speed safe. Once the playbook was proven on the smaller agents, the entire fleet, including a full rewrite of the most legacy agent, moved in about six weeks with zero migration-caused incidents.

Let the data set the thresholds

LLM calls fail differently from the RPCs that standard retry policies were designed for. They are slow, their latency variance is enormous, and their worst failure mode is not an error but a hang: a call that will never return, inside a worker that looks healthy from the outside.

A stock retry policy makes both problems worse. Retrying a hung call multiplies one slow failure into close to half an hour of waiting, and without heartbeats the only signal that a worker has died is a long start-to-close timeout expiring. Gating LLM calls properly comes down to choosing thresholds, and thresholds attract opinions.

We wanted better grounds than opinion, because an over-aggressive timeout kills legitimate slow calls, and in legal work a killed call is a lawyer’s lost answer. A week of production traffic supplied the grounds: p99 LLM-call latency sat around two minutes, and the slowest genuine call came in just under five. A three-minute per-attempt budget would clip roughly 0.45% of legitimate calls a week — a cost we could see, quantify and accept. The same data shaped the policy’s most important rule:

A timeout that fires after the user has given up should never be retried.

The policy has three parts, applied across the platform’s LLM call sites: per-attempt time budgets, so one hung call can’t consume the whole retry allowance; a non-retryable timeout class, so hopeless calls fail fast; and heartbeats on every call, so a dead worker surfaces in about a minute and the work resumes on a healthy one.

Then the move we care most about: the policy went into machinery, because a policy is only as good as its newest call site. A registry-driven CI advisory inspects changes at PR time and flags violations of these platform contracts, such as an unheartbeated LLM call or a retry policy without a per-attempt budget. Adding a rule means adding a registry entry, with no code change. Our test for whether a standard matters is whether we made a computer enforce it for us.

Fairness has to be designed in

Multi-tenant AI platforms have a physics problem: agent workloads are lumpy. One tenant uploads a thousand documents at 9am, and without deliberate countermeasures their burst becomes everyone’s latency. Fairness has to be built at the exact point where heavy processes meet shared resources, and as the options we rejected below show, more compute doesn’t move that point.

The first shared resource was database capacity: bounded, pooler-safe connection pools, rolled out in stages — dev burn-in first, then document workers, then the API tier.

The second was memory. Document processing is hungry for it: a single large legal document can claim a meaningful share of a worker’s budget, and enough of them landing together will eventually kill the process, taking every in-flight task with it. What makes this worth telling is what we rejected, and why:

• Per-worker memory recycling: the standard answer, and a poor fit for our runtime model.
• A dead-letter queue: moves the operational burden without removing it. Someone still has to fish oversized documents out by hand.
• Horizontal scaling: every new replica inherits the same failure mode. You can’t scale your way out of a per-process problem.

What we chose was admission control: each worker tracks its own memory position and takes on new work only when it can afford it, with safeguards so even the largest document is never starved out entirely. It shipped dark, in shadow mode, logging what it would have rejected, because I wanted its estimates calibrated against real document traffic before it was allowed to reject anything at all.

The results earned their keep: sustained zero overload incidents across two consecutive months, worst-case latency spikes down by an order of magnitude, and base latency several-fold lower.

What keeps agents alive

Looking across these stories, the common thread is a way of deciding.

Evidence over intuition. Retry budgets set by a week of latency data and a 0.45% threshold analysis, not by someone’s sense of a reasonable timeout. The same discipline cuts both ways: when measured frequency doesn’t justify a fix, we cancel the ticket rather than cargo-cult it.

Patterns over patches. “A timeout that outlives the user’s patience is never retried” and “admission control at the process boundary” close whole classes of failure, not single instances.

Learnings become forcing functions. The registry-driven CI advisory is the clearest example: platform contracts enforced at PR time, so the standard holds without anyone having to remember it. If a lesson matters, machinery carries it for us.

Discipline is what makes speed safe. Shadow first, calibrate dark, ramp with dice, delete the old path the same week. That rigour is precisely why the fleet migration took six weeks rather than six months, and why nobody outside the team noticed it happening.

Agent reliability is not one mechanism. It is a stack of habits: durable state, bounded retries, calibrated thresholds, fair resource sharing, shadow runs, canaries, and deletion.

None of this is glamorous, but it’s the layer our customers actually stand on, and it compounds: every habit above made the next piece of work faster and safer to ship. Models will keep improving on their own schedule. Keeping a multi-tenant agent platform fair, observable and alive as tasks grow longer and more autonomous is a frontier we expect to be working on for years, and it’s the problem we’re signing up for next.

We’re hiring

If hardening AI agent platforms under real growth, with real legal documents and real stakes, sounds like your kind of unglamorous, we’d love to talk. See our open roles on our Careers Page, and if nothing quite matches your experience, connect with and message me directly on LinkedIn and I’d be happy to have a chat with you.