multi-robot-fleet-coordination

Fleet Algorithm Atlas

How each algorithm in the fleet works, one step at a time. Why it is here, what else could do its job, and when you would switch. Every animation runs a line-by-line JavaScript port of the C++ code, checked against the real thing.

  1. 0Setup
  2. 1Sim core
  3. 2Baseline
  4. 3Allocation
  5. 4MAPF + PIBT
  6. 5Traffic
  7. 6Faults + fuzz
  8. 8Isaac Sim
  9. 9aHIL
  10. 9bPerception
  11. 10Release
Aisle Shelf Station (P pickup, D dropoff) Parking To pickup Carrying Idle Working at a station Stopped by safety

Cells are written (row, column), counted from the top left. One step is one simulated second.

Overview

One coordinator tick

Every simulated second, the coordinator runs these eight steps in order. Solid chips are shipped. Outlined chips show the stage that adds them. Select any chip to see what it does and when you would pick it.

    Shared service, used by steps 4 to 6

    Distance oracle

    The exact walking distance between any two cells, worked out once at startup.

    Stage 1, shippedfleet_core/src/world/distance_oracle.cpp

    View
    Endpoint rule

    Click any free cell to make it the source.

    Hover a cell to read it.

    How it works

    1. Number every free cell. This map has 99.
    2. From each cell, run one breadth-first search. It spreads outward one layer per step of distance.
    3. Stations and parking cells get a distance but pass nothing on. A path may end there, never cross one.
    4. Store every distance in one table, 2 bytes each. A lookup is a single array read.

    Why we use it

    • Shelves make straight-line guesses wrong. Allocation needs real walking distance.
    • A* uses it as its guess of the distance left. An exact guess wastes almost no search.
    • It costs one BFS per cell, once. Every later lookup is free.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    BFS tableThe map is static and has up to 4,000 free cells.Grows with cells squared. Rebuilt when a cell closes or opens (Stage 5), so it routes around closed aisles.In use
    Manhattan distanceOpen floors, or maps too big for any table.Wrong behind shelves, so allocation picks far robots.distance_oracle: false
    BFS per queryFew lookups, or a map that changes every tick.Repeats the same search over and over.
    DijkstraMoves cost different amounts: turns, ramps, traffic.Slower than BFS. The same table trick still works.
    Floyd-WarshallSmall graphs with weighted edges.Cells cubed: 64 billion steps for 4,000 cells.
    Landmarks (ALT)Huge maps where a full table will not fit.Gives a lower bound, not the exact distance.

    Tick step 1: what the floor looks like now

    Aisle closures

    A closed aisle cell changes every distance around it. The coordinator rebuilds the distance table as soon as it hears of a closure, and the fleet replans around it.

    Stage 5, shippedfleet_core/src/world/distance_oracle.cpp

    Cell (2,4)

    Numbers: steps from each cell to the dropoff D. The line: R0’s shortest route.

    How it works

    1. A block_cells event closes aisle cells, and unblock_cells opens them again.
    2. A closure takes effect once no robot stands on any of its cells, on the floor and in the coordinator’s map, and then its cells close together. A robot inside a closing aisle drives out first: closing the cells around it would leave it no way out.
    3. The coordinator rebuilds the distance table. No path passes through a closed cell or ends on one.
    4. Coordinated planners replan the whole fleet at once, so the new plans fit together. The baseline replans each robot whose path now crosses a closed cell.
    5. Permits stop short of a closed cell, so no robot is sent into one.

    Why we use it

    • Aisles close in real warehouses: a spill, a dropped pallet, a person at work. Allocation and planning need the real distances, or they send robots toward dead ends.
    • A rebuild runs one breadth-first search per free cell, 330 on warehouse_small, and only when a cell changes.
    • A closure is accepted only if the open aisles still reach every station and parking cell, so no order becomes impossible.

    The catch: until its cells are empty a closure has not happened, so robots can still drive in. On a busy aisle it can wait a while. With six cells of the middle aisle closed for half of each hour, the four rolling planners delivered at most 0.7% fewer orders per hour than with the aisle open, with no safety stop or violation in 120 runs.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Rebuild the whole tableClosures are rare and the map is small.Every rebuild redoes every search, even far from the change.In use
    Repair what changed (LPA*, D* Lite)Cells change often, on big maps.Much more code, and one search per target.
    Keep the static table, let A* find the detourClosures are short and few.Allocation keeps seeing the old, shorter distances.

    Tick step 6: plan paths

    Independent A*

    Each robot gets its own shortest path. No robot knows the others exist.

    Stage 1, shippedfleet_core/src/planning/independent_astar.cpp

    Heuristic
    Next click sets

    Numbers are each cell's score f. Shaded: expanded. Outlined: waiting in the open list.

    Hover a cell to read it.

    How it works

    1. Put the start in the open list with score f = g + h. g is steps taken so far. h is the guess of steps left.
    2. Take the open cell with the lowest f. Ties go to the smaller h, then the lower cell number.
    3. If it is the goal, stop and follow the parents back. If not, add its free neighbours with g + 1.
    4. Skip stations and parking cells, unless one is the goal.
    5. With an exact h, every cell off a shortest path scores higher. So A* walks almost straight to the goal.

    Why we use it

    • It is fast. The default run's median planning call took 0.007 ms.
    • The oracle gives an exact h. On this map A* then expands exactly one cell per step of path, plus the goal, for all 9,702 start and goal pairs.
    • It is the baseline. Every coordinated planner in Stage 4 has to beat it.

    The catch: it plans as if the warehouse were empty. Two robots heading opposite ways down a one-cell aisle meet nose to nose and stop for good. The gridlock demo in Robot safety shows it.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Exact h (oracle)A distance table fits in memory.Needs the table first.In use
    Manhattan hNo table, or open floors.Searches more cells around shelves.distance_oracle: false
    No h (Dijkstra)Move costs vary and no good guess exists.Searches in every direction at once.
    Greedy best-firstAny path will do, quickly.Not shortest. Can wander toward dead ends.
    Jump point searchLarge open grids where every move costs the same.Little to gain in one-cell aisles.
    Space-time A*Robots must dodge each other over time.Searches cells times time steps. Slower per robot.Stage 4, inside every coordinated planner

    Tick step 4: allocate

    Nearest-robot allocation

    Oldest order first, to the free robot with the shortest walk to its pickup.

    Stage 1, shippedfleet_core/src/allocation/nearest_allocator.cpp

    Scene
    Show

    Distances are walking steps from the oracle, around shelves.

    Hover a cell to read it.

    How it works

    1. List pending orders, oldest first. Same age: lower order number first.
    2. For the oldest, look up each free robot's walking distance to its pickup.
    3. The closest robot gets it. On a tie, the lower robot number wins.
    4. Take that robot out of the pool. Repeat until orders or robots run out.

    Why we use it

    • It is cheap: one table lookup per order and robot.
    • Oldest first keeps waiting times fair.
    • With one order and several free robots, nearest is already optimal.

    The catch: with several orders at once, an early pick can take the robot a later order needed. In the two-order scene, nearest spends 9 steps of travel. The best matching spends 5.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    NearestOrders trickle in one at a time.Greedy: early picks can hurt later orders.In use
    HungarianMany orders and robots are free at once.Optimal only for this instant, and blind to traffic.Stage 3, shipped
    Cost-awareThe closest robot would sit in traffic or a station queue.Needs a congestion estimate and a weight to tune.Stage 3, shipped
    Learned costLogged runs exist and plain distance is a poor guess.Learns only from pairs earlier allocators chose.Stage 3, shipped
    AuctionRobots decide for themselves, or the network is patchy.Many messages. Quick rounds can miss the best matching.Stretch
    Global greedyTotal travel matters more than order age.Far orders can wait forever.
    Batching windowOrders arrive in bursts.Every order waits for the batch to close.

    Tick step 4: allocate

    Optimal matching

    Pairs every free robot with a waiting order so the total walk to the pickups is as short as it can be. hungarian, cost_aware and learned share this solver and differ only in what a pair costs.

    Stage 3, shippedfleet_core/src/allocation/assignment_solver.cpp

    Scene

    Blue: the matching so far. Dotted: the order a robot gave up when it moved.

    Hover a cell to read it.

    How it works

    1. Build a table: one row per free robot, one column per waiting order, each cell the robot’s walk to that pickup.
    2. Add the robots one at a time. A new robot reaches for an order. If another robot holds it, that robot moves to its next best order, and so on down a chain. The chain is called an augmenting path.
    3. Of all chains, the solver takes the one that adds least to the total. A price on every row and column, called a potential, turns that into a shortest-path search, so it never tries every matching.
    4. After the last robot joins, no matching is cheaper. With k free robots and n orders it costs O(k² n), and the other way round when orders are fewer.

    Why we use it

    • It fixes nearest’s trap: in Two orders it walks 5 steps where nearest walks 9.
    • It is exact and quick. 25 robots and 25 orders solve in 0.04 ms at p95, 100 and 100 in about 1 ms.
    • The cost of a pair is a plug-in. cost_aware and learned reuse the solver with a different cost.

    The catch: it is optimal for this instant only, and plain distance knows nothing about traffic. With a backlog it can serve closer orders first, so the oldest order waits longer than under nearest. See Backlog.

    Cost-aware: a busy station

    R0 is free. T1 at (2,0) is 1 step away and T2 at (6,0) is 3, but R1 already heads to (2,0) and gets that station first. cost_aware adds weight × (robots inside the rectangle between robot and pickup + service steps × robots already queued there). Change the weight.

    congestion_weight

    Solid blue: R0’s choice. Dotted: the order it passes up.

    Hover a cell to read it.

    Measured: the same rounds, solved both ways

    bench_allocation on warehouse_small: 1,000 random rounds per row, robots on random free cells, orders at random stations, seed 1. Hungarian is never worse. With twice as many orders, part of the gain is picking closer orders, not only who goes where. There is no traffic in these rounds. Throughput under coordinated planning is measured in the Rolling horizon section below.

    Hungarian’s total walk to the pickups, lower than nearest’s by

    All six rows as a table
    RobotsOrdersNearest, mean totalHungarian, mean totalLower byRounds strictly better

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    HungarianSeveral robots and orders are free at once.Optimal for this instant only. Blind to traffic.allocation.method hungarian
    Cost-awareThe closest robot would queue at a busy station or sit in traffic.The penalty is a rough guess, with one weight to tune.cost_aware
    Learned costYou have logged runs, and distance is a poor guess.Learns only from pairs earlier allocators chose.learned, trained by the learning track
    NearestOrders trickle in one at a time, or the oldest order must go first.Greedy: early picks can hurt later orders.nearest, the baseline
    AuctionRobots decide for themselves, or the network is patchy.Many messages. Quick rounds can miss the best matching.Stretch
    Batching windowOrders arrive in bursts.Every order waits for the window to close.

    Tick step 4: which orders go first

    Starvation guard

    Keeps an order far from the robots from being skipped for good. Orders that have waited more than 600 steps are matched first, oldest first.

    Stage 5, shippedfleet_core/src/allocation/matching_allocator.cpp

    Starvation guard
    Order 0 has waited

    Two free robots, three waiting orders. Lines: the assignments. Costs are walking steps to each pickup.

    How it works

    1. The matching sees pending orders oldest first. The ones pending longer than 600 steps come first: the starving orders.
    2. The oldest of them, at most one per free robot, are matched first, with the usual costs. The cost still decides which robot takes each.
    3. The robots left over then match the orders left over, as before.
    4. hungarian, cost_aware and learned all share it. nearest needs none: it already takes the oldest order first.

    Why we use it

    • The cheapest matching serves the closest orders. When orders come faster than robots can serve them, new orders near the robots keep winning, and a far order can wait for the rest of the run.
    • At 0.2 orders per step with 5 and 10 robots, offered at least twice what they can carry, the oldest order still open at the end of an hour had waited 2335 steps on average with the guard and 2556 without.

    The catch: those robots walk farther, so the same runs delivered 1.7% fewer orders per hour. Where robots keep up, no order reaches 600 steps and the guard changes nothing.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Starving orders firstA hard limit on waiting matters, and overload is the exception.The 600 steps are a choice, not a law.In use
    Age in the costYou want a smooth trade between distance and waiting.One more weight to tune, and an old order can still lose to a cheap pair.
    Oldest order, nearest robotFairness first.Robots cross the floor for old orders while near ones wait.nearest
    Due timesOrders carry deadlines.These orders carry none.

    Tick step 5: pick goals

    Station queues and the endpoint rule

    One robot per station. The rest wait on spare parking, out of the aisles.

    Stage 1, shippedfleet_core/src/tasks/station_manager.cpp

    Scenario

    Dotted lines: remaining paths. Dashed squares: waiting cells.

    Hover a cell to read it.

    How it works

    1. A robot with an order asks for its station. The first to ask gets the turn.
    2. Everyone else gets the nearest free spare parking cell (never a home, at most 8 per station) and waits there.
    3. When the robot at the station finishes, the next in line gets the turn and drives in.
    4. The endpoint rule: a path may start or end on a station or parking cell, but never pass through one. So parked and working robots block nobody.

    Why we use it

    • Without queues, robots wait in the aisle beside the station. In the Queues off scenario the station locks for good.
    • Without the rule, A* happily routes along the parking row, straight into parked robots.
    • Robots rest only on stations and parking, so aisles stay free for traffic.

    Scenario note: Going home, rule off is a what-if. The real planner always applies the rule. The other three scenarios run the real coordinator logic, step for step.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Queue on spare parkingStations sit near parking, as in our generated maps.Waiting robots walk a detour.In use
    No queue: drive in and waitEach station has its own waiting lane.Blocks the aisle and can trap the robot leaving.station_queues: false
    Queue lanes in the floor planYou get to design the warehouse.Fixed layout that costs floor space.
    Time-slot bookingService times are predictable.Needs good arrival estimates.
    Go home and waitHomes are close to the stations.Long round trips.Coordinated planners, when no spare cell is free
    Drop the endpoint ruleThe planner sees other robots and routes around them.Robots parked on endpoints would block paths again, and the coordinated planners rely on them never doing so.

    Tick step 6: plan paths

    Planning in space and time

    Finds one robot’s path through cells and steps, around paths other robots already hold. Every coordinated planner is built on it.

    Stage 4, shippedfleet_core/src/planning/space_time.cpp

    Scene
    Planner

    R1’s path is already planned (dotted). R0 plans around it. The small numbers are the steps R0 spends in each cell.

    Hover a cell to read it.

    How it works

    1. A search state is a cell and a step. From each state the robot can wait or move to a free neighbor, one step later.
    2. A state is out if a reserved robot stands in that cell at that step, or would swap places with this robot. Endpoints other than the goal stay out, as always.
    3. A* orders states by steps so far plus the distance oracle’s steps to go, so it still finds the earliest arrival first.
    4. It stops at the goal only when the robot can stay there: no reserved robot comes through later. Windowed planners stop at the window’s end instead.

    Why we use it

    • Waiting is a move here. In Crossing, R0 waits one step and arrives one step late instead of colliding.
    • One search serves every planner: prioritized and WHCA* reserve the paths found so far, CBS adds forbidden cell-steps, ECBS counts conflicts instead of forbidding them.
    • After the last reserved robot stops moving, every later step looks the same, so the search folds them into one layer and stays finite.

    The catch: it plans one robot. Whoever plans first takes the best cells and steps, and the robots after it get what is left. The next sections are about who plans first.

    Tick step 6: plan paths

    Prioritized planning

    Plans robots one at a time, oldest order first. Each robot plans around the robots before it. windowed_coop_astar (WHCA*) does the same, only a window ahead.

    Stage 4, shippedfleet_core/src/planning/prioritized.cpp

    Scene
    Looks ahead

    First the planning rounds, robot by robot. Then the robots run their plans.

    Hover a cell to read it.

    How it works

    1. Sort the robots: oldest order first, idle robots last, then by id.
    2. Each robot runs space-time A* around the paths already reserved, then reserves its own.
    3. If a robot finds no path, start over with it at the front. There are at most as many restarts as robots.
    4. WHCA* checks only the first planning.window steps (20 by default) and trusts the replanning to fix the rest.

    Why we use it

    • It is fast: one search per robot. At 25 robots it plans in 0.29 ms at p95, 0.22 ms with the window.
    • Oldest orders first is a fair rule that robots and people can predict.
    • The restart rescues the common bad order. In Restart, the robot inside the pocket must leave before the other comes in.

    The catch: it is not complete. In Head-on, both orders fail: whoever plans first walks straight through, and the other cannot reach the side cell in time. The fleet then keeps its old plans and tries again a step later.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Prioritized, to the goalPlans must hold until the robot arrives.Long paths make every later robot detour around them.planning.method prioritized
    Windowed (WHCA*)Lifelong traffic, where far-future plans go stale anyway.Ignores conflicts past the window.windowed_coop_astar
    Priority-based search (PBS)You want the planner to search over orders too.A tree search again, slower than one fixed order.
    Token passingRobots plan one at a time as they get orders.Needs every goal on an endpoint, as here, to be complete.

    Tick step 6: plan paths

    Conflict-based search

    Plans every robot alone, finds the first clash, and splits in two: one branch forbids the cell to one robot, the other to the other. ECBS takes any answer within a bound of the best.

    Stage 4, shippedfleet_core/src/planning/cbs.cpp

    Scene
    Search

    One frame per node of the search tree. The tree is on the right; the grid shows the node’s paths and its first clash.

    Hover a cell to read it.

    How it works

    1. The root plans every robot alone with space-time A*, ignoring the others.
    2. Take the node with the lowest total cost. Find its earliest clash: two robots in one cell, or swapping cells.
    3. Split: one child forbids that cell at that step to the first robot, the other to the second. Each child replans only that robot.
    4. The first node without a clash is the answer, and nothing cheaper exists. ECBS expands, among nodes within w of the best bound, the one with the fewest clashes.

    Why we use it

    • It is optimal: least total steps over all robots, within the window.
    • ECBS keeps most of that quality and is far quicker on crowds: in Siding it needs 2 expansions where CBS needs 8.
    • ECBS delivered the most at 25 robots and 0.2 orders per step, ahead of WHCA* in all 10 seeds.

    The catch: head-on in a one-cell aisle, CBS forbids the same clash one step later again and again, before it tries the siding. In the fleet benchmark that ran its budget out on 2,994 of 9,598 calls at 10 robots and 0.1 orders per step, and froze 2 of those 10 runs.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    ECBSYou want near-optimal plans for a crowd.A tree search: dense jams can still use up the budget.planning.method ecbs
    CBSA few robots, and the optimal total matters.Exponential when clashes pile up, worst in narrow aisles.cbs
    CBS with symmetry reasoningOptimal plans in narrow aisles.Corridor and target rules add a lot of code.
    Prioritized planningSpeed matters more than the total.Can fail where a tree search would not.prioritized

    Tick step 6: plan paths

    PIBT: priority inheritance

    Moves every robot one step at a time. A robot that needs an occupied cell lends its priority to the robot standing there, which must move first.

    Stage 4, shippedfleet_core/src/planning/pibt.cpp

    Scene
    Dead-end rule

    One frame per step. Curved arrows: a robot lending its priority to the robot in its way. Straight arrows: the moves.

    Hover a cell to read it.

    How it works

    1. Rank the robots: away from their goal first, longest away first, then a fixed hash.
    2. In rank order, each robot not yet moved tries its cells, nearest to its goal first, staying included.
    3. If another robot stands in the chosen cell, that robot must move first, anywhere but into the first robot’s cell. If it cannot, the first robot tries its next cell. That is the backtracking.
    4. Repeat for planning.window steps (20 by default) to make the plan, and replan every few steps.

    Why we use it

    • It never searches: 0.11 ms per call at p95 with 25 robots, the fastest planner here.
    • It never fails: every robot always has a next cell, so the fleet keeps moving.
    • The dead-end rule fixes its one blind spot on this map. Every parking cell is a dead end under the endpoint rule. In Dead end with the rule off, the two robots wait for each other forever.

    The catch: it looks one step ahead. In Head-on, the robot that gives way backs up to the end of the aisle instead of into the side cell, and the two shuffle without passing. In the fleet that costs throughput: 22% less than WHCA* at 25 robots and 0.2 orders per step.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    PIBT, a window aheadHundreds of robots, replanned every few steps.Detours in one-cell aisles.planning.method pibt
    PIBT with swapsRobots meet head-on in narrow aisles.More rules, still greedy.
    LaCAMYou want PIBT’s speed with a completeness guarantee.A search over PIBT steps: more memory and code.
    Windowed CBS or ECBSPlan quality matters more than speed.Tree search per call.ecbs

    Tick step 6: when to plan

    Rolling horizon

    Replans every robot every few steps, conflict-free for a short window, so new orders reach the plans quickly and no plan has to be right to the end.

    Stage 4, shippedfleet_core/src/fleet_manager.cpp

    replan_period
    window
    A call fails

    How it works

    1. Every replan_period steps, every robot gets a new plan from where it stands, conflict-free for the next window steps.
    2. A robot working at a station is no agent. It holds its station as an obstacle until its service ends.
    3. Between calls, robots follow their plans. replan_period is at most window, so plans always last until the next call.
    4. A call that finds nothing within the budget changes no plan. The old plans stay conflict-free, and the call runs again the next step.

    Why we use it

    • Orders keep arriving, so plans to the goal go stale. A short window costs less and is redone soon anyway.
    • All five planners plug into one loop. With the toggle off, prioritized, CBS and ECBS plan robots whose goal changed all the way to the goal.
    • When a station has no free waiting cell, a robot waits at home. Waiting on a station or in an aisle can block the very robot it waits for, and at 25 robots that locked the fleet.

    The catch: a robot whose goal changes waits for the next call, up to replan_period − 1 steps. Planning every step removes that wait and multiplies the planning work by replan_period.

    Measured: one hour of orders, every planner

    bench_planning on warehouse_small with the optimal matching: one simulated hour per run, seeds 1 to 10, the same orders for every planner. No coordinated run had a safety stop. Independent A* froze all 60 of its runs. CBS ran only up to 10 robots.

    Orders per step
    Robots

    Orders delivered per hour, steady state, mean over 10 seeds

    Every planner, fleet size and rate as a table
    PlannerOrders per stepRobotsOrders per hourCalls with no planFrozen runsPlanning p95

    Tick step 7: how far each robot may go

    Late robots: the dependency graph

    Plans are conflict-free only while robots keep to time. The dependency graph keeps the order in which the plans cross each cell, so a late robot holds up the robots behind it instead of meeting them.

    Stage 5, shippedfleet_core/src/traffic/traffic_manager.cpp

    Traffic rule
    R0 runs late

    Thin lines: what is left of each plan. Arrows: a move that did not happen. Curved arrows: who waits for whom.

    How it works

    1. When plans are committed, each one is cut into stays: which cell, arriving at which step, and the path index the robot reaches by leaving.
    2. For every cell, the stays are sorted by plan time. A move into a cell depends on the stay just before it there.
    3. Every tick, a robot may take its next move if the robot before it in that cell has left. It may also go while that robot leaves in the same step; it then follows it out, and yields if that robot misses the step after all.
    4. The permit runs on while each further move is already safe, and stops before a closed cell.
    5. It needs plans made together, which rolling replanning makes. Planning to goals uses reservations instead.

    Why we use it

    • Real robots run early or late: a slow wheel, a person in the way, a missed message. Keeping each cell’s order keeps conflict-free plans safe however late anyone runs (Hönig et al., 2019).
    • A late robot holds up only the robots planned after it in some cell. Everyone else keeps going.
    • In 320 hour-long runs with robots missing up to one step in three, no planner had a single safety stop. With both rules off, the same robots at one missed step in ten were stopped by robot safety 44,170 times.

    The catch: delays ripple. With 25 robots missing one step in five, PIBT robots spent 1329 robot-steps per run held and 2421 yielding, and delivered 4% less than on time. ECBS lost 0.1%.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Dependency graphPlans are made together.One slow robot slows everyone behind it.In use
    ReservationsPlans are made a few robots at a time.Ignores the planned order, so robots can end up face to face.Next section
    k-robust plansNo robot is ever more than k steps late.k-step gaps between robots cost throughput, and a longer delay still collides.
    Replan every stepPlanning is cheap.Planning work multiplies by replan_period.replan_period: 1
    Robot safety aloneStops are acceptable.Every stop is a near miss, and stops can lock up.Both rules off

    Measured: one hour with late robots

    bench_planning on warehouse_small with the optimal matching: one simulated hour per run, seeds 1 to 10, both traffic rules on. Every robot misses each step with the chance shown. 320 runs with late robots, no safety stop.

    Chance to miss a step
    Robots

    Orders delivered per hour, steady state, mean over 10 seeds

    Which rule keeps them apart, at one missed step in ten
    PlannerRobotsRules onOrders per hourSafety stopsCycles replanned per run

    Tick step 7: who may enter a cell

    Reservations and deadlock recovery

    When robots replan a few at a time, there is no shared plan order to keep. Reservations go by where the robots are, and robots that end up waiting for each other in a cycle get new plans.

    Stage 5, shippedfleet_core/src/traffic/traffic_manager.cpp

    Traffic rule
    Deadlock recovery

    Ringed robots: on a cycle. Recovery replans with the prioritized planner, the older order first.

    How it works

    1. A robot may enter a cell only while it holds the cell’s reservation: the cell is empty, or its robot leaves in the same step.
    2. One robot per cell per step. When two want the same cell, the older order wins, then the lower id.
    3. Two robots that want each other’s cells both wait.
    4. A held robot waits for at most one robot, so following those links finds every cycle.
    5. Recovery: rolling replanning moves its next call to the next step. Planning to goals replans the whole fleet together at the next step.

    Why we use it

    • Planning to goals plans each robot around the others’ plans, at different times. Late robots break those plans, and there is no single plan order for the dependency graph to keep.
    • Planning to goals with one robot step in ten missed: with reservations and recovery, 0 of 50 runs froze and none had a safety stop. Without recovery 50 of 50 froze. Without reservations robot safety stopped robots 1,052,387 times and 50 of 50 runs froze.

    The catch: reservations do not keep the planned order. With rolling replanning and the dependency graph off, PIBT robots at 25 robots formed 75 cycles per hour that needed a replan, where the dependency graph formed none.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Find cycles, replan the fleetA planner is at hand and cycles are rare.A replan per cycle, and the new plans can meet again.In use
    Replan only the robots on the cycleBig fleets, small cycles.They may find no way out around the others’ plans.Tried first; froze a run
    Back one robot outCheap local fixes.Who backs out, and where to, is hard to get right, and it can loop.
    One-way aislesThe layout allows a direction per aisle.Longer trips, and rules for every map.

    Tick step 7: how far each robot may go

    Lost reports: where a silent robot may be

    A lost report leaves the coordinator a step behind: the robot may already stand in the next cell. Until it is heard again, no other robot gets a permit into any cell it may have reached.

    Stage 6, shippedfleet_core/src/traffic/traffic_manager.cpp

    Coordinator
    R0’s reports

    Dashed outline: a cell R0 may have reached, held for it. Arrows: a move that did not happen. Curved arrows: who waits for whom.

    How it works

    1. Every command adds to the robot’s reach: the cells from its last reported place on its path up to its permit. A report that arrives resets the reach to the cell it gives.
    2. A robot heard this step is where it says. It wins its cell over a silent robot last seen there.
    3. No other robot gets a permit into a cell a silent robot may have reached, or a permit to follow it.
    4. A silent robot keeps its plan. Replanning it from where it was last seen would plan the others around the wrong cell.
    5. A permit is good only in the step its command arrives. A robot that hears nothing stands still, which the traffic rules already treat as a missed step. After 5 steps without a command it holds in safe stop, and station service waits.

    Why we use it

    • Radios drop messages. A rule that trusts the last report is wrong exactly when one is lost.
    • In 320 hour-long runs with up to three messages in ten lost, robot safety stopped no robot, no run broke an invariant, and none froze.
    • Two earlier versions failed. Trusting the last report let a robot that had moved on be run into, at step 135 of the first run with one message in five lost. Holding only the cells a silent robot may reach still failed at step 1,362: the coordinator believed two robots on one cell and told a third to follow the wrong one.

    The catch: every lost command costs a step. With 10 robots, 8% to 12% fewer orders per hour at one message in twenty lost, and 49% to 55% fewer at three in ten. With 25 robots there is slack: windowed_coop_astar, prioritized, and ecbs lost at most 2% up to one message in five; pibt lost 33% at one in five.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Hold every cell a silent robot may reachMessages get lost, and no robot may be run into.A lost message holds others up, and a lost command costs a step.In use
    Trust the last reportThe link never loses a message.A robot that moved on gets run into.Stage 5
    Acknowledge and resendThe link is slow but rarely loses a message twice.A resent command arrives a round trip late, and the robot must hold until then.
    Permits for several stepsMessages are lost often and robots are far apart.A lost report hides more cells, so more of the floor is held.
    Predict from the planRobots keep to time.A late robot breaks the guess.Late robots

    Measured: one hour with lost messages

    bench_planning on warehouse_small with the optimal matching and rolling replanning: one simulated hour per run, seeds 1 to 10. Every report and every command is lost with the chance shown; 0 is the run without loss. 320 runs with lost messages, no safety stop.

    Chance a message is lost
    Robots

    Orders delivered per hour, steady state, mean over 10 seeds

    Robots declared dead by mistake, all four planners and both fleet sizes
    Chance a message is lostFalse deathsBack before the run endedOrders delivered twice

    Tick step 2: check faults

    Leases: robots that go quiet

    The coordinator cannot tell a dead robot from one out of touch. A robot silent for longer than its lease is treated as dead: its order goes back to the queue, and the cells it may stand on close. If it reports again, it rejoins.

    Stage 6, shippedfleet_core/src/faults/fault_manager.cppfleet_core/src/fleet_manager.cpp

    R0

    Dark cells: closed in the coordinator’s map. Dashed outline: a cell held for silent R0. Gray: a dead robot. This coordinator plans only when cells close or open, or a robot rejoins.

    How it works

    1. Every report renews a robot’s lease. A robot silent for more than 5 steps (heartbeat_timeout_steps) is declared dead.
    2. Its order goes back to the queue. It keeps its age, so it is first in line again.
    3. The cells it may stand on close in the coordinator’s map, except a cell a robot heard this step stands on. A station among them stays its turn, so no other robot is sent there. The fleet replans around them.
    4. A dead robot stays where it stopped until an operator takes it away (rescue_steps, 120) and tells the coordinator. Its cells open.
    5. A robot declared dead that reports again rejoins. Its cells open, a delivery it made meanwhile counts, and it keeps the order it carries unless another robot has picked that order up.

    Why we use it

    • A lease needs nothing but the reports robots already send every step.
    • In 120 hour-long runs of the fault scenario below, the coordinator declared both dead robots dead every time. Robot safety stopped no robot, no run broke an invariant, and none froze.
    • The fuzz harness ran 1,000 random scenarios: generated warehouses, fleets, planners, delays, closures, and faults. Every one ran without a violation or an exception, and twice to the same log hash. They held 709 robot deaths, 4,532 false deaths, and 1,108,831 lost messages.

    The catch: a slow link looks like a death. With one message in five lost, the loss runs above declared robots dead by mistake 280 times in 80 runs, and 2,448 times at three in ten. A false death sends an order back to the queue, and an order can end up delivered twice: once in the 120 fault runs.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Heartbeat leaseReports already arrive every step.A shorter lease reacts faster but raises more false alarms.In use
    Safe stop on the robotA robot cut off must not wander.Keeps the floor safe, and tells the coordinator nothing.In use
    Separate health channelA second radio or a wired dock is at hand.More hardware, and it can fail too.
    Wait for an operatorDeaths are rare and someone is near.Orders and aisles wait until then.
    Agreement among coordinatorsSeveral coordinators must agree on who is alive.More messages, and slower to decide.

    Measured: deaths, outages, and lost messages together

    benchmarks/configs/faults.yaml: the planning scenario with one message in ten lost, robot 0 dying at step 600 and robot 3 at step 1800, robot 5 out of touch for 60 steps from step 900, and robot 7 for 3 steps from step 2400. Orders per hour in steady state, mean over seeds 1 to 10. No faults: the same runs without them.

    RobotsPlannerReplanningNo faultsFaultsOrders requeued per run

    On the robots: simulator ground truth

    Robot safety rules

    The last line of defence. It stops a robot before contact and records why.

    Stage 1, shippedfleet_core/src/execution/grid_executor.cpp

    Scenario

    Arrows: where each robot wants to go. Dashed circles: where a robot just came from.

    How it works, every step, in this order

    1. Intent: each robot wants the next cell on its path. Robots working at a station, waiting, or out of path stay put.
    2. Same cell: when two robots want one cell, the lower robot number goes.
    3. Head-on: two robots that want each other's cells both stop.
    4. Occupied: nobody enters a cell whose robot stays. Each new stop can block the robot behind, so repeat until nothing changes.
    5. Everyone left moves at once. Following is fine. So is a loop of four or more.

    Why we use it

    • The planner can be wrong, and independent A* always is. Safety keeps the physics honest: no shared cells, no swaps.
    • Every stop is recorded. With coordinated planners, any stop counts as a violation.
    • The same inputs give the same stops, so a run replays to the same trajectory hash.

    Why four or more? A grid has no loop of three cells. The smallest loop is a 2×2 block, and a loop of two is a swap.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Lower number winsEvery replay must give the same result.The same robots always yield.In use
    Random winnerFairness matters more than exact replays.Needs its own seeded random stream.
    Oldest order winsLate orders should catch up.Robots without orders can starve.
    Ban followingReal robots run early or late (k-robust plans).Gaps between robots cut throughput.
    Velocity obstacles (ORCA)Robots move freely, not on a grid.No deadlock guarantee either.
    Reservations, dependency graphPlans must be safe before anyone moves, even when robots run late.More work in the coordinator, every tick.In use since Stage 5

    Outside the coordinator: measurement

    Stall and deadlock detection

    Finds robots that stop making progress, and groups locked together. Stage 2 uses it to measure the baseline.

    Stage 2, shippedfleet_core/src/verification/progress_monitor.cpp

    Scenario

    Arrows: who waits for whom. Thick ring: on a cycle. Pill under a robot: steps without progress.

    How it works

    1. Every step, each robot stopped by robot safety points at the robot standing in the cell it wanted.
    2. A robot wants one cell, so it points at one robot at most. Following the pointers gives chains.
    3. A chain that ends at a robot that is not waiting is a queue. It clears when the front robot moves.
    4. A chain that loops back is a cycle. Every robot on it waits for the next, so none can move until a plan changes. A cycle that lasts 10 steps is a deadlock, and robots whose chain runs into it are behind it.
    5. Separately, a robot with work that makes no progress for 30 steps is stalled. If nobody makes progress for the rest of the run, the run froze at that step.

    Why we use it

    • Throughput says the baseline failed. The monitor says when, where and who.
    • Each robot has one pointer at most, so finding every cycle takes one pass over the fleet per step.
    • It reads ground truth only, like the invariant checker, so the coordinator never grades itself.
    • It turns “it got stuck” into numbers every later stage must beat: first deadlock, frozen from.

    Why wait 10 steps? A new plan can break a cycle in a step or two, as in Breaks early. A deadlock does not break. In the default run, one cycle broke after 16 steps, only because a station turn handed one robot a new path.

    The baseline, measured

    Nearest allocation and independent A* on warehouse_small: 5, 10 and 25 robots, seeds 1 to 5, one simulated hour each. Every run deadlocks, and every run freezes before step 350. The axis stops at step 400 because nothing happens after it.

    Robots still moving (the bar ends where the run froze) First deadlock Last delivery
    All 15 runs as a table
    RobotsSeedFirst deadlockLast deliveryFrozen fromDeliveredDeadlocks: robots, formed→broke

    Your default run, frozen

    The shipped config, 10 robots, seed 1, replayed from the C++ code with the monitor’s view: steps 111 to 146, then step 217. Showing columns 0 to 15 of 38, where the jam forms. After step 217 nothing moves for the remaining 3,382 steps.

    Alternatives, and when to pick them

    OptionPick it whenCatchHere
    Wait-for cycles on ground truthMeasuring a run: is the fleet locked, since when, and who?Sees robot-on-robot waits only. A robot waiting on parking for a station turn has no arrow.In use
    Stall timeoutA watchdog on one robot, with no view of the others.Cannot tell a queue from a deadlock. The threshold is a guess.Also here: 30 steps
    Fleet-wide no-progress alarmOne alert for an operator.Fires late and names nobody.Also here: frozen from
    Resource graph with stationsWaits for stations and reservations matter too.A bigger graph, and it needs the coordinator’s state.
    Detection in the coordinatorRecovery has to act on it.Works on beliefs, which lag the truth.In use since Stage 5
    Prevent deadlocks insteadYou can afford coordinated planning.You still need detection to prove it worked.Stages 4, 5