Blog

OpenClaw scheduler and lane isolation: how the concurrency model works

OpenClaw’s scheduler is a lane-aware, in-process FIFO queue: every agent run is admitted first through a per-session lane that allows one active run per session, then through a global lane (main for chat, cron-nested for cron and heartbeats, subagent for spawned children) whose width caps how many runs execute at once. This page describes the model as it ships in OpenClaw 2026.9.4, read from the source at that tag, including what changed since spring 2026 and which isolation gaps are still open on September 13, 2026.

The model at a glance, OpenClaw 2026.9.4
  • Two-level admission: session:<key> (width 1, always) → a global lane.
  • main is min(16, max(8, CPU parallelism)) unless agents.defaults.maxConcurrent is set.
  • cron, cron-nested and hook-dispatch are fixed at 8; cron.maxConcurrentRuns is a retired key.
  • Inside a lane, foreground turns (user, manual) start before normal, and normal before background (cron, heartbeat, memory, overflow).
  • A hung run holds its global slot for the run timeout plus 30 seconds; the upstream default timeout is 48 hours.
  • Still open: per-agent lanes (#43235) and a lane parameter on sessions_spawn (#10467). All agents share main.
Mid-incident right now?

If a cron job says enqueued and never runs, start with the cron enqueued-but-idle runbook. This page explains the machinery that runbook is poking at.

How does OpenClaw admit an agent run?

Every embedded agent run goes through two queues in sequence (src/agents/embedded-agent-runner/run-orchestrator.ts):

  1. The session lane. The runner resolves session:<sessionKey> and enqueues there. Lanes that nobody configured are created with maxConcurrent: 1, so a session can never have two runs executing at once. This is not a setting; it is how the lane is constructed. In 2026.9.4, CLI, embedded and Codex runs share the same session-key lane, and an idle session lane is retired from the lane map once it is empty.
  2. The global lane. Inside the session-lane task, the runner enqueues the actual run into a global lane. Chat turns go to main. A caller that passes cron is remapped to cron-nested, because the outer cron job already holds a cron slot and letting its inner agent turn wait on the same lane would deadlock. Spawned children go to subagent.

The ordering matters for how stalls look. A run waiting in the session lane is waiting for its own previous turn. A run waiting in the global lane is waiting for everyone else’s. The runner marks both phases on the reply operation (markWaitingForGlobalLane, markGlobalLaneWaitEnded), and the second marker is the one you will see in watchdog output as global_lane:wait_ended.

Which command lanes exist in OpenClaw 2026.9.4 and how wide are they?

Lane Width Configurable? What runs there
main min(16, max(8, os.availableParallelism())) agents.defaults.maxConcurrent inbound chat turns from every channel
session:<key> 1 no the per-session serializer in front of every global lane
cron 8 no (fixed constant) the outer wrapper of each cron job
cron-nested 8, shared budget with hook-dispatch no isolated cron agent turns and heartbeat runs
hook-dispatch 8 when hooks are enabled, 1 slot reserved inside the cron budget no external hook agent runs
subagent 8 agents.defaults.subagents.maxConcurrent every sessions_spawn child
nested 1 no sessions.send and other nested agent work
background 3 in total no Skill Workshop reviews (at most 1) and plugin background work such as dreaming

Two things in that table are new since mid-2026. First, main is sized from CPU parallelism; the 2026.8.1 release notes call it “CPU-scaled foreground concurrency … bounded between 8 and 16 simultaneous runs, while preserving explicit operator limits”. The 2026.6.2 source still hard-coded 4. Second, lanes can now belong to a capacity group: a hard aggregate budget across several lanes with per-member reservations. cron-nested and hook-dispatch share one budget of 8, and hooks reserve one slot inside it, so enabling hooks costs cron nothing in aggregate concurrency but guarantees hooks cannot be starved. main, cron, subagent, nested and every session lane are deliberately ineligible for groups, because other lanes wait on them synchronously and a group wait on top of that could deadlock.

What order does work run in inside a lane?

Each lane keeps three FIFO rings and always drains foreground, then normal, then background. The run’s trigger decides the ring:

Trigger Ring
user, manual foreground
cron, heartbeat, memory, overflow background
anything else normal
restart-recovery input, inter-session (agent-to-agent) input forced to background

This landed in 2026.5.18 (“prioritize manual user turns ahead of queued cron and maintenance work in the same session lane, so visible follow-ups no longer wait behind background runs”). Before it, a human reply queued behind a heartbeat in the same session waited its turn. It does not preempt: a background run that is already executing keeps its slot; the ring only decides who starts next.

What releases a lane slot when a run hangs?

The lane queue has had a task-level timeout since 2026.4.29. The budget for a global-lane task is the run’s own timeout plus a 30-second grace (EMBEDDED_RUN_LANE_TIMEOUT_GRACE_MS). The run timeout comes from agents.defaults.timeoutSeconds, and the upstream default is 172,800 seconds — 48 hours. On an unconfigured self-host, a run whose promise never settles holds its main slot for 48 hours and 30 seconds before the lane reclaims it.

When the lane does reclaim a slot, the error names the reason. 2026.9.4 distinguishes five:

  • task-budget — elapsed time reached the budget.
  • owner-deadline — the attempt handed the lane its own deadline and that deadline passed.
  • progress-idle — no progress signal for the budget window.
  • abort-grace — the run was aborted but did not unwind within 30 seconds, so the slot was forced free.
  • release-signal — the runtime asked for the lane back after an attempt timeout.
Command lane "<lane>" task timed out: <cause> (task budget <n>ms, elapsed <n>ms)

On timeout the runner also aborts the attempt’s action signal, so a released slot always means the run is being cancelled, not orphaned.

Two other mechanisms free lanes without a timeout:

  • Stall recovery. With diagnostics on, a session stuck in processing is classified session.long_running, session.stalled or session.stuck; session.stuck always runs a recovery that can release the session lane, and session.stalled past the abort threshold can trigger an active abort. The old diagnostics.stuckSessionWarnMs and stuckSessionAbortMs keys were retired in the same 2026.8.1 change; the thresholds are built in.
  • In-process restart. After a SIGUSR1 restart, resetAllLanes() bumps every lane’s generation, clears active task ids, keeps queued entries and re-pumps them, so stale completions from the previous generation are ignored instead of blocking new work. This has been in place since 2026.2.13.

What is fixed and what is configurable in 2026.9.4?

  • cron.maxConcurrentRuns is a retired config key. openclaw doctor --fix strips it, and the cron cap is a fixed 8 for both the outer cron lane and cron-nested. It was retired in 2026.8.1 (#111382, “purge numeric tuning knobs behind built-in defaults”); guides that tell you to raise it describe an older contract.
  • agents.defaults.maxConcurrent still sets main; leave it unset to get the CPU-scaled 8–16.
  • agents.defaults.subagents.maxConcurrent still sets subagent (default 8).
  • nested is 1 and has no knob. The 2026.9.4 source says so in one comment: “sessions.send work uses a shared nested lane with no config knob.”
  • hooks.enabled is the only switch that changes lane layout at runtime; it creates the cron-hooks group with the one-slot reservation.

Where are the lane isolation gaps as of September 13, 2026?

  • No per-agent lanes. Every agent’s inbound turns share main. Issue #43235 (“Support per-agent command lanes for multi-agent deployments”) is still open on September 13, 2026; the maintainers’ triage bot parked it on April 30, 2026 as an idea with no sponsor, and its last activity is September 9, 2026. Its predecessor #42686 was closed as a duplicate, and the workaround upstream wrote into it is blunt: “Deploy a separate openclaw gateway process for agents that run long tasks.”
  • No lane parameter on sessions_spawn. Issue #10467 is open; the maintainer review of September 9, 2026 confirms the tool schema still has no lane parameter and every spawn path dispatches to the single subagent lane.
  • The shared main cap is wider, not gone. Issue #16055 (five Telegram bots serialized through main) was closed as stale in April 2026; the mechanism it described is unchanged, the lane is just 8–16 wide instead of 4.
  • One correction to older write-ups, including our own runbook. Issue #48488 (“Lane queue has no task-level timeout”) was closed as not planned on June 16, 2026, which reads like the gap survived. It did not: the timeout shipped in 2026.4.29 under a different title. If a page tells you a hung promise jams a lane forever, it is describing a runtime older than April 2026.

How do you read lane state in logs and diagnostics?

The wait diagnostic fires when a task waited more than two seconds before starting:

lane wait exceeded: lane=<lane> waitedMs=<n> queueAhead=<n> activeAhead=<n> activeNow=<n> queueBehind=<n>

queueAhead and activeAhead are captured at enqueue time; activeNow and queueBehind at dequeue. A lane whose activeAhead equals its width was saturated when you enqueued; a lane with activeAhead=0 and a long wait was either suspended (width 0) or blocked by a group budget or a sibling’s reservation, which the diagnostics.lanes gateway method reports as blockedBy. The Control UI shows the same data in its System busyness overlay, with static lanes listed individually and all dynamic session lanes collapsed into one row. The real-world line that opened issue #43235 read lane=main waitedMs=151783 queueAhead=1 — eight Feishu bots sharing one main lane on a 4-wide default.

A worked reading from a managed instance on OpenClaw 2026.9.3, September 9, 2026 (UTC): a user message was appended at 20:37:55; the watchdog reclaimed the reply at 20:43:58 after 363 seconds with last marker global_lane:wait_ended; cooperative recovery reported aborted=false drained=false forceCleared=true released=1; the command lane hit its 30-second abort grace at 20:44:28. Everything the lane machinery did there is what the source says it does. The important inference is the marker: wait_ended means the run was admitted, so the six minutes were spent inside the run, not in the queue. The cause turned out to be prompt preparation waiting on a memory-manager rebuild, a 2026.9.3 defect unrelated to lanes. If you see wait_ended followed by silence, stop tuning maxConcurrent.

How does Lobsterland run the lane model?

Lobsterland does not replace the scheduler; it constrains the one number that decides how long a bad run can hold a slot. Managed instances are generated with agents.defaults.timeoutSeconds: 1800, so a hung run’s global slot is reclaimed after about 30 minutes 30 seconds instead of 48 hours 30 seconds. maxConcurrent and subagents.maxConcurrent are left at upstream defaults.

The isolation upstream lacks within one process, Lobsterland provides between customers by construction: each instance is its own gateway process in its own pod, so one customer’s saturated main lane cannot queue anyone else’s turns. If your multi-agent design needs a slow agent kept away from an interactive one today, the honest answer is upstream’s own: a separate gateway process per group — which is what a second managed OpenClaw instance is. Cron-heavy workloads, where the fixed cap of 8 and the cron-nested remap do the most work, are covered on the hosted cron automation page, and the routing side of running several agents on one host on the multi-agent OpenClaw hosting page.

OpenClaw lane concurrency FAQ

What is a command lane in OpenClaw?

A named FIFO queue inside the gateway process with a concurrency cap. Every agent run is admitted through a per-session lane (width 1) and then a global lane such as main, cron-nested or subagent. Lanes are plain TypeScript promises; there are no worker threads.

What is the default maxConcurrent in OpenClaw?

Since OpenClaw 2026.8.1, the main lane defaults to min(16, max(8, available CPU parallelism)). Before that it was 4. The subagent lane defaults to 8, cron and cron-nested are fixed at 8, nested is fixed at 1, and background plugin work shares a budget of 3.

Does cron.maxConcurrentRuns still work?

No. In OpenClaw 2026.9.4 it is a retired configuration key that openclaw doctor --fix removes, and cron concurrency is a fixed 8.

Why does a cron agent turn run in cron-nested instead of cron?

Because the outer cron job already holds a cron slot. If the inner agent turn queued on the same lane it could wait on itself. The runner remaps cron to cron-nested, and heartbeat runs use cron-nested for the same reason.

How long does a hung run block a lane?

For the run timeout plus 30 seconds. The upstream default timeout is 48 hours; Lobsterland sets 30 minutes on managed instances. Stall recovery (session.stuck, session.stalled) can release a session lane earlier.

Can I give one OpenClaw agent its own lane?

Not in OpenClaw 2026.9.4. Per-agent lanes are tracked in issue #43235 and remain unimplemented; the maintainers' stated workaround is a separate gateway process for the slow agent.

Run the scheduler with a 30-minute leash

Managed instances ship with the run timeout that reclaims a stuck lane slot in half an hour, one gateway process per customer, and the current stable OpenClaw image. Compare that with tuning it yourself before deciding where to run OpenClaw.

Cookie preferences