Hermes gateway restart loop after a crash: the state.db check

If your Hermes gateway was killed and now restarts every few minutes without coming up, it is probably stuck in its post-crash database check. After an unclean exit, Hermes scans the wholestate.db with PRAGMA quick_check(1) before it connects Telegram or Slack or starts cron. On a large store, the scan outlasts whatever deadline is watching the process. The process gets killed and the next start begins the same scan from zero. To recover, stop the gateway, run the check yourself with no deadline, and if it prints ok, move the stale lifecycle sentinel aside and start again. We hit this in September 2026 on a hosted Hermes instance with a 13 GB store under a 5-minute startup budget, and fixed it by moving the check off the boot path. Upgrading Hermes alone would not have been enough, for reasons covered below.

At a glance

How to confirm this is your restart loop

Paths below use the default Hermes home, ~/.hermes. The official Docker image uses/opt/data. If you set HERMES_HOME, use that.

cat ~/.hermes/state/gateway.lifecycle.json
ls -lh ~/.hermes/state.db*
tail -n 40 ~/.hermes/logs/gateway-startup-watchdog.log
tail -n 3 ~/.hermes/logs/gateway-exit-diag.log

You are in this loop if all of these are true:

• The sentinel says "phase": "running" with a PID that no longer exists. That is how Hermes records "the last run never shut down".
• state.db is several gigabytes. Hermes's own code comment estimates about 2 seconds to check a healthy 500 MB store. At multiple gigabytes, the check takes minutes.
• The logs go quiet after startup begins. Hermes writes its Previous gateway life (pid=…) exited UNCLEANLY warning and the gateway.previous_unclean_exit line ingateway-exit-diag.log only after the check returns, so neither appears while the loop continues.
• On a release before v2026.9.21, the watchdog log has a startup_watchdog.fired record with a stack through _report_unclean_exit or check_state_db_integrity. That confirms it.

The timing is the quickest tell. If the process restarts every few seconds, seeother restart loops below. If each attempt lasts about 300 seconds, or as long as your probe or service timeout, it is this check.

Why the check never finishes

Hermes keeps a lifecycle sentinel at state/gateway.lifecycle.json. A clean shutdown rewrites it to"phase": "exited". If the gateway dies without a shutdown path, from SIGKILL, the kernel OOM killer, a lost VM or node, the sentinel still says running. On the next start, Hermes treats that as an unclean exit. A crash in the middle of a WAL checkpoint can leave half-written pages, so Hermes checks the database before trusting it.

The order of operations is what turns a slow start into a permanent loop:

1. The startup watchdog is armed as the process starts.
2. record_startup() finds the stale sentinel and runs PRAGMA quick_check(1) on the whole store. The 1 limits how many errors are reported, not how much it reads.
3. Only after the check returns does Hermes claim the sentinel for the new process.
4. Only after that does it connect channels, start cron and report itself ready.

If anything kills the process during step 2, step 3 never happens. The sentinel still describes theprevious crashed process. The watchdog's own exit handler won't overwrite a sentinel owned by another PID. So the next start finds the same unclean exit and starts the full scan from the beginning.

Reported scan times show how quickly this goes beyond normal startup budgets:

Disk speed matters as much as size, so treat these as orders of magnitude, not a formula. The 5.42 GB store already used two thirds of Hermes's default 300-second startup deadline before any other startup work.

Recover a gateway stuck in the post-crash check

Run these steps in order. The reporter of upstream issue #115542 used the same order to recover a host with a 37 GB store. Stop the gateway and whatever restarts it (the systemd service, your process manager, or the Kubernetes deployment) first. Otherwise it keeps relaunching the scan beside yours.

# 1. Stop the loop
hermes gateway stop

# 2. Keep a copy of the sentinel; it is your evidence of the crash
cp ~/.hermes/state/gateway.lifecycle.json ~/hermes-lifecycle-before-recovery.json

# 3. Run the same check with no deadline, at low priority
#    (needs the sqlite3 CLI; drop ionice on macOS)
time nice -n 19 ionice -c3 sqlite3 ~/.hermes/state.db "PRAGMA quick_check(1);"

Wait for it to finish, however long it takes. Then act on the result:

• It prints ok. The database is fine; it was only slow. Move the stale sentinel aside so this one start skips the scan, then start the gateway:

  mv ~/.hermes/state/gateway.lifecycle.json ~/.hermes/state/gateway.lifecycle.json.checked-ok
  hermes gateway start

  With no sentinel, Hermes sees no unclean exit, claims a fresh sentinel and starts normally. Check that the new sentinel says running with the new PID, and that a channel message gets a reply.
• It prints anything else. Stop here and keep the sentinel. Hermes would logstate.db FAILED integrity check after an unclean gateway exit and point you tohermes doctor. Back up the database with its -wal and -shm files before you try any repair. The Hermes session-storage recovery guide covershermes sessions recover --inspect-only.

Don't skip the check and delete the sentinel blind. The sentinel is the only signal that makes Hermes look for corruption after a crash. Removing it without checking defeats the reason the check exists.

Stop the restart loop from coming back

Upgrade to Hermes v2026.9.21 or later. Upstream fixed issue #115542 on 2026-09-20. The check now renews a startup-watchdog lease named state_db_unclean_integrity_check while SQLite is making progress, so Hermes no longer kills its own healthy check with exit 75. The fix is in v2026.9.21 and not in v2026.9.11.

Know what the upgrade does not change:

• The check is still synchronous by design. Telegram, Slack and cron stay down until it finishes. On the 37 GB store, that is more than an hour after every crash.
• Deadlines outside Hermes still apply. A Kubernetes startup probe with a fixed budget kills the container whatever Hermes's own lease says. On systemd, the default generated unit is Type=simple and has no start deadline. If you enabled gateway.systemd_watchdog_seconds, the unit becomesType=notify. Hermes reports ready only after channels and cron are up, and the generated unit sets no TimeoutStartSec, so systemd's default start timeout (90 seconds unless your system changes DefaultTimeoutStartSec) covers the whole scan.

Make the store smaller. Scan time follows database size, and on older installs most of that size is a duplicated search index. In the 5.42 GB report, hermes sessions optimize-storage cut the check from 196.75 to 78.39 seconds without deleting any messages. We cover the procedure, including what to do when the disk is too full to run it, inshrinking a Hermes state.db that is too large.

Fit external deadlines to the scan, or keep the scan away from them. If a probe or service timeout watches startup, give it room for the scan time your store actually needs. Measure it once with step 3 above. The upstream issue explicitly warns against two shortcuts: deleting or ignoring the sentinel, and permanently raising Hermes's global startup timeout. Setting HERMES_STARTUP_WATCHDOG=0 belongs with them. The watchdog exists to restart a gateway that deadlocks before its event loop runs, and switching it off removes that protection for every start, not just the slow one.

Other Hermes gateway restart loops

How we handle the post-crash check on managed Hermes

Our hosted Hermes instances run on Kubernetes. The startup probe gives a gateway about 5 minutes to report that its channels are up. When a 13 GB store went through an unclean exit, the check ran past that budget, and Kubernetes restarted the container into the same scan, over and over. On 2026-09-21 we changed our Hermes image so the automatic post-crash check runs outside the boot path:

• Startup claims the sentinel and records the crash right away, with the integrity result markedpending. Channels and cron start normally.
• A background thread runs the same quick_check(1) on its own SQLite connection. It reads only and keeps Hermes's corruption detection intact.
• When it finishes, it appends a gateway.state_db_integrity record with the verdict and elapsed seconds to gateway-exit-diag.log. A failure is logged as an error that points tohermes doctor.
• pending never counts as healthy. If the process dies before the check finishes, the next unclean start schedules another check. Running hermes doctor by hand still checks synchronously.

The patch matches the exact upstream code it replaces, and our image build fails if a Hermes release changes that code. We re-checked it when we moved to v2026.9.21, which already contains the upstream lease fix. We kept our change because the lease doesn't help with a deadline outside Hermes, and an agent that is dark for the length of a scan after every crash is still an outage. This change does not repair databases. If the check ever reports corruption, we keep the store intact and investigate before anything touches the data.

If you would rather not own this kind of recovery, see managed Hermes runtime hosting and the plans and their storage sizes. If you are still choosing a runtime,compare the OpenClaw and Hermes runtimes or start withwhat the Hermes agent runtime is. Another Hermes failure we have fixed in production is thememory character-limit deadlock.

FAQ

Sources

• NousResearch/hermes-agent issue #115542, "Gateway restart loop on large state.db: unclean-exit quick_check lacks progress lease" (opened 2026-09-19, closed 2026-09-20), with the 37.24 GB and 5.42 GB measurements and the recovery sequence; fix commit 2a8c3b31, included in tag v2026.9.21.
• NousResearch/hermes-agent source (main, read 2026-10-08): gateway/lifecycle_ledger.py(sentinel, unclean-exit detection, the check and its lease), gateway/run.py (startup order),hermes_startup_watchdog.py (300-second default, exit code 75, dump file),hermes_cli/gateway.py (generated systemd unit).
• Hermes Agent documentation: environment-variable reference (HERMES_STARTUP_WATCHDOG,HERMES_STARTUP_WATCHDOG_TIMEOUT_S), the messaging guide's "Optional Linux event-loop watchdog", and the session-storage recovery guide.
• NousResearch/hermes-agent issue #111648 (rapid restart cycles after a deadlock).
• systemd documentation, systemd-system.conf: DefaultTimeoutStartSec=.
• SQLite documentation, PRAGMA quick_check.
• Lobsterland production operations, September 2026: the 13 GB restart loop and the hosted-image change that moved the check off the boot path.