Hermes agent Telegram stopped responding, but the gateway is still running
Read platforms.telegram.state in gateway_state.json before you restart
anything. If a Hermes agent that was answering on Telegram for weeks goes silent while Slack still
replies and cron still fires, the Telegram adapter has almost certainly exhausted its ten-attempt reconnect
ladder after a network or Telegram-side outage and handed the platform back to the gateway. On a current Hermes
build the gateway keeps retrying in the background and the state reads retrying; on builds before
v2026.7.30 it did nothing, and the bot stayed dead inside a process that looked healthy. Recover
in this order — wait one back-off interval, /platform resume telegram if it is paused, then and
only then a gateway restart — because a process restart is the one path that discards the messages Telegram queued while
the bot was down.
At a glance
| Question | Answer |
|---|---|
| Symptom | Telegram silent; Slack, cron and the HTTP API still work; the gateway process is up |
| Log line | Fatal telegram adapter error (telegram_network_error): Telegram polling could not reconnect after 10 network error retries… |
| Mechanism | Adapter ladder: 10 attempts, 5→60 s back-off; then a retryable fatal handed to the gateway |
| Where to look | $HERMES_HOME/gateway_state.json → platforms.telegram.state; or /platform list in chat |
| States | connected · retrying · fatal · paused · disabled |
| Recovery order | Wait one interval → /platform resume telegram → gateway restart (last) |
| Why restart is last | A cold start connects with drop_pending_updates=True; in-process reconnects keep the queue |
| Version line | Background reconnect queue + stranded-adapter exit are in v2026.7.30 and later; v2026.7.20 has neither |
| Not the answer | Token, allowlist, privacy mode, 409 conflicts, poll_timeout — those explain a bot that never worked |
What the log line means: the adapter's reconnect ladder
Hermes talks to Telegram by long-polling getUpdates. When that connection breaks — the host loses
connectivity, Telegram returns a burst of Bad Gateway responses, a NAT drops the idle socket — the
python-telegram-bot error callback fires with a NetworkError or TimedOut, and the
adapter's _handle_polling_network_error takes over. It is a bounded ladder:
MAX_NETWORK_RETRIES = 10,BASE_DELAY = 5,MAX_DELAY = 60— so the waits run 5 s, 10 s, 20 s, 40 s, then 60 s for the remaining attempts.- Each attempt logs
Telegram network error (attempt N/10), reconnecting in Ds, stops the updater, and starts polling again withdrop_pending_updates=False. - On the eleventh failure it logs
Telegram polling could not reconnect after 10 network error retries. Escalating to gateway recovery., sets a fatal error with codetelegram_network_errorandretryable=True, and notifies the gateway.
The ten back-off waits total 435 seconds, so the ladder gives up a little over seven minutes after the first
failure (plus whatever each connect attempt takes). That is the right shape for a laptop that went to sleep
and the wrong shape for a Telegram outage that lasts an hour, which is exactly why the escalation exists. There is also a second route to the same fatal code: a
heartbeat loop probes get_me() every 90 seconds on a separate request pool to catch sockets stuck
in CLOSE-WAIT, where the polling task blocks forever without ever raising. If stopping the updater
does not finish before the recovery deadline, the adapter declares the same retryable fatal rather than reuse
a wedged updater. Upstream issue #58270 (opened 2026-07-04) is the report that produced that
guard — "heartbeat loop stuck for hours", gateway silently dropping messages while appearing operational.
The gateway's own line is the one most people find first:
Fatal telegram adapter error (telegram_network_error): …. Note the word
fatal is about the adapter instance, not the platform. The error is flagged retryable, and the
gateway is supposed to act on that flag.
Why the gateway stays up while Telegram stays dead
Every platform runs as its own adapter inside one gateway process, and the process is designed to survive a
single adapter dying. That is a good design — a Telegram outage should not take Slack down — but it is also
why this failure is so easy to miss: process-level health checks, systemd, a Kubernetes readiness probe, all
keep reporting green. We hit the worst form of it in production. On 2026-08-10 at 21:42 UTC a hosted Hermes
instance on v2026.7.20 lost its Telegram adapter after a burst of Telegram-side Bad Gateway errors
exhausted the ladder. The adapter logged the fatal error with the text
… Restarting gateway. — and the gateway did not restart. It kept serving Slack, cron and the API
server, the pod stayed 2/2 Ready, and Telegram was silent for four days until a manual restart on
2026-08-14.
The reason is a version line. v2026.7.20 contained the polling heartbeat loop but not the two
pieces that make a retryable fatal actually retry. Once has_fatal_error was set on the adapter,
every recovery path — including the heartbeat — returned early, permanently. Both pieces landed in
v2026.7.30 (Hermes Agent v0.19.1) and are in every later tag, including the
v2026.8.31 image we ship. Read from that source, the current gateway does three things in
_handle_adapter_fatal_error_impl:
- Writes the platform's state to
gateway_state.json:retryingif the error is retryable,fatalif it is not. - Queues the platform for the background reconnect watcher before awaiting the adapter's disconnect, because a half-dead transport can wedge
close()and the old "disconnect, then queue" order left platforms permanently deaf inside a live process (upstream#80598). - In a
finallyblock, checks whether the adapter is retryable yet in neither the live set nor the queue. If so it logstelegram adapter was lost without entering the reconnection queue; exiting gateway so the service manager restarts itand stops the process on purpose, so systemd, launchd or the container runtime bring it back.
The watcher retries with a 30 s → 60 s → 120 s → 240 s → 300 s back-off and does not auto-pause on network
failures; it keeps going at the cap indefinitely so a long outage heals itself once the network returns.
Upstream issue #90386 (opened 2026-08-20) is the same shape from another deployment — gateway
running, cron and WeChat fine, Telegram dead for 4 h 17 min because the watcher never took over — and the
idempotent re-arm in _queue_retryable_fatal_platform is the fix for it. If you are on anything
older than v2026.7.30, none of this machinery exists on your instance and the version upgrade is
the fix, not a nice-to-have.
Where to read the channel state
The gateway persists per-platform state to gateway_state.json next to
gateway.pid under HERMES_HOME — ~/.hermes by default, or wherever you
pointed it (/opt/data on a hosted instance). The field you want is
platforms.telegram.state (values below are illustrative; the keys are the ones the gateway
writes):
{
"platforms": {
"telegram": {
"state": "retrying",
"error_code": "telegram_network_error",
"error_message": "Telegram polling could not reconnect after 10 network error retries. Escalating to gateway recovery.",
"retrying_since": "2026-08-10T21:42:07Z"
},
"slack": { "state": "connected" }
}
}
The values the gateway writes are connected, retrying, fatal,
paused and disabled. Two of them tell you to do nothing: retrying means
the watcher owns the problem, and paused means an operator ran /platform pause. From
inside a chat on any connected platform, /platform list shows the same queue and flags paused
entries; the gateway's own pause message points at it — "fix the underlying issue then run
/platform resume telegram to retry, or hermes gateway restart to restart the
gateway."
For proof from Telegram's side rather than Hermes', call getWebhookInfo with the bot token. It
returns pending_update_count, documented as "number of updates awaiting delivery", and the call
is read-only — it does not consume anything. A count that keeps climbing while the bot is silent is direct
evidence that nothing is polling. We used exactly this on the 2026-08-10 instance: four updates pending, and
none of them was ever delivered.
Recovery, in the order that keeps your messages
The reason order matters is a single flag. The adapter's connect() distinguishes a cold first
boot from a watcher reconnect: "is_reconnect distinguishes a cold first boot (False — drop any
stale Bot API queue) from a watcher reconnect after a prolonged outage (True — preserve the updates Telegram
queued while the bot was offline, otherwise every message sent during the outage is silently lost)". Polling
starts with drop_pending_updates=not is_reconnect. So:
- Read the state and wait one interval. If it says
retrying, the watcher will reconnect with the queue intact as soon as Telegram is reachable. Its back-off caps at five minutes; give it that before touching anything. -
/platform resume telegram. If the state ispaused, resume clears the pause, resets the attempt counter and schedules a retry on the next watcher tick — still in-process, stillis_reconnect=True, still queue-preserving. It is a no-op on a platform that is not paused; aretryingplatform is already being retried. - Restart the gateway. If the state is
fatal, orretrying_sinceis more than 45 minutes old and the state has not moved, in-runtime recovery is not working. On the CLI that ishermes gateway restart; on a hosted instance it is the Restart button on the instance overview. This connects as a cold boot and discards the pending queue. Tell the people who message the bot to re-send anything from the outage window.
This is the point most answers on this topic get backwards. "Just restart it" is the fastest route to a working bot and the only route that loses the backlog. If the backlog matters — a support bot, an alerting channel — the ten minutes spent letting the watcher finish are worth it.
Preventing the next one
- Do not run
v2026.7.20or earlier on a bot anyone depends on. The reconnect queue and the stranded-adapter exit are what turn this from "silent until a human notices" into "silent for one back-off interval". Upstream is atv2026.9.11(v0.21.2) as of 2026-09-11; our hosted image pinsv2026.8.31. - Watch
gateway_state.json, not the process. Whatever you use for monitoring, the process being up tells you nothing about a channel. Alert onstate != "connected"for the platforms you have configured, and ignore stale entries for platforms you removed — the file keeps them. - Use
/platform pausefor intent, not for outages. Pause is meant for "I am rotating this token, stop hammering Telegram". The watcher deliberately never auto-pauses on network errors, so if you find a platform paused, a person did it.
What this is not
The current search results for a silent Hermes Telegram bot answer a different question — why a bot
never worked. Those checks are correct for that case and useless for this one: an invalid
TELEGRAM_BOT_TOKEN, a user ID missing from TELEGRAM_ALLOWED_USERS, BotFather privacy
mode in groups, a 409 Conflict from two pollers on one token, and lowering
poll_timeout for networks that kill idle connections. If your bot replied yesterday and the
config has not changed, none of them moved. Start with the state file.
If you are on the OpenClaw runtime rather than Hermes, the mechanics are entirely different — see OpenClaw Telegram shows typing but never replies and the OpenClaw Telegram setup guide.
Stop babysitting your OpenClaw box
Fix it once — or stop fixing it for good.
Apply the checklist above and keep self-hosting, or skip the maintenance entirely: run your OpenClaw on managed hosting from $6.90/mo, starting with a 7-day free trial. We handle the stale locks, gateway restarts, version upgrades, and uptime — and you can import your existing instance in a couple of minutes. Cancel anytime.
- Managed hosting handles stale
.jsonl.lockfiles, gateway restarts, and version upgrades for you - Import your existing OpenClaw setup in minutes — keep your channels and configuration
- The optional $199 setup is scoped: no custom development, enterprise/SRE support, or unsupported self-hosting repair
If you would rather compare options first, review OpenClaw cloud hosting or see the best OpenClaw hosting options before deciding.
How managed Hermes hosting handles this
The four-day outage above is the reason Lobsterland runs a channel-health sweep for every hosted Hermes
instance. Every ten minutes it reads gateway_state.json from the pod and, for platforms the
instance actually has configured, restarts the gateway in place — same pod, same disk, back in seconds — when
a platform is fatal, or when it has been retrying for more than 45 minutes without
reconnecting. Forty-five minutes is nine watcher intervals at the 300-second cap: if in-runtime recovery has
not worked by then, it is not going to.
The guard rails are the interesting part. The sweep shares a 30-minute cooldown with crash-loop restarts so two
healers cannot fight over one instance; a per-platform breaker stops after three restarts without a healthy
observation in between, because a restart cannot fix a revoked token, and re-arms after 24 hours so a
multi-day upstream outage still gets one bounded attempt a day; a paused platform is operator
intent and is never healed; and pods that are not Ready are left to pod-level health. When a restart actually
happens you get an email saying so — and saying that messages sent while the channel was down were not
replayed, for the reason in the section above.
That is the whole managed claim, and it is worth being precise about its size: it turns a silent channel into a restarted one within a bounded window, and it does not replay the queue, because nothing can once the process has cold-started. Everything above still describes the runtime you are running. If you would rather not own the monitoring, see managed Hermes runtime hosting. If you are still choosing a runtime, compare the OpenClaw and Hermes runtimes, or read what the Hermes agent runtime is, and how it differs from the Hermes models. Two more Hermes failure modes with the same "documented remedy, undocumented mechanism" shape: Hermes posting too many Slack messages and the Hermes memory character-limit deadlock.
FAQ
Why did my Hermes agent stop responding on Telegram while Slack still works?
Because each messaging platform runs as its own adapter inside one gateway process. When Telegram's long-poll
connection breaks, the Telegram adapter retries ten times with a 5-to-60-second back-off and then declares a
retryable fatal error. The gateway keeps running Slack, cron and its HTTP API regardless, so everything except
Telegram looks healthy. What happens to Telegram next depends on the Hermes version: current builds queue it
for a background reconnect watcher, while builds before v2026.7.30 left it dead until the process
was restarted.
What does "Fatal telegram adapter error (telegram_network_error)" mean?
It is the gateway logging that the Telegram adapter exhausted its in-process reconnect ladder. The adapter's own line just before it reads "Telegram polling could not reconnect after 10 network error retries. Escalating to gateway recovery." The error is marked retryable, which means the gateway is expected to keep trying in the background rather than give up. It is not a token or permission problem, and re-checking the allowlist will not change anything.
Where does Hermes record whether the Telegram channel is connected?
In gateway_state.json under HERMES_HOME (~/.hermes by default,
/opt/data on a hosted instance). The platforms.telegram.state field is one of
connected, retrying, fatal, paused or
disabled, alongside error_code, error_message and
retrying_since. Inside a chat you can also run /platform list, which shows the same
queue and marks paused platforms.
How do I recover Telegram without restarting the whole Hermes gateway?
If gateway_state.json shows the platform as retrying, the reconnect watcher is
already working on it and will keep retrying at a back-off that caps at five minutes; wait one interval before
intervening. If it shows paused, send /platform resume telegram from a connected
platform or the CLI chat; that clears the pause, resets the attempt counter and retries on the next watcher
tick. Only if the state is fatal, or the file has not changed in more than 45 minutes, restart the gateway process.
Will Telegram messages sent during the outage be delivered after recovery?
Only if recovery happens inside the running process. The in-process reconnect ladder and the background
watcher both reconnect with drop_pending_updates=False, so Telegram's server-side queue is
preserved and delivered. A cold gateway start connects with drop_pending_updates=True and
discards that queue. Restarting the process is therefore the one recovery that loses what people sent while
the bot was down, which is why it should be the last resort rather than the first.
How do I check whether the bot is actually polling Telegram?
Call the Bot API's getWebhookInfo method with your bot token and read
pending_update_count, which Telegram documents as the number of updates awaiting delivery. The
call is read-only and does not consume updates. A count that keeps growing while your bot is silent is proof
that nothing is calling getUpdates; a count that drops to zero after recovery is proof that it is.
Sources
-
NousResearch/hermes-agentv2026.8.31(Hermes Agent v0.21.0) source, read from the published image:plugins/platforms/telegram/adapter.py(_handle_polling_network_error,_polling_heartbeat_loop,connect()),gateway/run.py(_handle_adapter_fatal_error_impl,_queue_retryable_fatal_platform,_pause_failed_platform,_RECONNECT_BACKOFF_CAP),gateway/status.py(gateway_state.json). -
NousResearch/hermes-agentreleases —v2026.7.30= v0.19.1 (2026-07-30),v2026.8.31= v0.21.0 (2026-08-31),v2026.9.11= v0.21.2 (2026-09-11). -
NousResearch/hermes-agentissue #58270, "fix(telegram): updater.stop() in reconnect ladder hangs on CLOSE-WAIT sockets — heartbeat loop stuck for hours" — opened 2026-07-04, closed. -
NousResearch/hermes-agentissue #90386, "Gateway self-heal wedges after Telegram polling network outage — reconnect watcher never takes over" — opened 2026-08-20, closed. -
Telegram Bot API reference —
getWebhookInfo/WebhookInfo.pending_update_count("Number of updates awaiting delivery"). - Nous Research, Hermes Agent documentation — "Telegram" (the setup-time troubleshooting list: token, allowed users, group privacy).
-
Lobsterland production incident 2026-08-10 → 2026-08-14 on Hermes
v2026.7.20, and the resulting channel-health sweep and its thresholds (10-minute cadence, 45-minute retrying threshold, 30-minute cooldown, three-strike breaker re-armed after 24 hours).