The n8n stuck-channel hunt

n8n's external task-runner mode had a failure nobody could see. The broker and the runner kept a healthy WebSocket (/healthz returned 200, ping/pong passed) while the application-level message channel was dead. Every task request timed out at 60 seconds, forever. The issue (#29372) documented stuck windows of ten to thirty-seven hours. Every monitor said the system was fine. It wasn't.

The broker already knew. It threw a TaskRunnerAcceptTimeoutError every time a runner failed to acknowledge a task, then logged it and dropped it on the floor. No per-runner failure tracking, no line connecting "this runner stopped answering" to the recovery machinery that already existed. The signal was there. Nothing listened to the pattern.

The right fix wasn't a new monitoring system; it was teaching the broker to hear what it was already saying. Same shape the maintainers had accepted before, in #25959: surface real health on existing signals, don't bolt on a parallel watcher.

Count consecutive accept-timeouts per runner. After three in a row (about six seconds at the default two-second accept timeout), emit a runner:unresponsive event with the offending runner's id. The WebSocket server disconnects that one runner through the existing removeConnection() path, and the launcher restarts it the same way it does after any connection loss. Attribution is per-runner, so in a multi-runner deployment only the stuck one drops; healthy peers are untouched. Six new unit tests cover the threshold, the per-runner isolation, and clean unsubscribe on stop.

Detection of that failure mode drops from hours to about six seconds. Submitted to n8n as PR #29516. The diagnosis is grounded in the code path line by line; the manual repro needs a sustained cloud deployment, which is named as a boundary in the PR rather than glossed over. The scope is honest about what it doesn't catch: a runner that goes quiet without ever accepting an offer never trips the counter, flagged as a follow-up, not hidden.

Three months on, a maintainer closed it in favour of PR #35456. A separate rework had changed the same acceptOffer path in the meantime, so the original patch no longer applied cleanly. #35456 re-implements the consecutive-timeout disconnect on current master and carries it further: reject and defer replies reset the counter too, the runner is reported exactly once at the threshold, and the disconnect gets its own runner-unresponsive reason for observability. It merged on 4 August, credits the diagnosis by name, and shipped to the 2.33 and 2.34 release lines.