A server that has been up for months is running on things nobody wrote down: a service started but never enabled, a container with no restart policy, a script someone launched by hand. All of it works until the reboot, then it is quietly gone. We built a read-only check that lists those things before you reboot. On our agent box it found one. The first version also found two that were not real, which is its own lesson.
Why we looked
Security updates for the Linux kernel install automatically but only take effect after a reboot. On a box that runs an agent fleet, a reboot is the one routine operation that feels risky, so it keeps getting put off, and the fixes pile up on disk unused. Ubuntu is moving to weekly kernel patches, so that pile is about to grow faster. The honest reason nobody wanted to reboot was not the reboot. It was not knowing what would fail to come back, and the only way to find out was to try.
What we built
reboot_ready.py asks one question of everything running on the box: what
starts this again after a restart? It answers from the machine, not from anyone's memory.
- Lifelines first. Remote login, the private network, cron and Docker. If one of these does not come back, you cannot fix anything else without a console.
- Services. Every running system service is either enabled at boot or it is not. "Running" and "will run after a reboot" are separate settings, and the difference is invisible until it matters.
- Containers. Every running container either has a restart policy or it stays down.
- Orphans. Anything long-lived that no service or container owns: a process started in a terminal, or one launched by a cron job. The check reads which process group each one lives in, then looks for a cron line that names its script or folder, either at boot or as a periodic "start it if it is not running" keepalive. No match, and it will not come back.
It changes nothing and restarts nothing. It reads, reports, and exits non-zero if anything would be lost.
What it found
32 running services, 18 containers, two long-lived processes launched by cron. One real problem: our voice agent was running but had never been enabled. It had been started by hand at some point, it worked, and after the next reboot it would simply not have been there. Nobody would have noticed until someone tried to call it.
The two cron-launched processes were fine. One is started at boot, the other is restarted every five minutes by a keepalive. By eye, both looked like orphans. The check was right and a quick look would have been wrong.
And the first version cried wolf twice. It said remote login would not come back, because on current Ubuntu the SSH service shows as disabled: it is started on demand by a socket instead. It said the same about the serial console, which systemd recreates at every boot. Both were false alarms that would have sent someone chasing a non-problem at exactly the moment they wanted to be calm. Both are now recognised, and there are tests for how each kind of process is classified.
What's still off
It checks that things will be started, not that they will work. A service that starts and then falls over on a missing disk still passes. A process started inside another service's process group is counted as part of that service. And the cron match is a heuristic: if a keepalive starts something through an indirect wrapper, the check says it will not come back, which is the safe way to be wrong. The reboot itself stays a human job, with a console open and a runbook that runs this check before and after.
What's now in the stack
reboot_ready.py: a read-only, standard-library-only readiness check for systemd boxes, with a human report or JSON, and unit tests for the classifier.- A before-and-after reboot runbook for our own box, which stays private because it is full of our own paths.
- On GitHub. Run it on anything you have been scared to restart.