Security Update Reboots on Managed Linux Machines

Certain security updates require machines to be rebooted. On the managed departmental Linux machines, there is an automated process running that watches for such a situation. When a security update is applied that requires a reboot, it starts a process of warnings and watches for an appropriate moment to reboot the machine.

Process overview

Once there are no users logged in and no user processes running in the background, the machine will be rebooted within about 5 minutes. Since 60-80% of machines are not in use at night, most machines reboot within 1 day.

If the reboot does not occur the first night/weekend, warning emails will be sent to anyone already logged in asking them to log out so the reboot can occur. These will be repeated periodically. Users will also be shown a warning when they try to log in.

If there are still users logged in after 4-6 days, a final warning email will be sent to anyone who is still logged in or has background processes running, and a forced reboot will be scheduled to occur 24 hours later.

Delaying reboots

It is important that security updates are applied as soon as possible, which is why the above process ultimately results in a forced reboot if no user action is taken. To allow the security updates to go unapplied for longer would, in general, pose an unreasonable risk to the departmental systems.

However, in specific cases when an important process is running and more time is needed, you can email @email and we will look at the specific risk and delay the forced reboot if it is safe to do so. We can only do this in limited circumstances, as we need to comply with university policy that requires security updates to occur in a timely fashion.

Planning for reboots

Wherever possible, we recommend writing code that can checkpoint itself periodically (write progress to disk) and resume from such a checkpoint, to minimise the impact of both planned reboots and unexpected issues such as power or network outages.

The compute servers have additional live kernel patching software, which significantly reduces the number of security updates that require a reboot compared to the desktop machines - so they may be a safer choice if you are unable to reduce the time your code needs to run.

Why did the machine not reboot when I logged out?

There are two common situations where the machine does not reboot even though you log out:

1. Even though you have logged out there are still processes running on the machine owned by you
2. Another user is remotely logged into the machine or has left processes running on the machine

In both these cases the reboot trigger detects the presence of user processes and hence will not reboot the machine.

Case 1 can occur if a process has gone rogue and thus does not close/die properly when you log out. See below for how to kill off rogue processes.

Case 2 is most likely to occur on the departmental compute machines that are used by lots of different people. If the machine is being used by lots of people, it is unlikely they will all log out and stay logged out, and hence the most likely outcome is the machine will go through a forced reboot.

Case 2 may also occur on an ordinary desktop if it is being remotely used by someone else (which is quite rare but does happen). In this case, with only a couple of users, it is quite likely the remote user will log out, after which the local user can log out and the security reboot will occur. Contact IT Support if you need help identifying/contacting the remote user.

Killing off rogue processes

You can look at the list of processes running on a machine with the command:

ps aux

from which you can identify a process that was started prior to your most recent login and then kill it with a command of the form:

kill -9 PID

where you replace PID with the process ID number of the relevant process, as listed in the output of the previous command.

Killing off all processes / remotely logging out

Sometimes you may simply want to kill off all your process on a machine, which also has the effect of logging you out.

To kill off all your processes on a machine, run a command of the form:

pkill -9 -u USERNAME

where you replace USERNAME with your username. Please note that this command will kill all your applications on the machine, no matter whether those have unsaved data. All unsaved data will be lost.

To log yourself out remotely, you can use SSH to connect to the machine and run those commands.