To begin with, I decided to look at the logs from just before the previous boot, so that I could tell whether this was really a reboot coming from the OS or a problem around the hardware power supply.
On AlmaLinux (RHEL based), running journalctl -b -1 shows the logs of the one previous boot all together, so I used that command as a starting point and went over the kernel messages and the systemd logs.
watchdog.service was not there, but dmesg had an NMI watchdog
When I hear about frequent reboots, the first thing that comes to mind is a watchdog.
I suspected that a software watchdog daemon was misdetecting something and triggering the reboots, so I first checked whether the systemd service existed.
sudo systemctl status watchdogWhat came back, however, was the following message.
Unit watchdog.service could not be found.From this result I learned that, at least, the userland daemon called watchdog.service did not exist.
Next, to check whether any watchdog related mechanism was running at the kernel level, I searched dmesg.
sudo dmesg | grep watchdogThat turned up the following line in the boot initialization log.
NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.Only here did I notice that on this machine it is not a software watchdog but the NMI watchdog inside the kernel (the hard lockup detection mechanism) that is enabled.
I still did not know whether the NMI watchdog was the direct trigger of the reboots, but I judged that it was well worth including as a candidate cause in the investigation.
What the NMI watchdog does, and why I decided to turn it off
The Linux kernel provides a lockup watchdog for detecting a state in which the CPU or the system has frozen (locked up).
Of these, the mechanism that uses NMI to detect a hard lockup of the CPU is what is called the NMI watchdog, and it is controlled with the nmi_watchdog kernel parameter and /proc/sys/kernel/nmi_watchdog.
The NMI watchdog is originally a useful feature that detects abnormal states in which the CPU stops responding completely and provides the trigger for a panic or for log output.
However, depending on how well it gets along with particular hardware or virtualization environments, or on the behaviour around the CPU performance counters, it can produce false detections and unintended behaviour, and as a result it can become the cause of frequent reboots or kernel panics.
In this environment neither a software watchdog nor any other obvious cause of the reboots was to be found, so I judged that it was worth disabling the NMI watchdog once and seeing how things went.
Since this means weakening the hang detection mechanism, I made it a policy to disable it in stages only, and to proceed while observing the behaviour.
Practical steps to disable nmi_watchdog safely
First disable it temporarily and check the behaviour
If I were to rewrite the kernel parameter and reboot right away, rolling back would be troublesome should there be any side effects.
So I began with the procedure of using sysctl to disable the NMI watchdog temporarily on the currently running kernel.
# disable temporarily
sudo sysctl kernel.nmi_watchdog=0
# check the state
cat /proc/sys/kernel/nmi_watchdog
# → if it has become 0, it is disabledThis setting goes back to the original value when the machine reboots, which makes it just right for changing the behaviour without taking the OS down and watching for a while.
In fact, after running in this state for a certain amount of time, the sort of mysterious reboots that had been happening no longer occurred, and I saw no particular ill effects on other features or on performance either.
From this observation I judged that, at least in this environment, disabling the NMI watchdog causes no problems in actual operation and rather improves stability, and I switched to a policy of turning it off permanently.
Disable it permanently with a GRUB kernel parameter
As a way to make this permanent, the most reliable approach, very much in the AlmaLinux (RHEL based) manner, is to add nmi_watchdog=0 to the GRUB kernel parameters.
Concretely, edit /etc/default/grub and add the parameter to GRUB_CMDLINE_LINUX.
GRUB_CMDLINE_LINUX="rhgb quiet nmi_watchdog=0"After editing, regenerate the GRUB configuration. The output location differs between BIOS and UEFI, so you need to specify the path that matches your environment.
sudo grub2-mkconfig -o /boot/grub2/grub.cfg # example for BIOS
# on a UEFI environment, match it to something like /boot/efi/EFI/almalinux/grub.cfgAfter rebooting, confirm with the following command that the NMI watchdog is disabled.
cat /proc/sys/kernel/nmi_watchdog
# → if it is 0, the kernel parameter is in effectIf necessary, you can keep the same setting on the sysctl side as well by creating a file such as /etc/sysctl.d/disable_nmi_watchdog.conf and writing kernel.nmi_watchdog = 0 in it.
Note, however, that when it is already set to 0 through the kernel parameter, sysctl only has a reinforcing meaning.
Summary: the reboots stopped, but monitoring needs a separate design
After running the machine for a while with the NMI watchdog disabled, the reboots of unknown cause that had been troubling me no longer occurred.
NMI related messages stopped appearing in the logs and the behaviour on the application side is stable too, so I conclude that in this environment the NMI watchdog was one of the sources of instability.
On the other hand, the NMI watchdog is fundamentally a safety net that detects fatal hangs, and it is not a feature you can simply switch off and be done with.
It is preferable to combine monitoring designed at another layer: on a bare metal server, use alive monitoring through IPMI or an external monitoring system as well; in a virtual environment, leave it to the monitoring mechanism on the hypervisor side.
What I learned from this case is that when you are troubled by mysterious reboots, you should suspect not only watchdog.service but also the kernel level watchdogs, including the NMI watchdog, with dmesg | grep watchdog.
I hope this is useful to anyone else who is facing frequent reboots on AlmaLinux or another RHEL based distribution.