Avoid Linux locking up in low memory situations using earlyoom

Comment on Mastodon

Introduction

Within operating system kernels, at least for Linux and the BSDs, there is a mechanism called "out of memory killer" which is triggered when the system is running out of memory and some room must be made to make the system responsive again.

However, in practice this OOM mechanism doesn't work well. If the system is running out of memory, it will become totally unresponsive, and sometimes the OOM killer will help, but it may take like 30 minutes, but sometimes it may be stuck forever.

Today, I stumbled upon a nice project called "earlyoom", which is an OOM manager working in the user land instead of inside the kernel, which gives it a lot more flexibility about its actions and the consequences.

earlyoom GitHub project page

How it works

earlyoom is simple in that it's a daemon running as root, using nearly no memory, that will regularly poll for remaining swap memory and RAM memory, if the current level are below the threshold of both, actions will be taken.

What's cool is you can tell it to prefer some processes to terminate first, and some processes to avoid as much as possible. For some people, it may be preferable to terminate a web browser first and instant messaging than their development software.

I use it with the following parameters:

earlyoom -m 2 -s 2 -r 3600 -g --avoid '^(X|plasma.*|konsole|kwin)


 --prefer '^(electron|libreoffice|gimp)

The command line above means that if my system has less than 2% of its RAM and less than 2% of its swap available, earlyoom will try to terminate existing program whose binary matches electron/libreoffice/gimp etc.... and avoid programs named X/Plasma.*/konsole/kwin.

For configuring it properly as a service, explanations can be found in the project README file.

NixOS setup

On NixOS, there is a module for earlyoom, to configure it like in the example above:

{
  services.earlyoom = {
      enable = true;
      freeSwapThreshold = 2;
      freeMemThreshold = 2;
      extraArgs = [
          "-g" "--avoid '^(X|plasma.*|konsole|kwin)


"
          "--prefer '^(electron|libreoffice|gimp)


"
      ];
  };
}

Conclusion

This program is a pleasant surprise to me, I often run out of memory on my laptop because I'm running some software requiring a lot of memory for good reasons, and while the laptop has barely enough memory to run them, I should have most of the other software close to make it fit in. However, when I forget to close them, the system would just lock up for a while, which most often require a hard reboot. Being able to avoid this situation is a big plus for me. Of course, adding some swap space would help, but I prefer to avoid adding more swap as it's terribly inefficient and only postpone the problem.