Managing the OOM Killer among us

Like me you have probably had your share off issues when something runs amok and OOM (The default out of memory killer) slaughters something that was behaving well and is, as it turns out, critical to your operation.
In my case MySQL is the primary target for OOM because it consumes the most memory on any given server. But occasionally someones PHP code will start to get out of hand on those same servers. And I want OOM to kill said out of hand code and thus protect my primary application! But oom does not… Instead it kills my one big app to protect the small but hungry programs. And this is not a gentle death either, it is a full blown Kill -9 to the face! So how do I make my app less of a target?

The method changes over the course of OS’s and versions but the two ways showcased here are mostly compatible and work on a wide variety of systems.
The Adjustment, and the Score.

The OOM Adjust

Adjustments are done live, in proc, per pid. You can see why this method has started to loos favor, but oom still respects it so go nuts.

echo -15 > /proc/1234/oom_adj

Valid values are from -15 to +15 where higher values incur more agro from oom killer and thus make you a more likely target. If you go low enough, say -17, you are supposed to be exempt from the relentless blade of our favorite executioner.
However you can see how this would be extremely cumbersome, manually adjusting our frightful pids each restart! As such I quickly added the following to our init script

getpid() {
#Give up to 10 seconds for a pidfile to be created
    for i in {1..20}; do
        if [[ -e $pidfile ]]; then
            pid=$(cat $pidfile 2> /dev/null )
            break
        fi
        sleep .5
    done
}

start() { 
...
    getpid
    if [[ -n $pid ]]; then
        echo "-1" > /proc/${pid}/oom_adj
    fi    
...
}

NOTE: How you chose to set your pid variable is up to you. For MySQL we set a pid file in the conf so all I need to do is read it. Others can be found in the default /var/run/ directory, particularly if you are using a init.d script with __pids_var_run(). However for others you may be forced to run a ps | grep to retrieve your pid.
Also this only sets the pid on start. If your app trips and have an agile processes restart you then you lose your buff and can once again draw OOM aggro.

 

OOM Score

Another method is the oom score conf file! This one also provides finer control with scores ranging from -1000-ish to +1000-ish**. Also this method is seriously easy, just lay down a conf file in /etc/init/${SERVICE}.conf (Yes ${SERVICE} is a variable to be replaced by the name of your service).
The contents of this file runs far and wide and I recommend looking into it, however, the the purpose of this discussion you will want a line that simply says something like oom score -1000. Again the lower the score the better protected you are agents OOM and the higher the score the more likely our oom friend will hunt you down.
While I recommend using puppet (or chef or other configuration management system) to manage this file, or at least in absence of that lay it down with the install rpm, I will provide a init script snipit you can use.

start() {
...
    if [[ ! -e /etc/init/${SERVICE}.conf ]]; then
        echo "oom score -50" >> /etc/init/${SERVICE}.conf
    else
        #Only set it if not other wise set
        grep oom /etc/init/${SERVICE}.conf &> /dev/null
        if [[ $? -eq 1 ]]; then
            echo "oom score -50" >> /etc/init/${SERVICE}.conf
        fi
    fi
...
}

**The score is based on either the old badness() function or a strait 10x percentage of available memory. If you are running kernel 2.6+ then you are no longer using the badness heuristic making calculations easy.

 

Panic at the Disco

I would like to point out that some people prefer the full reboot. If this is you you can choose to panic on oom by the flowing means

sysctl vm.panic_on_oom=1
sysctl kernel.panic=X
echo "vm.panic_on_oom=1" >> /etc/sysctl.conf
echo "kernel.panic=X" >> /etc/sysctl.conf

 

Kill the Killer

You can also disable the OOM Killer entirely. And while I do not recommend this course of action at all, I will show you how to do it.

sysctl vm.overcommit_memory=2
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf

 

I do believe it’s working!

If you want some immediate gratification you can see the settings in place with a simple

tail /proc/${pid}/oom_*

But you can also allocate a tun of memory and see what gets killed, assuming you are not testing on production servers 😉
Personally I would just  launch something and abuse it to allocate memory because I am cautious, but lazy. Or I would grab a cheep bit of C that does little more then stuff ram or run stress… But this page gives some decent ideas as well
http://unix.stackexchange.com/questions/99334/how-to-fill-90-of-the-free-memory

 

Hope this helps 🙂

 

This blog post is dedicated to Collectd and the person on our team who wrote the MySQL collector for it in PHP. As well as the person who complained enough about the oom killings one night that I stopped my work making other things better to improve our situation on this front with some quick puppet work 🙂

And no, it does not happen enough to warrant tracking down the memory problems in PHP. At least not for me. I hate PHP 😉

Leave a Reply

Your email address will not be published. Required fields are marked *