Description

This article explains the meaning of the following syslog message:
 
May 30 09:56:27 cRR1 kernel: [525378.464556] rpd invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
May 30 09:56:27 cRR1 kernel: [525378.464608]  oom_kill_process+0x220/0x440
May 30 09:56:27 cRR1 kernel: [525378.464660] Task in /docker/22833268f683986378fd0547f3b6781d5adc42f0814a0aa656f379fdb25c04d3 killed as a result of limit of /docker/22833268f683986378fd0547f3b6781d5adc42f0814a0aa656f379fdb25c04d3
May 30 09:56:27 cRR1 kernel: [525378.465349] Memory cgroup out of memory: Kill process 23549 (rpd) score 955 or sacrifice child
May 30 09:56:27 cRR1 kernel: [525378.469235] Killed process 23549 (rpd) total-vm:2941752kB, anon-rss:2026836kB, file-rss:34404kB, shmem-rss:0kB

Symptoms

  1. The RPD would utilize more than the memory+swap assigned to it while bringing up the container. For example, in the following case, the container was spawned with memory+swap limit of 2G

    docker run --rm -detach --name cRR1 -h cRR1 --privileged -v cRR1-config:/config -v cRR1-Log:/var/log  --memory="2g" --memory-swap="2g" -it crpd:20.3X75-D5.5 
     
  2. As seen in the docker stats output below, it shows the container is utilizing all the memory that was allocated to it - 2G

    root@cRR1:/home/regress# ps aux | grep "/usr/sbin/rpd -N"
    root     23549 53.2  1.5 1397400 516616 ?      Rl   02:54   0:12 /usr/sbin/rpd -N     >> This is the RPD PID for cRR1            

    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
    68baf744ac24        cRR1                95.53%              2GiB / 2GiB         100.00%             101MB / 55.3MB      8.28GB / 9.38GB     53
     
  3. The task memory output shows that container utilized maximum memory of around 2G on '22/05/30 09:56:20' - that is exactly when it ran out of memory and RPD crashed 

    root@cRR1> show task memory    
    Memory                 Size (kB)  Percentage  When
      Currently In Use:        37249          0%  now
      Maximum Ever Used:     2055076          8%  22/05/30 09:56:20
      Available:            32346653        100%  now

Solution

  1. “Memory cgroup out of memory” is a message coming from the kernel when it kills something because of a memory cgroup restriction, such as those we place on containers while spinning them. 
  2. One major thing that could lead to this event is if insufficient memory is allocated to container which is insufficient to handle the route scale.
  3. Another factor which could lead to memory contention is a huge route churn which pushes the memory to the edge of extinction.
  4.  As a reference for RIB/FIB scale vs minimum memory required to support it, refer the documentation on cRPD Resource Requirements.

Modification History

2022-08-26: First publication.