Backup routing engine utilizing above average memory than that of master routing engine
As seen in the below output RE0 [master] memory utilization is lower than that of RE1 [backup] and as per the Customer was more than average usage:
Routing Engine status:
Slot 0:
Current state Master
Election priority Master (default)
Temperature 25 degrees C / 77 degrees F
CPU temperature 25 degrees C / 77 degrees F
DRAM 110350 MB (131072 MB installed)
Memory utilization 15 percent
Slot 1:
Current state Backup
Election priority Backup (default)
Temperature 29 degrees C / 84 degrees F
CPU temperature 29 degrees C / 84 degrees F
DRAM 62243 MB (131072 MB installed)
Memory utilization 52 percent
Verify if any process is hogging memory and failing to release by collecting multiple iterations of the below preliminary outputs:
show chassis routing-engine
show system processes extensive node re<node id>
show system memory node re<node id>
show system virtual-memory node re<node id>
show system memory statistics node re<node id>
RE shell outputs:
cat /proc/meminfo
vmstat -s
For instance, in the scenarios shared in the symptom section:
Memory segregation does not indicate any abnormal utilization and inactive memory percentage matches that of memory utilization seen in the above output:
root@router_re0> show system memory node re1
System memory usage distribution:
Total memory: 131405608 Kbytes (100%)
Active memory: 22316576 Kbytes (16%)
Inactive memory: 69551072 Kbytes (52%)
Free memory: 20410248 Kbytes (15%)
root@router_re0> show system virtual-memory node re1
re1:
--------------------------------------------------------------------------
MemTotal: 131405608 kB
MemFree: 20403488 kB
MemAvailable: 63731364 kB
Buffers: 1015812 kB
Cached: 76059732 kB
SwapCached: 0 kB
Active: 22322060 kB
Inactive: 69551252 kB
<snip>
root@router_re0> show system processes extensive node re1 ---> no process is currently hogging memory
-------------------------------
node: re1
top - 22:53:02 up 630 days, 21:17, 1 user, load average: 1.35, 1.38, 1.39
Threads: 2867 total, 2 running, 2865 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.8 us, 1.2 sy, 0.0 ni, 92.6 id, 0.0 wa, 0.2 hi, 0.2 si, 0.0 st
KiB Mem : 13140560+total, 20291672 free, 30347468 used, 80766464 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 63680440 avail Mem
PID USER TID PRI NI SIZE RSS S TIME %MEM %CPU COMMAND
21125 root 26212 19 0 146360 133840 R 173-14:58:01 0.1 99.9 icmpd{icmpd}
14751 root 14751 19 0 318212 388688 S 1-14:01:21 0.3 6.7 hwdre{hwdre}
17200 root 1354 19 0 64804 47980 S 17-17:25:42 0.0 3.3 marvd{marvd}
[vrf:none] JUNOS-RW@router_re1:~$ systemctl status icmpd ---> Use this command to get more info on the process which is hogging memory/cpu
* icmpd.service - "ICMP daemon on RE"
Loaded: loaded (/etc/systemd/system/icmpd.service; static; vendor preset: enabled)
Active: active (running) since Sun 2023-03-12 01:39:13 -06; 1 years 8 months ago ---> timeframe
Main PID: 21125 (icmpd)
Memory: 26.0M (limit: 6.0G) ---> memory utilized by the process
CGroup: /system.slice/icmpd.service
`-21125 /usr/bin/icmpd --app-name icmpd -I object_select --shared-objects-mode 3
[vrf:none] JUNOS-RW@router_re1:~$ cat /proc/meminfo
MemFree: 20341908 kB
MemAvailable: 63688864 kB
Cached: 76079068 kB
Active: 22361424 kB
Inactive: 69570524 kB
Active(anon): 17026300 kB
Inactive(anon): 33934952 kB
Active(file): 5335124 kB
Inactive(file): 35635572 kB
Unevictable: 2204 kB
Mlocked: 2204 kB
SwapTotal: 4194300 kB
SwapFree: 4194300 kB
Dirty: 7496 kB
Writeback: 0 kB
AnonPages: 14839628 kB
Mapped: 36741112 kB
Shmem: 36122348 kB
[vrf:none] JUNOS-RW@router_re1:~$ vmstat -s
131405608 K total memory
30341924 K used memory
22361480 K active memory
69575728 K inactive memory
20335664 K free memory
1015812 K buffer memory
79712208 K swap cache
4194300 K total swap
0 K used swap
4194300 K free swap
This specific user scenario can be considered as an expected behaviour and it will be non-impacting to device performance. Inactive memory is memory pages no longer used/referenced by programs/processes running on the system and hence, the system can reallocate it to other programs when requested/needed.
Generally, the kernel delays the migrating of memory out of the Inactive queue into the Cache or Free list unless there is a memory shortage. Occasionally, this could lead to discrepancies between the values reported by the routing protocol process and the kernel, even after the routing protocol process has freed a large amount of memory. However, this will not cause any impact on the device performance as this memory can still be reallocated/utilised by the system as and when needed and this applies to both scenarios where Junos & EVO are used [however, the commands for the outputs listed above are applicable for EVO]