Description

on PTX10001-36MR with coherent optics (QSFP56-DD-400G-ZR, QSFP56-DD-400G-ZR-M, QSFP56-DD-400G-ZR-M-HP) ,PICD memory is increasing due to memory leak. PICD process get killed with the out of memory log message and all the interfaces  get flapped

 

Symptoms

issue is related to memory leak on picd due to many i2c writes that are required to fetch various performance monitoring parameters about coherent optics. during the memory leak, you will see the memory of picd is increasing with a rate depending on the number of coherent optics on the chassis. memory consumption can be tracked from RE shell using the below command

user@router> start shell      

[vrf:none] labroot@e08-45:~$ systemctl status picd

* picd.service - "picd"

  Loaded: loaded (/etc/systemd/system/picd.service; static; vendor preset: disabled)

  Active: active (running) since Wed 2024-06-12 14:31:39 PDT; 5 days ago

 Main PID: 10421 (picd)

  Tasks: 9

  Memory: 133.9M (limit: 1.0G) <<<<<<<<<<memory increasing

  CGroup: /system.slice/picd.service

      > 10421 /usr/sbin/picd --app-name picd -I object_select --shared-objects-mode 3 -p /var/pfe

 

once picd process will consumed its maximum memory (1 GB) plus the swapped memory , it will crash with the below log message and all the interfaces will flapped and cause service impact

Jun 18 11:02:09.233 <route-name> kernel: Memory cgroup out of memory: Killed process 9598 (picd) total-vm:10586100kB, anon-rss:1019184kB, file-rss:127420kB, shmem-rss:603316kB

Solution

customer need to have the fix of PR1818352:

https://prsearch.juniper.net/problemreport/PR1865403

The  memory leak on picd has been fixed via above PR. 
customer can increase the maximum memory of picd using the below command from RE shell but this just to delay the crash of picd

systemctl set-property --runtime picd MemoryLimit=2G   <<<<<<<increase the memory limit from 1G to 2G

 

Modification History

2024-07-12 : Article Created