on PTX10001-36MR with coherent optics (QSFP56-DD-400G-ZR, QSFP56-DD-400G-ZR-M, QSFP56-DD-400G-ZR-M-HP) ,PICD memory is increasing due to memory leak. PICD process get killed with the out of memory log message and all the interfaces get flapped
issue is related to memory leak on picd due to many i2c writes that are required to fetch various performance monitoring parameters about coherent optics. during the memory leak, you will see the memory of picd is increasing with a rate depending on the number of coherent optics on the chassis. memory consumption can be tracked from RE shell using the below command
user@router> start shell [vrf:none] labroot@e08-45:~$ systemctl status picd * picd.service - "picd" Loaded: loaded (/etc/systemd/system/picd.service; static; vendor preset: disabled) Active: active (running) since Wed 2024-06-12 14:31:39 PDT; 5 days ago Main PID: 10421 (picd) Tasks: 9 Memory: 133.9M (limit: 1.0G) <<<<<<<<<<memory increasing CGroup: /system.slice/picd.service > 10421 /usr/sbin/picd --app-name picd -I object_select --shared-objects-mode 3 -p /var/pfe
once picd process will consumed its maximum memory (1 GB) plus the swapped memory , it will crash with the below log message and all the interfaces will flapped and cause service impact
Jun 18 11:02:09.233 <route-name> kernel: Memory cgroup out of memory: Killed process 9598 (picd) total-vm:10586100kB, anon-rss:1019184kB, file-rss:127420kB, shmem-rss:603316kB
customer need to have the fix of PR1818352:
https://prsearch.juniper.net/problemreport/PR1865403
The memory leak on picd has been fixed via above PR. customer can increase the maximum memory of picd using the below command from RE shell but this just to delay the crash of picd
systemctl set-property --runtime picd MemoryLimit=2G <<<<<<<increase the memory limit from 1G to 2G