l2alm thread utilization stays high in QFX5k devices.
Show chassis fpc showing high cpu utilization.
root@QFX> show chassis fpc
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer
0 Online 23 44 0 44 45 45 16384 30 49
Looking further, L2ALM Manager thread is utlilizing high % of fpc cpu (58% here).
root@QFX> start shell
% vty fpc0
FPC0(QFX vty)#
FPC0(QFX vty)# show threads cpu
CPU hog check enabled
Max run time 2500
PID PR State Name Stack Use Time (Last/Max/Total) cpu
--- -- ------- --------------------- --------- ---------------------
51 M ready L2ALM Manager 6936/73824 1/3/48903506 ms 58%
89 M asleep bcmCNTR.0 4128/57440 0/9/5741010 ms 6%
7 M asleep SFP 2964/73824 0/76/3026105 ms 3%
47 M asleep PFE Manager 35308/73824 0/81/1515014 ms 1%
82 M asleep BRCM SALM Pending Cleanup 4368/73824 0/4/1159756 ms 1%
FPC0(QFX vty)# show thread 51
51 M ready L2ALM Manager 6936/73824 0/3/48927739 ms 58%
Wakeups:
Type ID Enabled Pending Context
Queue 03 Yes No 0xb1aedba8
Queue 08 Yes No 0xb1caf5c0
Queue 15 Yes No 0xb1caf450
Semaphore 00 Yes No 0xaf82a6d0
Semaphore 01 Yes No 0xaf82a728
Semaphore 02 Yes No 0xb1caf0e0
Semaphore 04 Yes No 0xb1adba90
Semaphore 05 Yes No 0xb1caf510
Semaphore 06 Yes No 0xb1caf568
Semaphore 07 Yes No 0xb1caf190
Semaphore 09 Yes No 0xaf82a5c8
Semaphore 10 Yes No 0xb1caf138
Semaphore 11 Yes Yes 0xb1caf240
Semaphore 13 Yes No 0xb1caf1e8
Semaphore 14 Yes No 0xb1caf298
Semaphore 17 Yes No 0xb1caf2f0
Semaphore 18 Yes No 0xb1caf348
Timer 00 Yes No 0xb1aede30
Socket 00 Yes No 0xb1caf7d0
Mutex 00 Yes No 0xb1883f78
Stack:
sp = 0xb1aed878, pc = 0x8083f77
sp = 0xb1aed898, pc = 0x8dbd53d
sp = 0xb1aedb58, pc = 0x8084280
FPC0(QFX vty)# show sched
Total uptime 0+23:25:47, (84347954 ms), 219962527 thread dispatches
CPU load is 100% (5 second), 0% (1 minute)
Total network interrupt time 0 (usec)
CPU Name Time(ms)
0% Idle 41586
99% Threads 84306368
0% ISR 0
On enabling l2alm debugs, below messages were seen which seems to be the cause for the l2alm manager thread to go high
FPC0(QFX vty)# show l2 manager debug debug information ------------------- level 3 flag 0x0 FPC0(QFX vty)# debug l2 manager level debug FPC0(QFX vty)# debug l2 manager flag all FPC0(QFX vty)# show l2 manager debug debug information ------------------- level 7 flag 0xffffffff BD_ID: 0 [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_mac_polling_handler:2102 MAC Polling entry does not exist [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_mac_polling_timer_handler_entry:2030 Polling timer entry: bd:ba:c0:17:ff:ba, bd:56136, num_exp:0, status:1000 [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:bd:ba:c0:17:ff:ba, bdid:56136) [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_mac_polling_handler:2102 MAC Polling entry does not exist [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_mac_polling_timer_handler_entry:2030 Polling timer entry: bd:ba:c0:17:ff:ba, bd:56136, num_exp:0, status:1000 [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:bd:ba:c0:17:ff:ba, bdid:56136) [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_mac_polling_handler:2102 MAC Polling entry does not exist [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_mac_polling_timer_handler_entry:2030 Polling timer entry: bd:ba:c0:17:ff:ba, bd:56136, num_exp:0, status:1000 [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:bd:ba:c0:17:ff:ba, bdid:56136) [Wed Jul 12 20:09:51.119 LOG: Debug] l2alm_mac_polling_handler:2102 MAC Polling entry does not exist [Wed Jul 12 20:11:18.176 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:bd:ba:c0:17:ff:ba, bdid:56136) [Wed Jul 12 20:11:18.183 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:bd:ba:c0:17:ff:ba, bdid:56136) [Wed Jul 12 20:11:18.190 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:bd:ba:c0:17:ff:ba, bdid:56136)
On checking the same on Jul 13, we see same messages but with different set of mac and bdid
[Thu Jul 13 17:30:14.849 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:ce:b1:c0:17:ff:ba, bdid:39320) [Thu Jul 13 17:30:14.855 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:ce:b1:c0:17:ff:ba, bdid:39320) [Thu Jul 13 17:30:14.860 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:ce:b1:c0:17:ff:ba, bdid:39320) [Thu Jul 13 17:30:14.869 LOG: Debug] l2alm_hw_query_src_mac:1364 Invalid bd (mac:ce:b1:c0:17:ff:ba, bdid:39320)0
The above shows that there is mac-polling messages are causing l2alm thread to go high. In releases prior to 21.4R3-S3 and 22.2R3-Sx, the mac-polling infrastructure is used. It has a bug that causes mac-polling resulting in the issue.
Releases after the mentioned releases uses a different infrastructure.