Connection is lost between two EX4400 switches configured with LACP on the link.
Below syslog message is logged when a link, configured with LACP, flaps:
labroot@switch> show log messages | match lacp
Feb 6 22:49:18 block134-EAS3 lacpd[18378]: LACP_INTF_MUX_STATE_CHANGED: ae2: xe-0/2/3: Lacp state changed from COLLECTING_DISTRIBUTING to ATTACHED, actor port state : |-|-|-|-|IN_SYNC|AGG|SHORT|ACT|, partner port state : |-|DEF|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|
If LACP flaps, but this is not associated with a physical interface flap then, most likely, one of the threads is hogging the CPU for multiple seconds and we need to check which one.
From shell, check the thread that hogs the CPU:
labroot@switch> start shell
% cprod -A fpc0 -c "show halp-pkt pkt-stats"
PKT TX STATS:
tx_req : 22087311
tx_success : 22520858
tx_arr_req : 18173167
tx_arr_success : 18173167
tx_with_tx_pbmp : 22087310
tx_with_no_tx_pbmp : 433548
PKT RX STATS:
rx_req : 27821853
rx_success : 26866812
rx_buf_alloc : 27821856
rx_vlan_ifl_map_get_failed : 85596
rx_pkt_ipv4 : 25972641
rx_pkt_vstp : 342387
rx_pkt_vstp_inserted : 342387
rx_pkt_stp_inserted : 91160
rx_pkt_vxlan_not_found_encap_pkt : 29823424
RX PER QUEUE STATS:
rx queue 8 : rxq_req 814945 rxq_success 814945
rx queue 9 : rxq_req 2821 rxq_success 2821
rx queue 14 : rxq_req 918 rxq_success 918
rx queue 19 : rxq_req 1272498 rxq_success 1272498
rx queue 23 : rxq_req 23502471 rxq_success 23502471
rx queue 34 : rxq_req 527 rxq_success 527
rx queue 38 : rxq_req 3017 rxq_success 3017
rx queue 39 : rxq_req 172938 rxq_success 0
rx queue 41 : rxq_req 839 rxq_success 839
rx queue 42 : rxq_req 889708 rxq_success 889708
rx queue 43 : rxq_req 1161171 rxq_success 379068
Max pkt delay 5038 msec thread pid 22 <<<<<<<<<<<<< thread with process ID 22
In this case, thread with process id 22 is hogging the CPU for over 5s.
Next, we're checking which thread has process id 22 assigned, from shell:
% cprod -A fpc0 -c "show threads"
CPU hog check enabled
Max run time 2500
PID PR State Name Stack Use Time (Last/Max/Total) cpu
--- -- ------- --------------------- --------- ---------------------
...
22 M asleep SFP 2312/73824 0/197/26772792 ms 0% <<<<<<<<<<<<< name of process 22 is SFP
If the thread hogging the CPU is "SFP" or "PoE", then issue is resolved in public PR1707762.
2024-10-30: changed wording for more clarity
2024-09-26: added public link and corrected some details2023-05-19: initial publication