Description

On EX4400, BFD session flap could be seen without specific trigger, but you may observe it along with SMBUS related logs.
There are some PRs that fixed this issue, and would share the information on this KB.

Symptoms

On EX4400, BFD flap might be seen due to 'Detect Timer Expiry'. If there are multiple BFD sessions, those sessions will also be affected at the same time.
Here is the sample logs.

Aug 30 21:16:17  Switch bfdd[9122]: %DAEMON-4-BFDD_STATE_UP_TO_DOWN: BFD Session 192.168.1.1 (IFL 543) state Up -> Down LD/RD(20/16) Up time:4d 20:51 Local diag: CtlExpire Remote diag: None Reason: Detect Timer Expiry.
Aug 30 21:16:17  Switch bfdd[9122]: %DAEMON-4-BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 20, new state: down, interface: irb.99, peer addr: 192.168.1.1
Aug 30 21:16:17  Switch rpd[9104]: %DAEMON-5-RPD_OSPF_NBRDOWN: OSPF neighbor 192.168.1.1 (realm ospf-v2 irb.99 area 0.0.0.0) state changed from Full to Down due to InActiveTimer (event reason: BFD session timed out and neighbor was declared dead)
~snip~
Aug 30 21:16:23  Switch bfdd[9122]: %DAEMON-5-BFDD_TRAP_SHOP_STATE_UP: local discriminator: 20, new state: up, interface: irb.99, peer addr: 192.168.1.1
When the BFD flap was seen along with SMBUS log below, this KB might be helpful to fix the issue.

Case1) 
%PFE-6: fpc0 PFE_SMBUS_UKERN_SLEEP: tvp_smbus_sleep: Ukern thread sleep for 1seconds
Case2) 
%USER-6: smb_transfer: SMBUS ioctl took 5001(ms) to execute slave:0xe6!
%USER-3: smb_transfer: SMBus ioctl failed, retry:1 slave:0xe6
%USER-6: i2c_flock: SMBUS flock took 4946(ms) to execute slave:0x0

Solution

There are two fixed for BFD flap occurred along with SMBUS log.

Case1) 
%PFE-6: fpc0 PFE_SMBUS_UKERN_SLEEP: tvp_smbus_sleep: Ukern thread sleep for 1seconds
https://prsearch.juniper.net/problemreport/PR1707762
The root cause was there on SMBus(System Management Bus) in PFE which blocks the process of fast path(used for control protocol) while slow path thread(used for gathering hardware stats/info) taking a time.
The fix was made to open the SMBus for fast path even though slow path is taking a time.

Case2) 
%USER-6: smb_transfer: SMBUS ioctl took 5001(ms) to execute slave:0xe6!
%USER-3: smb_transfer: SMBus ioctl failed, retry:1 slave:0xe6
%USER-6: i2c_flock: SMBUS flock took 4946(ms) to execute slave:0x0
https://prsearch.juniper.net/problemreport/PR1734435
PR1734435 takes care of an interrupt immediately after submitting an I2C request and before the submitter thread could sleep and wait for an interrupt.
Since the interrupt is early before the thread could go to sleep, when the thread goes to sleep, it is not waken up by interrupt leading the thread to timeout and assume that the transaction has failed.

Modification History

2023-10-16