Description

The intention of the article is to provide a workaround as to how to identify the prefix which causes NSR activation between the RE’s to fail causing BGP session on backup RE to flap on an EVO platform. The article also provides the log collection to be done in order to identify the prefix .

Symptoms

In a customer environment,  when upgrade activity was performed and NSR was activated between the RE’s BGP on Backup RE started flapping causing the NSR activation to be failed .

show task replication                                     
Stateful Replication: Enabled

RE mode: Master
Protocol        Synchronization Status
OSPF          Complete        
BGP           InProgress       
IS-IS          Complete        
PIM           Complete        
MPLS          Complete        
LDP           Complete        
RV           Complete 

May 29 16:05:06 2024 re1 
rpd[2956]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2600::2 (External AS 115) changed state from Established to Idle (event Closed) (instance master)
May 29 16:06:09 2024 re1 rpd[2956]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2600::2 (External AS 115) changed state from Idle to Established (event RecvEstab) (instance master)
May 29 16:07:36 2024 re1 rpd[2956]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2600::2 (External AS 115) changed state from Established to Idle (event Closed) (instance master)
May 29 16:08:38 2024 re1 rpd[2956]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2600::2 (External AS 115) changed state from Idle to Established (event RecvEstab) (instance master)


 

Solution

To provide resolution we need to first start collecting the Debug collector from both the RE’s . From the journalctl logs it was noticed that on the RE1 logs JSR(Junos Socket Replication)  was removed.

May 30 10:31:33.958886 JTASK_SYSTEM: JSR socket peek failed for task BGP_15169. 192.168.1.3 error:Unknown error Unknown error 1070731176

As suspicion was with JSR , below logs and traceoptions needs to be enabled in a service impacting MW.

set protocols bgp traceoptions file bgp.log size 1g files 100
set protocols bgp traceoptions flag nsr-synchronization detail
set protocols bgp traceoptions flag commit-synchronize
set protocols bgp traceoptions flag thread-io
set protocols bgp traceoptions flag state
set protocols bgp traceoptions flag task
set protocols bgp traceoptions flag timer


RPD live core(Both RE's) and below outputs to be collected as well from both RE's. 
request system core-dump routing-running 

show bgp neighbor
show bgp replication detail


From the JSR (Kernel side)

  • Enable the JSR logs  from shell prompt sysctl net.jsr.debug_level=13 on both master and backup.  Keep the above sysctl up for say 20 mins.
  • Collect the journal logs on both master and backup RE using: journalctl  -o short-precise  -since "30 minutes ago" > /var/tmp/journal.log
  • From both master and backup collect 1)cat /proc/net/jsr/*  2)cat /proc/net/tcpr/*  
  • Revert the sysctl net.jsr.debug_level=3 on both master and backup.
From the above collected logs from BGP traces , it was observed Backup RE is flapping due to failure in PEEK system call .

May 30 11:43:53.209414 JTASK_SYSTEM: JSR socket peek failed for task BGP_20001. 192.168.1.2 error:Unknown error Unknown error 1070731176
May 30 11:43:53.209455 RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 192.168.1.2 (Internal AS 20001) changed state from Established to Idle (event Closed) (instance master)

From the nsr debug logs rsynch session was seen flapping as socket pipe between the RE's were broken . Rsynch is used between the RE's to keep the RPD in synch so at the time of failure of Master RE , Backup RE's RPD is able to take up the role and keep protocols in synch. 
jtac-bgp.log:Jun  7 07:26:49.580545 rsync session terminated because New connection from backup
jtac-bgp.log:Jun  7 07:26:59.995919 rsync session terminated because New connection from backup
jtac-bgp.log:Jun  7 07:28:41.299967 rsync session terminated because Socket replication deactivated
jtac-bgp.log.0:Jun  7 07:24:11.154031 rsync session terminated because New connection from backup
jtac-bgp.log.0:Jun  7 07:24:21.578347 rsync session terminated because New connection from backup.

From the debugs and logs it was later identified that there could possibly be a prefix that is causing the rsynch connection to fail. In order to catch the faulty prefix .

TCPDUMP is required to be collected for the internal interface which is being used by RE’s to identify which prefix is causing the rsynch session to flap . JTDV0  is internal interface. Along with the above logs of BGP & NSR traces below tcpdump can be collected from shell of both RE's . 

tcpdump -w /var/tmp/master_re_dut_jtdv0.pcap -S -n -i jtdv0 tcp port 179

From the TCPDUMP below prefix was identified which did not have a local address when checked on the customer RSI & the same IP was configured on a local interface.

2024/174 06:07:01.086790 192.168.10..1 192.168.10.1 TCP 134 0 53317 ? 179 [SYN] Seq=7276056 Win=16384 Len=0 MD5 MSS=8192 SACK_PERM WS=1

Internet Protocol Version 4, Src: 128.0.0.4, Dst: 128.0.0.1
Internet Protocol Version 4, Src: 192.168.10.1, Dst: 192.168.10.1
Transmission Control Protocol, Src Port: 53317, Dst Port: 179, Seq: 7276056, Len: 0

Peer: 192.168.10.1 AS 115 Local: unspecified AS 101
 Description: RR
 Group: AS115_MARKET: master
 Forwarding routing-instance: master
 Type: External State: Idle Flags: <>
 Last State: NoState Last Event: NoEvent
 Last Error: None
Options: <AuthKey LogUpDown AddressFamily PeerAS PrefixLimit Refresh>
 Options: <MtuDiscovery>
 Options: <GracefulShutdownRcv>
 Authentication key is configured
 Address families configured: inet-unicast
 Holdtime: 90 Preference: 170
 Graceful Shutdown Receiver local-preference: 0
 Prefixlimit configured for NLRI: inet-unicast Limit: 100 Action: Log Only
 Number of flaps: 0
 Malformed attributes log interval: 300 route limit: 1000
 Trace options: all
 Trace file: /var/log//bgp_log size 1073741824 files 25

ae8 {
mtu 9192;
unit 0 {
family inet {
address 192.168.10.1/31;
 }
 }
 }

This is causing rscynch in BGP code as the Master RE is unable to identify whether this is a genuine rsynch connection or not. Removing the BGP configuration in idle state fixed the issue and NSR synch was successful .

Modification History

2024-06-28 : Article Created