Description

If there is a mis-configuration which causes the packet to bounce from one instance to another in an endless loop, it may congest the lookup chip and impact the router performance.

This usually happens when a static route is configured with "next-table" knob so the packet goes from one instance to another and somehow the packet is directed back to the same instance due to another route with next-table knob or filter based forwarding pointing the packet back to the same instance.

This article provides an example of such loop and tries to determine the packet which is getting looped. Based on that, corrective action can be taken.  

Symptoms

The following errors may appear sporadically or continuously based on the number of these packets that are coming to the router: 

Nov 12 04:41:57  jtac-lab-re0 : %PFE-5: fpc2 LUCHIP(0) Congestion Detected, Active Zones f:f:f:f:f:f:f:f:f:f:f:f:f:f:f:f
Nov 12 04:42:06  jtac-lab-re0 : %PFE-5: fpc2 LUCHIP(0) Congestion Detected, Active Zones f:f:f:f:f:b:f:f:f:f:f:f:7:f:f:f
Nov 12 04:42:06  jtac-lab-re0 : %PFE-5: fpc2 LUCHIP(0) Congestion Detected, Active Zones f:f:f:f:f:b:f:f:f:f:f:f:7:f:f:f

Nov 12 05:22:45  jtac-lab-re0 : %PFE-3: fpc2 LUCHIP(0): Secondary PPE 11 zone 17 timeout.
Nov 12 05:22:45  jtac-lab-re0 : %PFE-3: fpc2 PPE Thread Timeout Trap:  Count 36327, PC 48,     0x0048:  KT4Way_ARS00_trap
Nov 12 05:43:43  jtac-lab-re0 : %PFE-3: fpc2 LUCHIP(0): Secondary PPE 15 zone 29 timeout.
Nov 12 05:43:43  jtac-lab-re0 : %PFE-3: fpc2 PPE Thread Timeout Trap:  Count 45686, PC 6c0,     0x06c0:  KTLeaf_or_Span

Solution

  1. First, figure out the impacted traffic by checking if there is any ppe_trap file generated around the time congestion/ppe error appeared.

    file list /var/tmp/ detail | match ppe

    If there is any ppe trap file, get hex dump of the packet from this file:

    file show /var/tmp/ppe_trap_fpc5_LU_1_00.1 | find dispatch 

          Dispatch size is 92
      0006c0308008544b
      8c93afc1544b8c93
      bfc1884700013332
      4500003c00004000
      32064ff44043f737
      4044c109780f97d0
      986a971be082ee67
      a0127120c71e0000
      020405b40402080a
      3eda99553a1af64d
      01030307

    The highlighted portion of the dispatch is the actual packet.

  2. Copy/paste the hex dump into any hex to pcap conversion tools available online to see the packet details.

    Packet: 

      Ethernet II
 
    Destination: JuniperN_93:bf:c1 (54:4b:8c:93:af:c1)
    Source: JuniperN_93:df:c1 (54:4b:8c:93:bf:c1)
    Type: MPLS label switched packet (0x8847)
 
   MultiProtocol Label Switching Header
 
    0000 0000 0000 0001 0011 .... .... .... = MPLS Label: 19
    .... .... .... .... .... 001. .... .... = MPLS Experimental Bits: 1
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 0011 0010 = MPLS TTL: 50
 
   Internet Protocol Version 4
 
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 60
    Identification: 0x0000 (0)
    Flags: 0x4000, Don't fragment
    Time to live: 50
    Protocol: TCP (6)
    Header checksum: 0x4ff4
    Header checksum status: Unverified
    Source: 64.67.247.55
    Destination: 64.68.193.9

    Based on the MAC address of the interface in the above packet, it can be determined in which instance that packet was received and what is the destination address.

  3. Check the destination route in the relevant instance: 

    show route 64.68.193.9 table instance-1.inet.0
    
instance-1.inet.0: 579 destinations, 2021 routes (576 active, 0 holddown, 4 hidden)
+ = Active Route, - = Last Active, * = Both
    
64.68.192.0/22     *[Static/5] 30w6d 15:26:25, tag 1421197
                    to table instance-2.inet.0

    There appears to be static route which points it to instance-2 to next-table knob

    set routing-instances instance-1 routing-options static route 64.68.192.0/22 next-table instance-2.inet.0

    On instance-2, there is a filter based forwarding (FBF) configuration which is sending the packet back to instance-1 thus causing the loop 

    set firewall family inet filter JTAC-FBF term term-1 from source-prefix-list JTAC-FBF-PREFIX-LIST
set firewall family inet filter JTAC-FBF term term-1 then routing-instance instance-1
set firewall family inet filter JTAC-FBF term default then accept

set routing-instances instance-2 forwarding-options family inet filter input JTAC-FBF
    
set policy-options prefix-list JTAC-FBF-PREFIX-LIST 64.68.192.0/22

In order resolve the issue, we can rectify the configuration which causes the forwarding loop.

Related Information