Description

This article describes a PFE lookup loop caused by PBR(Policy based routing) is incorrectly configured. Lookup loop will cause the packet to stay within the PPE thread for a long time, leading to PPE thread timeout, which might be followed by WAN or Fabric parcel timeouts with a major alarm raised. Then you may see PFE disable script triggers and disable the PFE.

Here are the logs seen when such a situation occurs.

Aug 2 10:42:04 mx-re0 fpc0 EA[0:0]_PPE 72 Errors thread timeout error
Aug 2 10:42:04 mx-re0 fpc0 EA[0:0].disp[3] PRIMARY_TIMEOUT (PPE 0 Zone 16).
Aug 2 10:42:08 mx-re0 fpc0 EA[0:0]_PPE 25 Errors thread timeout error
Aug 2 10:42:08 mx-re0 fpc0 EA[0:0].disp[1] PRIMARY_TIMEOUT (PPE 1 Zone 9).
...
Aug 2 11:49:52 mx-re0 fpc0 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1144
Aug 2 11:49:52 mx-re0 fpc0 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1144, Data = d04780200000000000000000000000000000
Aug 2 11:49:53 mx-re0 fpc0 MQSS(0): DRD: RORD0: CMD reorder ID error - Command 13, Reorder ID 1144, QID 0
Aug 2 11:49:53 mx-re0 fpc0 EA[0:0]_PPE 0 Errors thread timeout error
Aug 2 11:49:53 mx-re0 fpc0 EA[0:0].disp[0] PRIMARY_TIMEOUT (PPE 0 Zone 13).
Aug 2 11:50:13 mx-re0 fpc0 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-0/1/3 279

If these events are seen without any other HW related errors from this local FPC or remote PFE, then it might be caused by a lookup loop. In such cases, auto ttrace will be generated for such packets, which can be used to explore the incoming packet header and processing inside the PPE.

Solution

Auto ttraces file collected from “/var/tmp” folder, the auto ttrace file name looks like "ttrace_fpcX_LKUP_xxxxx".

From auto ttrace file, "fbf_init_forwarding_context" repeated indicated that loop may caused by PBR was incorrectly configured.

138 fbf_init_forwarding_context @ 0x053e
245 fbf_init_forwarding_context @ 0x053e
352 fbf_init_forwarding_context @ 0x053e
459 fbf_init_forwarding_context @ 0x053e
566 fbf_init_forwarding_context @ 0x053e
673 fbf_init_forwarding_context @ 0x053e
780 fbf_init_forwarding_context @ 0x053e
887 fbf_init_forwarding_context @ 0x053e
994 fbf_init_forwarding_context @ 0x053e

Decode the packet in auto ttrace file, we can get the source/destination addresses info.

For example, the packet source and destination address as below:

Source Address: 10.0.0.1
Destination Address: 192.168.0.1

The destination route in one route table(vr2) is pointed to other route table(vr1) in this example. 

user@mx-re0> show route 192.168.0.1 extensive 
vr2.inet.0: 198212 destinations, 198215 routes (198212 active, 0 holddown, 0 hidden)
192.168.0.0/27 (1 entry, 1 announced)
TSI:
KRT in-kernel 192.168.0.0/27 -> {Table}
    *Direct Preference: 0
        Next hop type: Interface, Next hop index: 0
        Address: 0x1ed22ea4
        Next-hop reference count: 3
        Next hop: via irb.223, selected
        State: <Secondary Active Int>
        Local AS: 64515 
        Age: 2w3d 6:12:56 
        Validation State: unverified 
        Task: IF
        Announcement bits (2): 0-KRT 2-Resolve tree 6 
        AS path: 64515 I 
        Primary Routing Table vr1.inet.0  

From configuration, matching source address, this packet in vr1 table will be sent to vr2 accourding to PBR. 

routing-instances {
  vr1 {
    forwarding-options {
      family inet {
        filter {
          output PBR-FILTER;
        }
      }
    }

firewall {
  family inet { 
    filter PBR-FILTER {
      term 1 {
        filter PBR-ACL;
      }
      term DEFAULT {
        then accept;
      }
    }
    filter PBR-ACL {
      term 1 {
        from {
          source-prefix-list {
            SOURCE-PREFIX;
          }
        }
        then {
          routing-instance vr2;    
        }
      }

policy-options {      
  prefix-list SOURCE-PREFIX {
    10.0.0.0/24;
  }  

Then this packet lookup looped between table vr1 and vr2.
To resolve this issue, break the loop according to the actual packet situation.

 

Modification History

2024-08-27 : Article Created