This article describes a PFE lookup loop caused by PBR(Policy based routing) is incorrectly configured. Lookup loop will cause the packet to stay within the PPE thread for a long time, leading to PPE thread timeout, which might be followed by WAN or Fabric parcel timeouts with a major alarm raised. Then you may see PFE disable script triggers and disable the PFE.
Here are the logs seen when such a situation occurs.
Aug 2 10:42:04 mx-re0 fpc0 EA[0:0]_PPE 72 Errors thread timeout error Aug 2 10:42:04 mx-re0 fpc0 EA[0:0].disp[3] PRIMARY_TIMEOUT (PPE 0 Zone 16). Aug 2 10:42:08 mx-re0 fpc0 EA[0:0]_PPE 25 Errors thread timeout error Aug 2 10:42:08 mx-re0 fpc0 EA[0:0].disp[1] PRIMARY_TIMEOUT (PPE 1 Zone 9). ... Aug 2 11:49:52 mx-re0 fpc0 MQSS(0): DRD: Error: WAN reorder ID timeout error - Valid 1, Reorder ID 1144 Aug 2 11:49:52 mx-re0 fpc0 cmtfpc_mqss_drd_reorder_id_timeout_callback: Injecting recovery parcel PFE/CC/DRD/RID=0/0/0/1144, Data = d04780200000000000000000000000000000 Aug 2 11:49:53 mx-re0 fpc0 MQSS(0): DRD: RORD0: CMD reorder ID error - Command 13, Reorder ID 1144, QID 0 Aug 2 11:49:53 mx-re0 fpc0 EA[0:0]_PPE 0 Errors thread timeout error Aug 2 11:49:53 mx-re0 fpc0 EA[0:0].disp[0] PRIMARY_TIMEOUT (PPE 0 Zone 13). Aug 2 11:50:13 mx-re0 fpc0 PFE 0: 'PFE Disable' action performed. Bringing down ifd et-0/1/3 279
If these events are seen without any other HW related errors from this local FPC or remote PFE, then it might be caused by a lookup loop. In such cases, auto ttrace will be generated for such packets, which can be used to explore the incoming packet header and processing inside the PPE.
Auto ttraces file collected from “/var/tmp” folder, the auto ttrace file name looks like "ttrace_fpcX_LKUP_xxxxx".
From auto ttrace file, "fbf_init_forwarding_context" repeated indicated that loop may caused by PBR was incorrectly configured.
138 fbf_init_forwarding_context @ 0x053e 245 fbf_init_forwarding_context @ 0x053e 352 fbf_init_forwarding_context @ 0x053e 459 fbf_init_forwarding_context @ 0x053e 566 fbf_init_forwarding_context @ 0x053e 673 fbf_init_forwarding_context @ 0x053e 780 fbf_init_forwarding_context @ 0x053e 887 fbf_init_forwarding_context @ 0x053e 994 fbf_init_forwarding_context @ 0x053e
Decode the packet in auto ttrace file, we can get the source/destination addresses info.
For example, the packet source and destination address as below:
Source Address: 10.0.0.1 Destination Address: 192.168.0.1
The destination route in one route table(vr2) is pointed to other route table(vr1) in this example.
user@mx-re0> show route 192.168.0.1 extensive vr2.inet.0: 198212 destinations, 198215 routes (198212 active, 0 holddown, 0 hidden) 192.168.0.0/27 (1 entry, 1 announced) TSI: KRT in-kernel 192.168.0.0/27 -> {Table} *Direct Preference: 0 Next hop type: Interface, Next hop index: 0 Address: 0x1ed22ea4 Next-hop reference count: 3 Next hop: via irb.223, selected State: <Secondary Active Int> Local AS: 64515 Age: 2w3d 6:12:56 Validation State: unverified Task: IF Announcement bits (2): 0-KRT 2-Resolve tree 6 AS path: 64515 I Primary Routing Table vr1.inet.0
From configuration, matching source address, this packet in vr1 table will be sent to vr2 accourding to PBR.
routing-instances { vr1 { forwarding-options { family inet { filter { output PBR-FILTER; } } } firewall { family inet { filter PBR-FILTER { term 1 { filter PBR-ACL; } term DEFAULT { then accept; } } filter PBR-ACL { term 1 { from { source-prefix-list { SOURCE-PREFIX; } } then { routing-instance vr2; } } policy-options { prefix-list SOURCE-PREFIX { 10.0.0.0/24; }
Then this packet lookup looped between table vr1 and vr2.To resolve this issue, break the loop according to the actual packet situation.