After FPC (Flexible PIC Concentrator) is rebooted or replaced, log messages such as ‘fpc0 ICHIP(0)_REG_ERR:first cell drops in ichip fi rord : %PFE-3: 38' are reported. No errors are seen on the PFE (Packet Forwarding Engine) level.

An FPC rebooted in M320 and reported the following log messages:

May 18 01:21:04.833 2016 r01.hk.re0 fpc0 ICHIP(0)_REG_ERR:first cell drops in ichip fi rord : %PFE-3: 62
May 18 01:21:04.833 2016 r01.hk.re0 fpc0 ICHIP(1)_REG_ERR:first cell drops in ichip fi rord : %PFE-3: 297
May 18 02:15:21.112 2016 r01.hk.re0 fpc0 ICHIP(0)_REG_ERR:first cell drops in ichip fi rord : %PFE-3: 63
May 18 05:44:23.668 2016 r01.hk.re0 fpc0 ICHIP(0)_REG_ERR:first cell drops in ichip fi rord : %PFE-3: 64
May 18 09:23:29.209 2016 r01.hk.re0 fpc0 ICHIP(0)_REG_ERR:first cell drops in ichip fi rord : %PFE-3: 65
May 18 12:00:16.143 2016 r01.hk.re0 fpc0 ICHIP(0)_REG_ERR:first cell drops in ichip fi rord : %PFE-3: 66

FPC 0 REV 09 710-025464 CACY2956 M320 E3-FPC Type 3
PIC 0 REV 09 750-024784 CADW8638 1x 10GE(LAN/WAN) IQ2E
Xcvr 0 REV 01 740-031833 UT81TC0 XFP-10G-LR
PIC 1 REV 22 750-009553 CACZ7971 4x OC-48 SONET
Xcvr 0 REV 01 740-016069 PRE2K1X SFP-IR
Xcvr 1 REV 01 740-016069 PRE2K1V SFP-IR
Xcvr 2 REV 01 740-016069 PRE2K1G SFP-IR
Xcvr 3 REV 01 740-016069 PRE2K1D SFP-IR
I3MB A REV 07 710-016681 CACY2445 M320 E3-FPC I3 Mez Board
I3MB B REV 07 710-016681 CACY2388 M320 E3-FPC I3 Mez Board

However, no errors were found when checking the PFE statistics of this rebooted FPC. After performing ping tests where traffic crossed the PFE, the log messages disappeared.

This is the RODR (Reordering Engine) sequence number issue on the old Gimlet chipsets. Although the rebooted FPC was not a Gimlet chipset, there are Gimlet FPCs in the same chassis.

FPC 1 REV 04 710-013511 WP2652 M320 E2-FPC Type 1
CPU REV 04 710-013562 WP2686 M320 FPC CPU
PIC 0 REV 03 750-027457 YD5823 10x CHE1 IQ
PIC 1 REV 13 750-002911 WN2885 4x F/E, 100 BASE-TX
PIC 2 REV 13 750-002911 WN2911 4x F/E, 100 BASE-TX
PIC 3 REV 17 750-012266 YT0578 4x 1GE(LAN), IQ2
Xcvr 0 REV 01 740-031850 14T459600111 SFP-LX10
Xcvr 1 REV 01 740-031850 14T459600128 SFP-LX10
Xcvr 2 REV 01 740-011783 PD23A18 SFP-LX10
Xcvr 3 REV 01 740-031850 14T459600126 SFP-LX10
MMB 1 REV 03 710-010556 WP6006 MMB-1M-288mbit

In Gimlet, when sending a fabric package across, you keep a sequence number of per-PFE basis which is used to reorder cells.

If you have any board reset, the restart board will have the sequence number start from 0, but the far end will not reset it. As a result, all cells from the restarting board to that Gimlet board will be dropped in the RODR board until you have "wasted" enough data to make the sequence number sync up again.

Pick up the neighbors via the Gimlet FPC and send transit ping across the fabric to sync up the RODR sequence number again. 

Example

1. Neigbouring routers for R01.hk(M320):
   
   R1---FPC1/PFE1--(M320)--FPC0/PFE0---R2
R1---FPC5/PFE0--(M320)--FPC0/PFE0---R2
R1---FPC5/PFE1--(M320)--FPC0/PFE0---R2
R1---FPC6/PFE0--(M320)--FPC0/PFE0---R2
R1---FPC6/PFE1--(M320)--FPC0/PFE0---R2
R1---FPC7/PFE0--(M320)--FPC0/PFE0---R2
R1---FPC7/PFE1--(M320)--FPC0/PFE0---R2

R1---FPC1/PFE1--(M320)--FPC0/PFE1---R2
R1---FPC5/PFE0--(M320)--FPC0/PFE1---R2
R1---FPC5/PFE1--(M320)--FPC0/PFE1---R2
R1---FPC6/PFE0--(M320)--FPC0/PFE1---R2
R1---FPC6/PFE1--(M320)--FPC0/PFE1---R2
R1---FPC7/PFE0--(M320)--FPC0/PFE1---R2
R1---FPC7/PFE1--(M320)--FPC0/PFE1---R2


• Detailed neighbor routers for R01.HK for fpc (1/5/6/7) and fpc0:
  
  ======================================================
  fpc0/pfe0
20.33.208.167------------------------<20.33.208.166> <ge-1/3/3> r01.hk <xe-0/0/0> <20.33.208.178>----<20.33.208.179> <xe-0/0/0> r02.hk
10.115.144.124-----------------------<10.115.144.125> <xe-5/0/0> r01.hk <xe-0/0/0> <20.33.208.178>----<20.33.208.179> <xe-0/0/0> r02.hk
11.27.106.154------------------------<11.27.106.153> <ge-5/1/1> r01.hk <xe-0/0/0> <20.33.208.178>----<20.33.208.179> <xe-0/0/0> r02.hk
r02.hk <xe-4/0/0> <20.33.208.39>-----<20.33.208.38> <xe-6/0/0> r01.hk <xe-0/0/0> <20.33.208.178>----<20.33.208.179> <xe-0/0/0> r02.hk
11.27.106.250------------------------<11.27.106.249> <xe-6/1/0> r01.hk <xe-0/0/0> <20.33.208.178>----<20.33.208.179> <xe-0/0/0> r02.hk
63.218.211.45------------------------<63.218.211.46> <xe-7/0/0> r01.hk <xe-0/0/0> <20.33.208.178>----<20.33.208.179> <xe-0/0/0> r02.hk
r01.klj <so-5/2/3> <20.33.208.45>----<20.33.208.44> <so-7/1/0> r01.hk <xe-0/0/0> <20.33.208.178>----<20.33.208.179> <xe-0/0/0> r02.hk

fpc0/pfe1
20.33.208.167-------------------------<20.33.208.166> <ge-1/3/3> r01.hk <so-0/1/0> <20.33.208.76>----<20.33.208.77> <so-5/0/0> r01.tpe
10.115.144.124------------------------<10.115.144.125> <xe-5/0/0> r01.hk <so-0/1/0> <20.33.208.76>----<20.33.208.77> <so-5/0/0> r01.tpe
11.27.106.154-------------------------<11.27.106.153> <ge-5/1/1> r01.hk <so-0/1/0> <20.33.208.76>----<20.33.208.77> <so-5/0/0> r01.tpe
r02.hk <xe-4/0/0> <20.33.208.39>------<20.33.208.38> <xe-6/0/0> r01.hk <so-0/1/0> <20.33.208.76>----<20.33.208.77> <so-5/0/0> r01.tpe
11.27.106.250-------------------------<11.27.106.249> <xe-6/1/0> r01.hk <so-0/1/0> <20.33.208.76>----<20.33.208.77> <so-5/0/0> r01.tpe
63.218.211.45-------------------------<63.218.211.46> <xe-7/0/0> r01.hk <so-0/1/0> <20.33.208.76>----<20.33.208.77> <so-5/0/0> r01.tpe
r01.klj <so-5/2/3> <20.33.208.45>-----<20.33.208.44> <so-7/1/0> r01.hk <so-0/1/0> <20.33.208.76>----<20.33.208.77> <so-5/0/0> r01.tpe

• Ping test:
  
  {MASTER}
[email protected]> ping 20.33.208.167 source 20.33.208.179 interface xe-0/0/0 count 1000 rapid
PING 20.33.208.167 (20.33.208.167): 56 data bytes
..............................................................!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--- 20.33.208.167 ping statistics ---
1000 packets transmitted, 917 packets received, 8% packet loss
round-trip min/avg/max/stddev = 48.155/49.569/96.596/3.877 ms