The XMCHIP(.): FI: Reorder cell timeout message reports there was a packet drop in fabric traffic.

When an XMCHIP(.): FI: Reorder cell timeout event occurs, a message similar to the following is reported:

The Fabric input block in each PFE maintains a re-order pointer for each fabric stream ( fabric stream is a number derived from the destination FPC and PFE ). If the incoming cell has a sequence number which is higher than this reorder pointer, the cells are believed to have arrived early and the cell timeout logs are reported.

This is usually due to a communication issue between an SCB and an XM chip-based FPC card.  The syslog message is generated either by software or hardware interactions, and log analysis will be needed to determine which one.  Hardware sources can include the FPC involved, or the Control Board between FPCs that is governing their respective traffic.  For example, manually taking an XMCHIP FPC offline or pulling a Control Board might cause this message to occur on an XM chip FPC.  Software can produce this syslog message if a core dump has occurred as a result of some other issue.

Perform these steps to determine the cause and resolve the problem (if any):

start shell pfe network fpc#
show syslog messages
show nvram
show pfe statistics traffic
show pfe statistics error
show xmchip # fi stats
show xmchip # fo stats
show xmchip # wi stats 0
show xmchip # wo stats 0
show xmchip # wi stats 1
show xmchip # wo stats 1
exit 

Replace the '#' character in the above list with the corresponding FPC or PFE chip number, as provided in the syslog messages. Collect the FPC outputs three times with a 1-minute time difference between each collection. This will show any changes in the error counts. Also look for any related events that occurred at or just before the FI: Reorder cell timeout message in the syslog or chassisd outputs.

1. Collect the show command output to help determine the cause of this message.

   {SYSLOGSERVICENOWTOKEN.EN_US}

   Capture the output to a file (in case you have to open a technical support case). To do this, configure each SSH client/terminal emulator to log your session.
    

2. Analyze the show command output. Look for any related events that occurred at or just before the error messages started reporting.
    

3. If there was a commit , or work on hardware being done such as inserting/removing/offlining cards, then this syslog message is expected and can be ignored.
   
   If there is a single occurrence of this error not followed by any other errors, then it can be ignored. If there are multiple occurrences of these errors, then follow the steps given below to troubleshoot further.
    

4. If a core dump was also generated , open a case with your technical support representative and provide the core dump for analysis.
    

5. During a maintenance window, as it will impact transit traffic, try the following:

   • Reset the XMCHIP FPC referenced in the syslog message.

   • Reseat the FPC card and the Control Board. Also check for bent pins on the connectors with a flashlight when doing this action.

   • Swap the FPC with a spare, or swap two FPC cards in each other's slots.
      

6. If these efforts do not resolve the problem, contact your technical support representative to investigate the issue further.

   {SYSLOGSERVICENOWCASE.EN_US}

If you are not running Junos Space Service Now, attach the information collected above to the case.

• Syslog message: MQCHIP(.) FI Reorder cell timeout