Description

The "Command sequence error" message reports a problem in the DRD (Dispatch and Reorder Block) state machine.


This is a Troubleshooting Article for a PFE ASIC Syslog Event.
To view other documented syslog events related to XMCHIP, XLCHIP, MQCHIP, LUCHIP, EACHIP, and PECHIP, see KB31893 - Master Index of Articles for Troubleshooting PFE ASIC Syslog Events [juniper.net] .

.

Symptoms

When a "Command sequence error" event occurs, a message similar to the following is reported:

Apr 3 18:43:31 re0 fpc6 XMCHIP(0):XMCHIP(0): DRD0: Command sequence error - Reorder ID 816
Sep 6 05:44:32 router0 fpc9 :XMCHIP(0): %PFE-0-DRD1: Command sequence error - Reorder ID 1443

Indications:

  • Service impact: p ermanent impact of packet forwarding .
  • Other error messages will be reported, such as fabric self-ping or host loopback wedge to illustrate that forwarding is compromised.
    • MPC fabric selfping blackhole (1) exceed raising threshold (1) occurrance (0) for module/pfe (8:3)
    • ​​ Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID  <x>
  • Alarm will be raised.

 

Solution

The issue is with the DRD0 block for the WAN side and the DRD1 for the fabric side. This could be caused by either hardware, transient memory event, or software. Mostly this is Software related. Review the XLCHIP/LUCHIP PPE traps, error messages, and inter-ASIC HSL2 errors to isolate the cause.

Some other PRs that could cause this: PR1082932 , PR1157173 , PR1166479 . Also, PR1067234 addressed an exposure.

The following KB entries might be relevant:
KB31617 - Syslog message: XMCHIP.*DRD.*Fabric parcel timeout error [juniper.net]
KB31701 - Syslog message: XMCHIP.*DRD.*Wan parcel timeout error [juniper.net]
KB32168 - Syslog message: HOST LOOPBACK WEDGE DETECTED [juniper.net]

 

{DIDYOUKNOWSERVICENOWTOKEN.EN_US}

Perform these steps to determine the cause and resolve the problem (if any).  Continue through each step until the problem is resolved.

  1. Collect the show command output.

    {SYSLOGSERVICENOWTOKEN.EN_US}

    Capture the output to a file (in case you have to open a technical support case). To do this, configure each SSH client/terminal emulator to log your session.

    show log messages
    show log chassisd
    start shell network pfe <fpc#>
    show nvram
    show syslog messages
    exit

  2. Analyze the show command output.

    In the 'show log messages', review the events that occurred at or just before the appearance of the "Command sequence error" message. Frequently these events help identify the cause.  You will need to look for XL/LU PPE traps, error messages, and inter-ASIC HSL2 errors. 

    1. No RMA required.
    2. With the fix for PR1186421 , major alarms will default to take the disable-pfe action. An FPC reset during a maintenance window is needed to bring the PFE back into service.
    3. Without all the PR fixes, the generic pfe-disable event script will detect the wedge condition that happens as a result of this event and invoke the pfe-disable action.
    4. An FPC reset at a later time is needed to bring the PFE back into service.
 

This article is indexed in  KB31893 - Primary Index of Articles for Troubleshooting PFE ASIC Syslog Events [juniper.net] ; tag XMCHIPTSG


Tip: When looking at an event in the logs, it is important to focus on the first error message in a collection of syslog messages. The first error message is usually the cause of all the follow-on error messages. The follow-on collateral damage error messages can be ignored.

 

Modification History

2019-10-14: Article reviewed for accuracy; added this symptom is mostly Software related.