Description

This article addresses an issue where the Physical Forwarding Engine (PFE) encounters a PCS (Physical Coding Sublayer) modify error on a Ethernet interface. The error manifests as "Pfe: pcsE-X/X/X:X-fwd0 Brcm PFE PCS modify: ignoring mismatched response guid." This can lead to port flapping and eventual port downtime, accompanied by generic port flap logs.

Symptoms

Here are symptoms to this kind of issue : 

  • Physical interface (e.g., et-0/0/2) shows as enabled, but the physical link is down.

  • Interface statistics show that the port has never flapped.

  • Device flags indicate "Present Running Down".

  • Interface flags show "Hardware-Down SNMP-Traps".

  • Journal logs display PCS modify errors and mismatched response GUIDs.

  • Message logs show transceiver lane alarms, particularly Tx CDR LOL (Loss of Lock) and Tx LOS (Loss of Signal) alarms.

Commands to Gather Information:

To view interface status: show interfaces et-0/0/2 extensive

To collect debug information: request system debug-info mode lite node re0

To access shell and collect journal logs:
start shell user root
journalctl -a -b --output="short-precise" > /var/tmp/journal_re.log

To view PFE syslog: show log pfe-show_syslog

Output: 

Physical interface: et-0/0/2, Enabled, Physical link is Down

 Interface index: 1048, SNMP ifIndex: 509, Generation: 588410

 Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Disabled, Media type: Fiber

 Device flags  : Present Running Down

 Interface flags: Hardware-Down SNMP-Traps

 CoS queues   : 8 supported, 8 maximum usable queues

 Hold-times   : Up 0 ms, Down 0 ms

 Damping    : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed

 Current address: 68:22:8e:15:4e:14, Hardware address: 68:22:8e:15:4e:14

 Last flapped  : Never

 Statistics last cleared: Never

 

Journel_re_log:

 

Oct 28 20:58:00.162395 evo-pfemand[8837]: [Info] Pfe: processPcsFec pcsE-0/0/2:0-fwd0 publishing pcsE guid: 743029342801 fec_state resp guid: 807459542470 txn ID: 12

Oct 28 20:58:00.163184  evo-pfemand[8837]: [Info] Pfe: Brcm PFE PCS modify port: et-0/0/2:0

Oct 28 20:58:00.163202  evo-pfemand[8837]: [Info] Pfe: processPcsInit pcsE-0/0/2:0-fwd0 OpState: OpState_OpSuccess

Oct 28 20:58:00.163223  evo-pfemand[8837]: [Info] Pfe: pcsE-0/0/2:0-fwd0 Brcm PFE PCS modify: ignoring mismatched response guid: 807459542470 init_state tx ID: 11

Oct 28 20:58:00.163240  evo-pfemand[8837]: [Info] Pfe: processPcsFec pcsE-0/0/2:0-fwd0 OpState: OpState_OpSuccess

Oct 28 20:58:00.163253 evo-pfemand[8837]: [Info] Pfe: processPcsFec pcsE-0/0/2:0-fwd0Deleting resp 807459542470 fec_state tx ID: 12

Oct 28 20:58:01.309185 evo-pfemand[8837]: [Info] Interface: controller modify for IFD:et-0/0/2 index:1011 GUID:588410520591 disabled:0 old disabled:0 harddown:1 old harddown:1 flags:0x40001 old flags:0x1 last flags:0x1 old pseudo flags: 0x40 new pseudo flags: 0x40 old flow control: 1 new flow control: 1 mode:0

Oct 28 20:58:01.309441 evo-pfemand[8837]: [Info] Pfe: Setting port link for Port: 5, pfeId: 0, enable: 1, protocolState1

Oct 28 20:58:01.309452  evo-pfemand[8837]: [Info] Pfe: enable link, IFD: et-0/0/2, port: 5

Oct 28 20:58:01.309468 evo-pfemand[8837]: [Info] BrcmPlusIf: ptr->unit0 0x7fa4093ce9d0

 

Message logs

 

Oct 28 03:45:08.823 2023  picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 0  Tx CDR LOL      alarm set 

Oct 28 03:45:08.823 2023  picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 1  Tx CDR LOL      alarm set 

Oct 28 03:45:08.823 2023  picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 2  Tx CDR LOL      alarm set 

Oct 28 03:45:08.823 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 3  Tx CDR LOL      alarm set 

Oct 28 03:45:09.774 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 0  Tx LOS        alarm clear 

Oct 28 03:45:09.774 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 1  Tx LOS        alarm clear 

Oct 28 03:45:09.774 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 2  Tx LOS        alarm clear 

Oct 28 03:45:09.774 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 3  Tx LOS        alarm clear 

 

pfe-show_syslog

 

Oct 28 20:49:40.285 2023  picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 0  Tx Bias High     alarm clear

Oct 28 20:49:43.111 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 0  Tx LOS        alarm set

Oct 28 20:49:43.111 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 1  Tx LOS        alarm set

Oct 28 20:49:43.111 2023  picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 2  Tx LOS        alarm set

Oct 28 20:49:43.111 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 3  Tx LOS        alarm set

Oct 28 20:49:44.897 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 0  Tx Power Low     alarm clear

Oct 28 20:49:44.897 2023  picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 1  Tx Power Low     alarm clear

Oct 28 20:49:44.897 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 2  Tx Power Low     alarm clear

Oct 28 20:49:44.897 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 3  Tx Power Low     alarm clear

Oct 28 20:49:44.897 2023 picd: PICD_XCVR_LANE_ALARM: xcvr-0/0/2 chan 0  Tx Power Warn Low  alarm clear




 

Solution

Here are some quick steps to identify and resolve the problem: 

  • Verify if the SFP transceiver is Juniper-supported and validated.

  • Try moving the SFP and connection to a different et-//* interface to check if the interface comes up without issues.

  • Perform a looped SFP test:

  • Connect a loopback fiber between the Tx and Rx of the SFP.

  • Verify that optical levels are visible.

  • Check the interface state.

  • If the issue persists, collect the following logs when the issue is present:

Debug information: request system debug-info mode lite node re0

Journal logs:
start shell user root
journalctl -a -b --output="short-precise" > /var/tmp/journal_re.log


If the problem continues after these steps, contact Juniper Networks Support with the collected logs and information for further assistance.
Consider checking for any available software updates that may address this PFE PCS modify error.
Monitor the interface for recurring issues and document any patterns in the occurrence of the error for troubleshooting purposes.

Modification History

2023-11-17 : Article Created