Description

This KB explains about below log messages, associated impact, and solution as well.

Aug 16 05:38:49  device : %PFE-3: fpc0 FAILED to read eeprom for port: 0
Aug 16 05:38:49  device : %PFE-3: fpc0 qsfp_tk_get_status_indicator: qsfp-0/0/0 failed to read status byte (addr 2). Err 4294967295
Aug 16 05:38:49  device : %PFE-3: fpc0 FAILED to read eeprom for port: 0
Aug 16 05:38:49  device : %PFE-3: fpc0 qsfp_tk_read_mem_page: qsfp-0/0/0 failed to select addr 127 err 4294967295
Aug 16 05:38:49  device : %PFE-3: fpc0 FAILED to write eeprom for port: 0
Aug 16 05:38:49  device : %PFE-3: fpc0 qsfp_tk_select_mem_page: qsfp-0/0/0 failed to select mem page 0 addr 0x7f err 1000
Aug 16 05:38:49  device : %PFE-3: fpc0 CMQFX: waiting for pfeman resync to complete
Aug 16 05:38:50  device : %PFE-3: fpc0 FAILED to read eeprom for port: 0
Aug 16 05:38:54  device : %PFE-3: fpc0 qsfp-0/0/0 failed to mask LOS alarms
Aug 16 05:38:54  device : %PFE-0: fpc0 SCHED: Thread 8 (QSFP) ran for 1585 ms without yielding
Aug 16 05:38:54  device : %PFE-0: fpc0 Scheduler Oinker
Aug 16 05:38:54  device : %PFE-0: fpc0 Frame 0: sp = 0xaf7061b8, pc = 0x807a793
Aug 16 05:38:54  device : %PFE-0: fpc0 Frame 1: sp = 0xaf7061d8, pc = 0x82ae2a4
Aug 16 05:38:54  device : %PFE-0: fpc0 Frame 2: sp = 0xaf7062a8, pc = 0x807ab30
Aug 16 05:38:54  device : %PFE-3: fpc0 CMQFX: waiting for pfeman resync to complete

Symptoms

Multiple interfaces might go down on the device and simultaneous generation of all or some of the below logs might be seen:-


Aug 16 05:26:24  device : %PFE-3: fpc0 qsfp-0/0/12 failed to mask LOS alarms
Aug 16 05:26:25  device : %PFE-3: fpc0 FAILED to write eeprom for port: 12
Aug 16 05:26:25  device : %PFE-3: fpc0 qsfp-0/0/12 failed to mask LOS alarms
Aug 16 05:26:27  device : %PFE-3: fpc0 FAILED to write eeprom for port: 12
Aug 16 05:26:27  device : %PFE-3: fpc0 qsfp-0/0/12 failed to mask LOS alarms
Aug 16 05:26:29  device : %PFE-3: fpc0 FAILED to write eeprom for port: 12
Aug 16 05:26:29  device : %PFE-3: fpc0 qsfp-0/0/12 failed to mask LOS alarms

or

Aug 16 05:38:49  device : %PFE-3: fpc0 FAILED to read eeprom for port: 0
Aug 16 05:38:49  device : %PFE-3: fpc0 qsfp_tk_get_status_indicator: qsfp-0/0/0 failed to read status byte (addr 2). Err 4294967295
Aug 16 05:38:49  device : %PFE-3: fpc0 FAILED to read eeprom for port: 0
Aug 16 05:38:49  device : %PFE-3: fpc0 qsfp_tk_read_mem_page: qsfp-0/0/0 failed to select addr 127 err 4294967295
Aug 16 05:38:49  device : %PFE-3: fpc0 FAILED to write eeprom for port: 0
Aug 16 05:38:49  device : %PFE-3: fpc0 qsfp_tk_select_mem_page: qsfp-0/0/0 failed to select mem page 0 addr 0x7f err 1000
Aug 16 05:38:49  device : %PFE-3: fpc0 CMQFX: waiting for pfeman resync to complete
Aug 16 05:38:50  device : %PFE-3: fpc0 FAILED to read eeprom for port: 0
Aug 16 05:38:54  device : %PFE-3: fpc0 qsfp-0/0/0 failed to mask LOS alarms
Aug 16 05:38:54  device : %PFE-0: fpc0 SCHED: Thread 8 (QSFP) ran for 1585 ms without yielding
Aug 16 05:38:54  device : %PFE-0: fpc0 Scheduler Oinker
Aug 16 05:38:54  device : %PFE-0: fpc0 Frame 0: sp = 0xaf7061b8, pc = 0x807a793
Aug 16 05:38:54  device : %PFE-0: fpc0 Frame 1: sp = 0xaf7061d8, pc = 0x82ae2a4
Aug 16 05:38:54  device : %PFE-0: fpc0 Frame 2: sp = 0xaf7062a8, pc = 0x807ab30
Aug 16 05:38:54  device : %PFE-3: fpc0 CMQFX: waiting for pfeman resync to complete

or 

Aug 16 05:39:52  device : %PFE-3: fpc0 FAILED to write eeprom for port: 0
Aug 16 05:39:54  device : %PFE-3: fpc0 qsfp-0/0/0 failed to mask LOS alarms
Aug 16 05:39:57  device : %PFE-3: fpc0 CMQFX: waiting for pfeman resync to complete
Aug 16 05:39:59  device : %PFE-3: fpc0 FAILED to write eeprom for port: 1
Aug 16 05:39:59  device : %PFE-3: fpc0 qsfp-0/0/1 failed to mask LOS alarms
Aug 16 05:39:59  device : %PFE-3: fpc0 CMQFX: waiting for pfeman resync to complete


Please note that you may see all the above patterns or a few on the device.

Solution

The log messages are related to i2c failure for QSFP on the device.
When these log messages are seen, multiple interfaces might go down on the device, and the Output of "show chassis hardware " will not display the Serial number details. 
user@device> show chassis hardware detail no-forwarding
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                        xxxxxxxxxxxxxxx      QFX5200-32C-32Q
Pseudo CB 0
Routing Engine 0          BUILTIN      BUILTIN           RE-QFX5200-32C-32Q
  ada0   9316 MB  QEMU                 QM00001           Virtio Block Disk
  ada1   4096 MB  QEMU                 QM00002           Virtio Block Disk
  ada2    512 MB  QEMU                 QM00003           Virtio Block Disk
  ada3   1024 MB  QEMU                 QM00004           Virtio Block Disk
  usb0 (addr 0.1) UHCI root HUB 0      Intel             uhub0
  usb1 (addr 1.1) EHCI root HUB 0      Intel             uhub1
  usb2 (addr 1.2) product 0x0020 32    vendor 0x8087     uhub2
FPC 0            REV 35   650-059719   xxxxxxxxxxxxxxx      QFX5200-32C-32Q
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           32X40G/32X100G-QSFP
    Xcvr 0                NON-JNPR                       UNKNOWN
    Xcvr 1                NON-JNPR                       UNKNOWN
    Xcvr 2                NON-JNPR                       UNKNOWN
    Xcvr 3                NON-JNPR                       UNKNOWN
    Xcvr 4                NON-JNPR                       UNKNOWN
    Xcvr 5                NON-JNPR                       UNKNOWN
    Xcvr 6                NON-JNPR                       UNKNOWN
    Xcvr 8                NON-JNPR                       UNKNOWN
    Xcvr 9                NON-JNPR                       UNKNOWN
    Xcvr 10               NON-JNPR                       UNKNOWN
    Xcvr 11               NON-JNPR                       UNKNOWN
    Xcvr 12               NON-JNPR                       UNKNOWN
    Xcvr 13               NON-JNPR                       UNKNOWN
    Xcvr 14               NON-JNPR                       UNKNOWN
    Xcvr 15               NON-JNPR                       UNKNOWN


i2c errors are transient in nature and the recommendation is to power cycle the device and check if the issue is resolved. 
If the errors are still seen then the issue might be permanent hardware failure and the device needs to be replaced with an RMA. Please open a JTAC case if the issue is seen even after power-draining the device.

 

Modification History

2023-09-07 :- version 1