Description

This article explains how to find the SFP for which i2c poll is failing.

Symptoms

Syslog messages as below seen on the device: 

Mar 16 02:43:25 2021  re0.test fpc1 PQ3_IIC(WR): I/O error (i2c_stat=0xa3, i2c_ctl[1]=0xb0, bus_addr=0x51) 
Mar 16 02:43:25 2021  re0.test fpc1 I2C Failed device: group 0xa0 address 0x51 
Mar 16 02:43:25 2021  re0.test fpc1 PQ3_IIC(WR): no target ack on byte 0 (wait spins 2) 

Solution

These messages are seen when SFPs are being inserted or taken out of the port. In this scenario, you can ignore the messages if the timestamp correlates to the SFPs in/out timestamp.

If you the messages are scrolling continuously, it indicates there is a component for which the chassis is not able to read the i2c values . The i2c (inter-integrated circuits) is a serial communication mechanism in which states for various components are polled at regular intervals by the chassis.

When these messages are seen only for a specific FPC (fpc 1 in this example), you can narrow down the component as below: 
  1. Log into the FPC shell: 

    start shell pfe network fpc 1
  2. It is common to see a lot of messages scrolling on the FPC shell. If so, disable logging for this FPC telnet session: 

    set syslog tty disable
  3. Check the syslog message on the FPC:

    [Mar 16 10:41:53.287 LOG: Err] I2C Failed device: group 0xa0 address 0x51
    [Mar 16 10:41:53.288 LOG: Debug] mic_sfp_id_read(): Failed to read SFP in link 0 , loc 96
    This is a hint that the chassis failed to read the SFP for link 0.
  4. Narrow down the SFP for which the i2c polls are failing by checking the sfp list. The command, 'show sfp list' returns the SFP index numbers which come in handy for the next set of commands:

    show sfp list
    SFP Toolkit summary:
      wakeup count: 6725, debug: 0, periodic enabled: 1, diagnostics enabled: 1
      thread: 0x470f5ba8, itable: 0x460eb438, itable iterator: 0, sem: 0x460f3d18
      polling interval delay:  1000 ms, polling max cpu:  100 ms
      poll for diags every  3 wakeups , SFPs polled for diags last time: 24
      last periodic CPU time:  101 ms, maximum periodic CPU time:    681 ms
      max SFPs to poll per interval: 0
    SFP Toolkit syslog throttling parameters:
      period: 120 samples , disable threshold:  10, enable threshold   0
      SFP periodic diagnostics polling count: 65535
    
                                                                             diag
    Index   Name             Presence     ID Eprom  PNO         SNO          calibr   Toxic
    -----   --------------   ----------   --------  ----------  ----------- -------  -------
        1      MIC(5/0)(0)      Present   Complete  NON-JNPR  10131411125189  int    Unknown
        2      MIC(5/0)(1)      Present   Complete  NON-JNPR  10131212207686  int    Unknown
        3      MIC(5/0)(2)      Present   Complete  NON-JNPR  H53T0180        EXT**  Unknown
        4      MIC(5/0)(3)      Present   Complete  NON-JNPR  10131403042757  int    Unknown
        5      MIC(5/0)(4)      Present   Complete  NON-JNPR  10131311262033  int    Unknown
        6      MIC(5/0)(5)      Present   Complete  NON-JNPR  10131403042750  int    Unknown
        7      MIC(5/0)(6)      Present   Complete  NON-JNPR  GW1205155523    EXT**  Unknown
        8      MIC(5/0)(7)      Present   Complete  NON-JNPR  10131704125003  int    Unknown
        9      MIC(5/0)(8)      Present   Complete  NON-JNPR  10131211089186  EXT**  Unknown
       10      MIC(5/0)(9)      Present   Complete  NON-JNPR  10131212249907  int    Unknown
       11      MIC(5/1)(0)      Present   Complete  NON-JNPR  10131402172160  int    Unknown
       12      MIC(5/1)(1)      Present   Complete  NON-JNPR  H67P0120        EXT**  Unknown
       13      MIC(5/1)(2)      Present   Complete  NON-JNPR  10131409205537  int    Unknown
       14      MIC(5/1)(3)      Present   Complete  NON-JNPR  10131704205713  int    Unknown
       15      MIC(5/1)(4)      Present   Complete  NON-JNPR  10131402211281  int    Unknown
       16      MIC(5/1)(5)      Present   Complete  NON-JNPR  10131309141065  int    Unknown
       17      MIC(5/1)(6)      Present   Complete  NON-JNPR  10131208079479  EXT**  Unknown
       18      MIC(5/1)(7)      Present   Complete  NON-JNPR  10131308081039  int    Unknown
       19      MIC(5/1)(8)      Present   Complete  NON-JNPR  10131704205758  int    Unknown
       20      MIC(5/2)(0)      Present   Complete  NON-JNPR  10131402172059  int    Unknown
       21      MIC(5/2)(1)      Present   Complete  NON-JNPR  10131301029206  int    Unknown
       22      MIC(5/2)(2)      Present   Complete  NON-JNPR  10131311051807  int    Unknown
       23      MIC(5/2)(3)      Present   Complete  NON-JNPR  10131311051803  int    Unknown
       24      MIC(5/2)(4)      Present   Complete  NON-JNPR  10131211089526  EXT**  Unknown
       25      MIC(5/2)(5)      Present   Complete  NON-JNPR  10131402071658  int    Unknown
       26      MIC(5/2)(6)      Present   Complete  NON-JNPR  10131506101215  int    Unknown
       27      MIC(5/2)(8)      Present   Complete  NON-JNPR  H72P0399        EXT**  Unknown
       28      MIC(5/2)(9)      Present   Complete  NON-JNPR  10131303055537  int    Unknown
       29      MIC(5/3)(0)      Present   Complete  NON-JNPR  10131704205744  int    Unknown
       30      MIC(5/3)(1)      Present   Complete  NON-JNPR  10131402071650  int    Unknown
       31      MIC(5/3)(2)      Present   Complete  NON-JNPR  10131208079485  EXT**  Unknown
       32      MIC(5/3)(3)      Present   Complete  NON-JNPR  H67P1503        EXT**  Unknown
       33      MIC(5/3)(4)      Present   Complete  NON-JNPR  10131311051907  int    Unknown
       34      MIC(5/3)(5)      Present   Complete  NON-JNPR  10131506183659  int    Unknown
       35      MIC(5/3)(6)      Present   Complete  NON-JNPR  10131212207729  int    Unknown
       36      MIC(5/3)(7)      Present   Complete  NON-JNPR  10131311051954  int    Unknown
       37      MIC(5/3)(8)      Present   Complete  NON-JNPR  10131403042732  int    Unknown
       38      MIC(5/3)(9)      Present   Complete  NON-JNPR  10131301029081  int    Unknown
    

    The syslog on the FPC shell did indicate a failure to read the state for SFP in link 0, but did not specify the MIC.

  5. Confirm the port:

    show sfp <index> info

    show sfp 1 info
      index:                  0x01
      sfp name:               MIC(5/0)
      pic context:            0x62a344e8
      id mem scanned:         true
      linkstate:              Up
      sfp_present:            true
      sfp_changed:            false
      i2c failure count:      0x0
      diag polling count:     0x443
      no diag polling from RE:0x1
      run_periodic:           false
    show sfp 11 info
      index:                  0x0B
      sfp name:               MIC(5/1)
      pic context:            0x62c54428
      id mem scanned:         true
      linkstate:              Up
      sfp_present:            true
      sfp_changed:            false
      i2c failure count:      0x3E9   << Non-zero i2c failure count.
      diag polling count:     0x3E8
      no diag polling from RE:0x1
    
  6. Reseat the SFP on this port and monitor the log messages. There is a possibility that during a MW, the SFP on this port was not properly inserted.

If i2c failures are still seen after the SFP reseat, swap the SFP with a working SFP of the same type and monitor the status.