Description

This article explains the meaning of the ISR 2 (MIC error interrupt) messages, giving the reason that they are reported and suggesting the best way to deal with them. 

Nov 17 18:05:16 MX104 afeb0  mic_periodic : fpc 0 mic slot 0 phy interrupt count  125
Nov 17 18:13:36 MX104 afeb0  mic_periodic : fpc 0 mic slot 1 phy interrupt count  4
Nov 17 18:21:56 MX104 afeb0  mic_periodic : fpc 0 mic slot 0 phy interrupt count  125
Nov 17 18:30:16 MX104 afeb0  mic_periodic : fpc 0 mic slot 1 phy interrupt count  4

 

Symptoms

The ISR2 (MIC error interrupt) messages can mostly be seen after insertion of NON-Juniper qualified SFPs or malfunctioning SFPs (Juniper qualified). The MIC requests information from SFPs via the I2C bus, and if an EEPROM on an SFP in the MIC does not respond to the I2C request, the I2C bus transaction will fail, and MIC error interrupt message will be generated. And the messages will stop only if the NON-Juniper SFPs or malfunctioning SFPs are replaced with healthy Juniper-qualified SFPs.

We can identify the SFP that is causing these error log messages by checking the following outputs: 

User@MX> show chassis hardware
FPC X                     BUILTIN      BUILTIN        MPC BUILTIN
  MIC 0          REV 27   750-028392   xxxxxxx        3D 20x 1GE(LAN) SFP
    PIC 0                 BUILTIN      BUILTIN        10x 1GE(LAN) SFP
      Xcvr 3              NON-JNPR   xxxxxxx        SFP-T
    PIC 1                 BUILTIN      BUILTIN        10x 1GE(LAN) SFP


User@MX>start shell pfe network fpcX

vty# show sfp list
vty# show sfp <index from show sfp list> info

vty# show sfp 3 info
  index:                   0x03
  sfp name:                MIC(0/1)
  pic context:             0x4CD44248
  id mem scanned:          false
  linkstate:               Up
  sfp_present:             false
  sfp_changed:             false
  i2c failure count:       0x3C <<<--- non-zero i2c errors
  diag polling count:      0x8D42
  no diag polling from RE: 0x1
  run_periodic:            false
 

Check the i2c failure count on each SFP. If any SFP shows a non-zero i2c failure count, have that specific SFP replaced.

 

Solution

Using NON-Juniper SFPs is not recommended because it can cause the following ISR2 interrupt related issues.

  • PR1235475 : When a NON-Juniper SFP is used in MIC-3D-20GE-SFP-E or MIC-3D-20GE-SFP-EH MIC, the ISR 2 interrupt might exceed 2.5 seconds due to unknown reasons, and then the FPC hosting the MIC might be restarted and crashed with core dump. PR1235475 added an interrupt throttling mechanism for these ISR2 interrupts. If the interrupts are more than the threshold (> 2500 per 5 mins), the MIC is restarted.

  • PR1270420 : PR1235475 implemented an interrupt threshold on MIC-3D-20GE-SFP-E and MIC-3D-20GE-SFP-EH. If the MIC error interrupts are more than the threshold (> 2500 per 5 mins), the MIC is restarted. However, the fix of PR1235475 was not complete to avoid the MPC crash due to ISR2 interrupts in some conditions. As a result, MIC error interrupts may excessively hog the CPU when restart is initiated. Moreover, after fixing PR1235475, although the interrupt count is non-zero and not increasing, the following message with severity ERROR is reported on periodic basis: 

Jul 19 10:26:18 MX240 : %PFE-3: fpc1  mic_periodic : fpc 1 mic slot 1 phy interrupt count  52
Jul 19 10:34:38 MX240 : %PFE-3: fpc1  mic_periodic : fpc 1 mic slot 1 phy interrupt count  52
Jul 19 10:42:58 MX240 : %PFE-3: fpc1  mic_periodic : fpc 1 mic slot 1 phy interrupt count  52

PR1270420 also changed the log severity for the message from ERROR to lower (info or debug) if the interrupt count is not increasing.

  • PR1302246 : PR1235475 and PR1270420 added a mechanism to bounce the MIC when the interrupt rate crosses the threshold. However, it was noted that the threshold (> 2500 per 5 min) was fairly easy to cross. PR1302246 therefore implemented a hidden knob to prevent interrupts from bouncing the MIC. 

[edit]
User@MX# show |compare
[edit chassis]
+   fpc 1 {
+       pic 0 {
+           interrupt-threshold 25000;
+       }
+   }

  • PR1348107 : MIC-3D-20GE-SFP-E might crash due to ISR 2 interrupt hogging when SFP-T optics is installed.

  • PR1260517 ( PR1246209 ): On MIC-3D-20GE-SFP-E or MIC-3D-20GE-SFP, when SFP diagnostic information is being read out periodically, due to a malfunctioning SFP or I2C bus noise, the SFP thread might hog CPU resources, and a watchdog check restarts the MPC to recover. PR1260517 prevents the SFP thread hogging and the MPC restart.

 

The ISR2 interrupt messages can be seen with NON-Juniper SFPs or malfunctioning SFPs. If the number of these messages is not increasing, there should not be any issue and the messages can be ignored as being harmless. They just notify that interrupts were seen on the MIC.

However, if they are caused by a real defective hardware, the interrupt count is likely to keep increasing. In such case, we have to figure out the source of the interrupts, and replace it to avoid service impact, such as interfaces going down or software crashes.

In general, however, using NON-Juniper SFPs is not recommended because we might encounter multiple ISR2 interrupt related issues.

 

Related Information