Description

This article will explain the isolation & remediation of "SIB Link Error" on QFX10008 & QFX10016 devices.

Symptoms

The below-mentioned alarm would be seen.

QFX10008> show chassis alarms
7 alarms currently active
Alarm time Class Description
2024-05-10 17:32:03 UTC Minor SIB 2 FPC Link Error
2024-05-10 17:31:44 UTC Minor FPC 0 SIB Link Error
2024-05-10 17:31:43 UTC Minor FPC 3 SIB Link Error
2024-05-10 17:31:42 UTC Minor FPC 1 SIB Link Error
2024-05-10 17:31:41 UTC Minor FPC 4 SIB Link Error
2024-05-10 17:31:40 UTC Minor FPC 5 SIB Link Error
2024-05-10 17:31:39 UTC Minor FPC 2 SIB Link Error
 

Check RSI


############################ Fabric topology ############################
~
SIB 2 FCHIP 0 FCORE 0 :
-----------------------
          In-links               State             Out-links              State
--------------------------------------------------------------------------------
FPC00FE0(0,05)->S02F0_0(01,0,01) Error   S02F0_0(00,0,00)->FPC00FE0(0,17) Error
FPC00FE0(0,17)->S02F0_0(02,0,02) Error   S02F0_0(00,1,00)->FPC00FE0(0,05) Error
FPC00FE0(0,15)->S02F0_0(02,2,02) Error   S02F0_0(00,2,00)->FPC00FE0(0,15) Error
FPC00FE1(0,12)->S02F0_0(01,1,01) Error   S02F0_0(00,3,00)->FPC00FE1(0,14) Error
FPC00FE1(0,14)->S02F0_0(01,2,01) Error   S02F0_0(01,1,01)->FPC00FE1(0,12) Error
FPC00FE1(0,10)->S02F0_0(01,3,01) Error   S02F0_0(01,3,01)->FPC00FE1(0,10) Error
FPC00FE2(0,04)->S02F0_0(00,4,00) Error   S02F0_0(00,4,00)->FPC00FE2(0,16) Error
FPC00FE2(0,16)->S02F0_0(01,6,01) Error   S02F0_0(00,5,00)->FPC00FE2(0,04) Error
FPC00FE2(0,14)->S02F0_0(01,7,01) Error   S02F0_0(00,6,00)->FPC00FE2(0,14) Error
FPC01FE0(0,15)->S02F0_0(16,1,16) Error   S02F0_0(15,6,15)->FPC01FE0(0,05) Error
FPC01FE0(0,17)->S02F0_0(16,3,16) Error   S02F0_0(17,0,17)->FPC01FE0(0,15) Error
FPC01FE0(0,05)->S02F0_0(19,7,19) Error   S02F0_0(17,2,17)->FPC01FE0(0,17) Error
FPC01FE1(0,10)->S02F0_0(19,4,19) Error   S02F0_0(15,4,15)->FPC01FE1(0,14) Error
FPC01FE1(0,14)->S02F0_0(19,5,19) Error   S02F0_0(15,5,15)->FPC01FE1(0,10) Error
FPC01FE1(0,12)->S02F0_0(19,6,19) Error   S02F0_0(15,7,15)->FPC01FE1(0,12) Error
FPC01FE2(0,04)->S02F0_0(15,3,15) Error   S02F0_0(15,2,15)->FPC01FE2(0,04) Error
FPC01FE2(0,14)->S02F0_0(16,5,16) Error   S02F0_0(16,4,16)->FPC01FE2(0,14) Error
FPC01FE2(0,16)->S02F0_0(16,6,16) Error   S02F0_0(16,7,16)->FPC01FE2(0,16) Error
FPC02FE0(0,05)->S02F0_0(16,2,16) Error   S02F0_0(17,3,17)->FPC02FE0(0,05) Error
FPC02FE0(0,15)->S02F0_0(17,4,17) Error   S02F0_0(18,4,18)->FPC02FE0(0,15) Error
FPC02FE0(0,17)->S02F0_0(17,6,17) Error   S02F0_0(18,6,18)->FPC02FE0(0,17) Error
FPC02FE1(0,14)->S02F0_0(16,0,16) Error   S02F0_0(17,1,17)->FPC02FE1(0,14) Error
FPC02FE1(0,10)->S02F0_0(17,5,17) Error   S02F0_0(18,5,18)->FPC02FE1(0,10) Error
FPC02FE1(0,12)->S02F0_0(17,7,17) Error   S02F0_0(18,7,18)->FPC02FE1(0,12) Error
FPC02FE2(0,04)->S02F0_0(16,7,16) Error   S02F0_0(16,6,16)->FPC02FE2(0,04) Error
FPC02FE2(0,14)->S02F0_0(18,0,18) Error   S02F0_0(18,0,18)->FPC02FE2(0,14) Error
FPC02FE2(0,16)->S02F0_0(18,2,18) Error   S02F0_0(18,2,18)->FPC02FE2(0,16) Error
FPC03FE0(0,17)->S02F0_0(13,1,13) Error   S02F0_0(12,0,12)->FPC03FE0(0,17) Error
FPC03FE0(0,15)->S02F0_0(13,3,13) Error   S02F0_0(12,2,12)->FPC03FE0(0,15) Error
FPC03FE0(0,05)->S02F0_0(14,3,14) Error   S02F0_0(13,4,13)->FPC03FE0(0,05) Error
FPC03FE1(0,10)->S02F0_0(14,0,14) Error   S02F0_0(13,5,13)->FPC03FE1(0,12) Error
FPC03FE1(0,14)->S02F0_0(14,1,14) Error   S02F0_0(13,6,13)->FPC03FE1(0,14) Error
FPC03FE1(0,12)->S02F0_0(14,2,14) Error   S02F0_0(13,7,13)->FPC03FE1(0,10) Error
FPC03FE2(0,16)->S02F0_0(12,5,12) Error   S02F0_0(12,4,12)->FPC03FE2(0,16) Error
FPC03FE2(0,14)->S02F0_0(12,7,12) Error   S02F0_0(12,6,12)->FPC03FE2(0,14) Error
FPC03FE2(0,04)->S02F0_0(14,7,14) Error   S02F0_0(14,6,14)->FPC03FE2(0,04) Error
FPC04FE0(0,17)->S02F0_0(11,0,11) Error   S02F0_0(10,0,10)->FPC04FE0(0,17) Error
FPC04FE0(0,15)->S02F0_0(11,2,11) Error   S02F0_0(10,2,10)->FPC04FE0(0,15) Error
FPC04FE0(0,05)->S02F0_0(13,0,13) Error   S02F0_0(12,1,12)->FPC04FE0(0,05) Error
FPC04FE1(0,12)->S02F0_0(11,1,11) Error   S02F0_0(10,1,10)->FPC04FE1(0,12) Error
FPC04FE1(0,10)->S02F0_0(11,3,11) Error   S02F0_0(10,3,10)->FPC04FE1(0,10) Error
FPC04FE1(0,14)->S02F0_0(13,2,13) Error   S02F0_0(12,3,12)->FPC04FE1(0,14) Error
FPC04FE2(0,16)->S02F0_0(10,4,10) Error   S02F0_0(10,4,10)->FPC04FE2(0,16) Error
FPC04FE2(0,14)->S02F0_0(10,6,10) Error   S02F0_0(10,6,10)->FPC04FE2(0,14) Error
FPC04FE2(0,04)->S02F0_0(12,4,12) Error   S02F0_0(12,5,12)->FPC04FE2(0,04) Error
FPC05FE0(0,15)->S02F0_0(05,5,05) Error   S02F0_0(05,2,05)->FPC05FE0(0,05) Error
FPC05FE0(0,17)->S02F0_0(05,7,05) Error   S02F0_0(06,4,06)->FPC05FE0(0,15) Error
FPC05FE0(0,05)->S02F0_0(09,3,09) Error   S02F0_0(06,7,06)->FPC05FE0(0,17) Error
FPC05FE1(0,10)->S02F0_0(09,0,09) Error   S02F0_0(05,0,05)->FPC05FE1(0,14) Error
FPC05FE1(0,14)->S02F0_0(09,1,09) Error   S02F0_0(05,1,05)->FPC05FE1(0,10) Error
FPC05FE1(0,12)->S02F0_0(09,2,09) Error   S02F0_0(05,3,05)->FPC05FE1(0,12) Error
FPC05FE2(0,14)->S02F0_0(06,1,06) Error   S02F0_0(06,0,06)->FPC05FE2(0,14) Error
FPC05FE2(0,16)->S02F0_0(06,3,06) Error   S02F0_0(06,2,06)->FPC05FE2(0,16) Error
FPC05FE2(0,04)->S02F0_0(09,7,09) Error   S02F0_0(09,6,09)->FPC05FE2(0,04) Error

SIB 2 FCHIP 1 FCORE 0 :
-----------------------
          In-links               State             Out-links              State
--------------------------------------------------------------------------------
FPC00FE0(0,09)->S02F1_0(15,4,15) Error   S02F1_0(16,4,16)->FPC00FE0(0,09) Error
FPC00FE0(0,11)->S02F1_0(17,4,17) Error   S02F1_0(18,4,18)->FPC00FE0(0,11) Error
FPC00FE0(0,13)->S02F1_0(17,6,17) Error   S02F1_0(18,6,18)->FPC00FE0(0,13) Error
FPC00FE1(0,16)->S02F1_0(15,6,15) Error   S02F1_0(16,6,16)->FPC00FE1(0,16) Error
FPC00FE1(0,08)->S02F1_0(17,5,17) Error   S02F1_0(18,5,18)->FPC00FE1(0,08) Error
FPC00FE1(0,04)->S02F1_0(17,7,17) Error   S02F1_0(18,7,18)->FPC00FE1(0,04) Error
FPC00FE2(0,08)->S02F1_0(16,0,16) Error   S02F1_0(16,0,16)->FPC00FE2(0,08) Error
FPC00FE2(0,10)->S02F1_0(18,0,18) Error   S02F1_0(18,0,18)->FPC00FE2(0,10) Error
FPC00FE2(0,12)->S02F1_0(18,2,18) Error   S02F1_0(18,2,18)->FPC00FE2(0,12) Error
FPC01FE0(0,13)->S02F1_0(13,1,13) Error   S02F1_0(12,0,12)->FPC01FE0(0,13) Error
FPC01FE0(0,11)->S02F1_0(13,3,13) Error   S02F1_0(12,2,12)->FPC01FE0(0,11) Error
FPC01FE0(0,09)->S02F1_0(14,1,14) Error   S02F1_0(13,6,13)->FPC01FE0(0,09) Error
FPC01FE1(0,08)->S02F1_0(14,0,14) Error   S02F1_0(13,4,13)->FPC01FE1(0,16) Error
FPC01FE1(0,04)->S02F1_0(14,2,14) Error   S02F1_0(13,5,13)->FPC01FE1(0,04) Error
FPC01FE1(0,16)->S02F1_0(14,3,14) Error   S02F1_0(13,7,13)->FPC01FE1(0,08) Error
FPC01FE2(0,12)->S02F1_0(12,5,12) Error   S02F1_0(12,4,12)->FPC01FE2(0,12) Error
FPC01FE2(0,10)->S02F1_0(12,7,12) Error   S02F1_0(12,6,12)->FPC01FE2(0,10) Error
FPC01FE2(0,08)->S02F1_0(14,5,14) Error   S02F1_0(14,4,14)->FPC01FE2(0,08) Error
FPC02FE0(0,13)->S02F1_0(11,0,11) Error   S02F1_0(10,0,10)->FPC02FE0(0,13) Error
FPC02FE0(0,11)->S02F1_0(11,2,11) Error   S02F1_0(10,2,10)->FPC02FE0(0,11) Error
FPC02FE0(0,09)->S02F1_0(13,2,13) Error   S02F1_0(12,3,12)->FPC02FE0(0,09) Error
FPC02FE1(0,04)->S02F1_0(11,1,11) Error   S02F1_0(10,1,10)->FPC02FE1(0,04) Error
FPC02FE1(0,08)->S02F1_0(11,3,11) Error   S02F1_0(10,3,10)->FPC02FE1(0,08) Error
FPC02FE1(0,16)->S02F1_0(13,0,13) Error   S02F1_0(12,1,12)->FPC02FE1(0,16) Error
FPC02FE2(0,12)->S02F1_0(10,4,10) Error   S02F1_0(10,4,10)->FPC02FE2(0,12) Error
FPC02FE2(0,10)->S02F1_0(10,6,10) Error   S02F1_0(10,6,10)->FPC02FE2(0,10) Error
FPC02FE2(0,08)->S02F1_0(12,6,12) Error   S02F1_0(12,7,12)->FPC02FE2(0,08) Error
FPC03FE0(0,11)->S02F1_0(05,5,05) Error   S02F1_0(05,0,05)->FPC03FE0(0,09) Error
FPC03FE0(0,13)->S02F1_0(05,7,05) Error   S02F1_0(06,5,06)->FPC03FE0(0,11) Error
FPC03FE0(0,09)->S02F1_0(09,1,09) Error   S02F1_0(06,7,06)->FPC03FE0(0,13) Error
FPC03FE1(0,08)->S02F1_0(09,0,09) Error   S02F1_0(05,1,05)->FPC03FE1(0,08) Error
FPC03FE1(0,04)->S02F1_0(09,2,09) Error   S02F1_0(05,2,05)->FPC03FE1(0,16) Error
FPC03FE1(0,16)->S02F1_0(09,3,09) Error   S02F1_0(05,3,05)->FPC03FE1(0,04) Error
FPC03FE2(0,10)->S02F1_0(06,1,06) Error   S02F1_0(06,1,06)->FPC03FE2(0,10) Error
FPC03FE2(0,12)->S02F1_0(06,3,06) Error   S02F1_0(06,3,06)->FPC03FE2(0,12) Error
FPC03FE2(0,08)->S02F1_0(09,5,09) Error   S02F1_0(09,4,09)->FPC03FE2(0,08) Error
FPC04FE0(0,09)->S02F1_0(05,4,05) Error   S02F1_0(06,4,06)->FPC04FE0(0,09) Error
FPC04FE0(0,11)->S02F1_0(07,0,07) Error   S02F1_0(08,0,08)->FPC04FE0(0,11) Error
FPC04FE0(0,13)->S02F1_0(07,2,07) Error   S02F1_0(08,2,08)->FPC04FE0(0,13) Error
FPC04FE1(0,16)->S02F1_0(05,6,05) Error   S02F1_0(06,6,06)->FPC04FE1(0,16) Error
FPC04FE1(0,04)->S02F1_0(08,1,08) Error   S02F1_0(08,1,08)->FPC04FE1(0,08) Error
FPC04FE1(0,08)->S02F1_0(08,3,08) Error   S02F1_0(08,3,08)->FPC04FE1(0,04) Error
FPC04FE2(0,08)->S02F1_0(06,0,06) Error   S02F1_0(06,0,06)->FPC04FE2(0,08) Error
FPC04FE2(0,10)->S02F1_0(07,4,07) Error   S02F1_0(07,4,07)->FPC04FE2(0,10) Error
FPC04FE2(0,12)->S02F1_0(07,6,07) Error   S02F1_0(07,6,07)->FPC04FE2(0,12) Error
FPC05FE0(0,11)->S02F1_0(02,0,02) Error   S02F1_0(02,1,02)->FPC05FE0(0,13) Error
FPC05FE0(0,13)->S02F1_0(02,2,02) Error   S02F1_0(02,3,02)->FPC05FE0(0,11) Error
FPC05FE0(0,09)->S02F1_0(04,7,04) Error   S02F1_0(03,6,03)->FPC05FE0(0,09) Error
FPC05FE1(0,04)->S02F1_0(04,4,04) Error   S02F1_0(03,4,03)->FPC05FE1(0,16) Error
FPC05FE1(0,16)->S02F1_0(04,5,04) Error   S02F1_0(03,5,03)->FPC05FE1(0,04) Error
FPC05FE1(0,08)->S02F1_0(04,6,04) Error   S02F1_0(03,7,03)->FPC05FE1(0,08) Error
FPC05FE2(0,10)->S02F1_0(02,4,02) Error   S02F1_0(02,5,02)->FPC05FE2(0,12) Error
FPC05FE2(0,12)->S02F1_0(02,6,02) Error   S02F1_0(02,7,02)->FPC05FE2(0,10) Error
FPC05FE2(0,08)->S02F1_0(04,3,04) Error   S02F1_0(04,2,04)->FPC05FE2(0,08) Error
~

############################ Fabric summary ############################
FRU         State          Errors

SIB0        Online         None
SIB1        Online         None
SIB2        Online         Link Errors
SIB3        Online         None
SIB4        Online         None
SIB5        Online         None

FPC0        Online         Link Errors
FPC1        Online         Link Errors
FPC2        Online         Link Errors
FPC3        Online         Link Errors
FPC4        Online         Link Errors
FPC5        Online         Link Errors
FPC6        Empty
FPC7        Empty

############################ Fabric sibs ############################
Fabric management SIB state:
~
SIB #2  Online
    FASIC #0 (plane 4) Active
    FPC #0
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #1
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #2
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #3
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #4
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #5
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error

    FASIC #1 (plane 5) Active
    FPC #0
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #1
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #2
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #3
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #4
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
    FPC #5
        PFE #0  : Error
        PFE #1  : Error
        PFE #2  : Error
~
############################ Fabric fpc ############################
Fabric management FPC state:
FPC #0

    PFE #0
    ~
    SIB2_FASIC0 (plane  4)   Plane Disabled, Links Error
    SIB2_FASIC1 (plane  5)   Plane Disabled, Links Error
    ~
    PFE #1
    ~
    SIB2_FASIC0 (plane  4)   Plane Disabled, Links Error
    SIB2_FASIC1 (plane  5)   Plane Disabled, Links Error
    ~

Same Plane Disabled/Links Error would be seen for each FPC with SIB2

Solution

  • Looking at the above-mentioned logs confirms that "SIB Link Error " is seen with all FPC which indicates that the issue would mostly be with SIB here & not on the FPC end.

    • There is also a SIB 2 indicating "FPC Link Error" errors.

  • After reviewing the Fabric topology, Fabric summary, Fabric sibs & Fabric fpc outputs from RSI, we can see the issue mostly looks to be with SIB 2.

  • Solution:

    1. Offline & online via a Junos CLI

      • > request chassis sib slot <> offline

      • > request chassis sib slot <> online

    2. Try physical re-seating of the impacted SIB

    3. If re-seating is not helping, reach JTAC support

      • JTAC would review & move ahead with the RMA of SIB

Modification History

2024-05-17 : Article Created