Description

In Junos fusion setup with cascade ports could raise Satellite offline alarm for specific slot due to physical interface 
issue or satellite device issue

Symptoms

The following log messages are seen on the Flexible PIC Concentrator (FPC)

Scenario 1 : During the issue time flap could be observed for the cascade port
Apr 12 02:17:25 spmd[34342]: %DAEMON-4-EP_ALARM: FPC 103: Alarm set: color=RED, class=CHASSIS, reason=Slot 103: Satellite offline Apr 12 02:17:25 spmd[34342]: %DAEMON-4-EP_ALARM: FPC 103: Alarm set: color=RED, class=ETHER, reason=et-103/1/0: Link down The below command output will show the physical port which is connected to the satellite device

 

root@RE0> show chassis satellite

Device    Cascade   Port   Extended Ports

Alias     Slot    State       Ports   State  Total/Up

EX4300T01-HH1 103 Online    et-0/0/2  online  50/1

 

root@RE0> show chassis satellite neighbor

Interface  State     Port Info  System Name Model          SW Version

et-0/0/2   Two-Way   et-0/1/0   sd103       EX4300-48T     20.4R3-S5.4

et-0/0/1   Two-Way   et-0/1/0   sd102       EX4300-32F     20.4R3-S5.4

et-0/0/0   Two-Way   et-0/0/96  sd101       QFX5100-96S-8Q 20.4R3-S5.4

 

 

 

Scenario 2:Remote satellite down / Hang

Feb 9 05:48:35 Lab_MX960 spmd[20060]: %DAEMON-4-EP_ALARM: FPC 101: Alarm set: color=RED, class=CHASSIS, reason=Slot101: Satellite offline 
Feb 9 05:48:35 Lab_MX960 spmd[20060]: %DAEMON-4-EP_ALARM: FPC 101: Alarm set: color=RED, class=ETHER, reason=xe-101/0/32: Link down

Also, the physical interface connected to the satellite is set with receive power low alarm.

Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 0 SFP receive power low alarm set 
Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 0 SFP receive power low warning set 
Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 1 SFP receive power low alarm set 
Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 1 SFP receive power low warning set

Solution

Scenario 1 :
In this case, there was flap observed for the Physical port connected to EX4300T01 raising alarm for FPC103

Apr 12 20:46:23 mib2d[34354]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 586, ifAdminStatus up(1), ifOperStatus up(1), ifName et-0/0/2

Apr 12 20:58:33 mib2d[34354]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 586, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/2

Apr 12 20:58:38 mib2d[34354]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 586, ifAdminStatus up(1), ifOperStatus up(1), ifName et-0/0/2

Apr 12 21:16:30 mib2d[34354]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 586, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/2

 

With Local link Delta fault - 

Apr 13 02:04:17 : %PFE-6: fpc0 MQSS(0): CHMAC0: Detected Ethernet MAC Local Fault Delta Event for Port 8 (et-0/0/2) 

Apr 13 02:04:20 : %PFE-6: fpc0 MQSS(0): CHMAC0: Cleared Ethernet MAC Local Fault Delta Event for Port 8 (et-0/0/2) 

 

Please perform the below TS to resolve the issue

- Change the cable and SFP on both side

- Perform a port switch to a different port with config to isolate possible port issue with Aggregation or Satellite device

- Collect the below logs from Aggregation device in case of any FPC issue  

 

 

Scenario 2:

The satellite device was down / Hang .

Feb 9 05:48:35 Lab_MX960 spmd[20060]: %DAEMON-5-SPMD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 107, jnxFruL1Index 102, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName SD101 FPC: EX4300-32F @ 101/*/*, jnxFruType 3, jnxFruSlot 101, jnxFruOfflineReason 1, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Feb 9 05:48:35 Lab_MX960 spmd[20060]: %DAEMON-4-EP_ALARM: FPC 101: Alarm set: color=RED, class=CHASSIS, reason=Slot101: Satellite offline 
Feb 9 05:48:35 Lab_MX960 spmd[20060]: %DAEMON-4-EP_ALARM: FPC 101: Alarm set: color=RED, class=ETHER, reason=xe-101/0/32: Link down 
Feb 9 05:48:35 Lab_MX960 : %PFE-4: fpc2 user.warning aftd-trio: [Warn] AM : IPC handling - No handler found for type:27 subtype:9
Feb 9 05:48:35 Lab_MX960 : %PFE-1: fpc2 user.alert aftd-trio: [Alert] IF:Ifd new linkState, ifdName: xe-101/0/32 ifdIndex: 316 state:Down
Feb 9 05:48:35 Lab_MX960 : %PFE-1: fpc2 user.alert aftd-trio: [Alert] IF:Ifd new linkState, ifdName: xe-101/0/33 ifdIndex: 317 state:Down

Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 0 SFP receive power low alarm set 
Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 0 SFP receive power low warning set 
Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 1 SFP receive power low alarm set 
Feb 9 05:51:05 Lab_MX960 : %PFE-5: fpc3 MIC(3/0) link 1 SFP receive power low warning set

 

In this situation, please verify 

- Please verify the satellite switch status, power, SFP on both sides, and the cabling.

- Check if able to console into the device satellite switch.

- Please try to power cycle the satellite switch and verify the status.

 

 

Additional logs can be collected:

start shell pfe network fpcX

Set syslog tty disable

show syslog messages

show qsfp list   <<< pick the optics index from this output

show qsfp # info   <<< replace # with index no

show qsfp # alarms

show qsfp # diagnostics

Contact Support to TS if any further assistance is needed.

Modification History

2024-04-30 : Article Created

2025-02-11 : Article updated