This article outlines the troubleshooting steps you should take when all FEBs go down (and, as a result, all FPCs also go down).

Under some circumstances, the alarms below are active:

lab@R1> show chassis alarms

6 alarms currently active
Alarm time Class Description
2012-01-17 16:18:10 AST Major FEB 4 Failure
2012-01-17 16:18:08 AST Major FEB 3 Failure
2012-01-17 16:18:05 AST Major FEB 2 Failure
2012-01-17 16:18:02 AST Major FPC 4 Hard errors
2012-01-17 16:17:59 AST Major FPC 3 Hard errors
2012-01-17 16:17:56 AST Major FPC 2 Hard errors

These alarms indicate that all FEBs are down.

In a fully populated chassis, all the FEBs for all FPCs must function, because FPC-to-FEB mapping is one-to-one, by default (for example, FPC1 maps to FEB1).

When the above alarms are active, the logs below are to be expected:

Jan 17 16:08:01 CHASSISD_FRU_UNRESPONSIVE_RETRY: Attempt 1 to power on FPC 2 timed out; restarted it
Jan 17 16:08:01 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 2 offline: Restarting unresponsive board
Jan 17 16:08:01 I2CS write cmd to FPM#0 [0x0], reg 0x4a, cmd 0x0
Jan 17 16:08:01 I2CS write cmd to FPM#0 [0x0], reg 0x49, cmd 0x1c
Jan 17 16:08:01 I2CS write cmd to FPM#0 [0x0], reg 0x4b, cmd 0x1c
Jan 17 16:08:01 fpc_down slot 2 reason Restarting unresponsive board cargs 0x0
Jan 17 16:08:01 fpc_offline_now - slot 2, reason: Restarting unresponsive board, error OK
Jan 17 16:08:01 I2CS write cmd to FPC#2 [0xc], reg 0x20, cmd 0x0
Jan 17 16:08:01 FPC#2 - power off [addr 0xc] reason: Restarting unresponsive board

To solve this issue, follow the steps below.

Step 1: Collect the information below from both RE0 and RE1 in a redundant RE environment.

lab@R1_RE0> show tnp addresses
Name TNPaddr MAC address IF MTU E H R
master 0x1 02:00:00:00:00:04 em0 1500 0 0 3
master 0x1 02:00:01:00:00:04 em1 1500 0 1 3
re0 0x4 02:00:00:00:00:04 em0 1500 0 0 3
re0 0x4 02:00:01:00:00:04 em1 1500 0 1 3
re1 0x5 02:01:01:00:00:05 em1 1500 2 1 3
backup 0x6 02:01:01:00:00:05 em1 1500 2 1 3
bcast 0xffffffff ff:ff:ff:ff:ff:ff em0 1500 0 0 3
bcast 0xffffffff ff:ff:ff:ff:ff:ff em1 1500 0 1 3

Step 2: Switchover mastership and collect the same logs from RE1 as shown below.

lab@R1-RE1> show tnp addresses
Name TNPaddr MAC address IF MTU E H R
master 0x1 02:01:00:00:00:05 em0 1500 0 0 3
master 0x1 02:01:01:00:00:05 em1 1500 0 1 3
re0 0x4 02:00:01:00:00:04 em1 1500 2 1 3
re1 0x5 02:01:00:00:00:05 em0 1500 0 0 3
re1 0x5 02:01:01:00:00:05 em1 1500 0 1 3
backup 0x6 02:00:01:00:00:04 em1 1500 2 1 3
fpc2 0x12 02:00:00:00:00:12 em0 1500 4 0 3
fpc3 0x13 02:00:00:00:00:13 em0 1500 4 0 3
fpc4 0x14 02:00:00:00:00:14 em0 1500 4 0 3
feb2 0x22 02:00:00:00:00:22 em0 1500 4 0 3
feb3 0x23 02:00:00:00:00:23 em0 1500 5 0 3
feb4 0x24 02:00:00:00:00:24 em0 1500 4 0 3
bcast 0xffffffff ff:ff:ff:ff:ff:ff em0 1500 0 0 3
bcast 0xffffffff ff:ff:ff:ff:ff:ff em1 1500 0 1 3

Step 3: Collect the information below.

lab@R1_RE0> show interfaces em0 extensive | no-more
Physical interface: em0, Enabled, Physical link is Down
Interface index: 8, SNMP ifIndex: 17, Generation: 2
Type: Ethernet, Link-level type: Ethernet, MTU: 1514, Clocking: Unspecified,
Speed: 1000mbps
Device flags : Present Running No-Carrier
Interface flags: Hardware-Down SNMP-Traps
Link type : Full-Duplex
Physical info : Unspecified
Hold-times : Up 0 ms, Down 0 ms
Current address: 02:00:00:00:00:04, Hardware address: 02:00:00:00:00:04
Alternate link address: Unspecified

Explanation

As shown above, the em0 interface shows down. This would prevent communication from the RE/CB to the FRUs installed in the chassis. Thus, these FRUs would be considered down, creating the alarms.

To fix the situation, try to reseat/reboot the RE where the em0 interface shows down. This should solve the problem.

If the problem comes back or if reseating/rebooting does not solve the issue, replace the RE. This should permanently fix the problem.