A control link connects two SRX Series Firewalls and sends chassis cluster control data, including heartbeats and configuration synchronization, between them. The link is a single point of failure: If the control link goes down, the secondary SRX Series is disabled from the cluster. Dual control links prevent downtime due to a single point of failure. Two control link interfaces connect each device in a cluster. Dual control links provide a redundant link for controlling traffic. Unlike dual fabric links, only one control link is used at a time.
The SRX4600, SRX5600, and SRX5800 Services Gateways support dual control links.
One of the control links is consistently down after changes like replacing SPC card in FPC0. This document notes down a generic approach to take when troubleshooting an issue with one of the Control Link is down in a SRX 5K Chassis Cluster
SRX cluster will be configured to use dual control link:
control-ports { fpc 0 port 0; fpc 12 port 0; fpc 1 port 1; fpc 13 port 1;
root@SRX> show chassis cluster interfaces Control link status: Up Control interfaces: Index Interface Monitored-Status Internal-SA Security 0 ixlv0 Down Disabled Disabled 1 igb0 Up Disabled Disabled
Physical Layer tests:
1. Replace the SFP.2. Replace the cable.3. Make sure the control link is directly connected, if a switch is used, the ports connected to the switch are up/up
SPC Level Tests:
Collect the below logs on both nodes. We need to check if heartbeats have reached the SPC3 switch and SPC3 switch has sent these heartbeats out to RE.
Working State: Port connections in SPC3 switch: Port 0 ? cpu0 port 1 ? cpu1 Port 4 ? ha 1 port 5 ? ha 0 <<< Port 14 ? scb0/re0 <<< port 15 ? scb1/re1
1. To check port status in spc3 console:
root@SRX:/var/tmp # cty -f0 fpc0 pic0 (hit enter twice) root@SRX-fpc0:~# telnet 127.0.0.1 12345 Console# show interfaces status all Dev/Port Mode Link Speed Duplex Loopback Mode --------- ---------------- ----- ----- ------ ------------- 0/0 1000_BaseX Up 1G Full None 0/1 1000_BaseX Up 1G Full None 0/2 n/a Down n/a Full N/A 0/3 n/a Down n/a Full N/A 0/4 KR Down 10G Full None 0/5 KR Up 10G Full None <<< ha port 0 0/6 n/a Down n/a Full N/A 0/7 n/a Down n/a Full N/A 0/8 n/a Down n/a Full N/A 0/9 n/a Down n/a Full N/A 0/10 n/a Down n/a Full N/A 0/11 n/a Down n/a Full N/A 0/12 n/a Down n/a Full N/A 0/13 n/a Down n/a Full N/A 0/14 1000_BaseX Up 1G Full None <<< to scb0/re0 0/15 1000_BaseX Down 1G Full None 0/32 n/a Down n/a Full N/A Console# show interfaces mac counters all Interface UC Received MC Received BC Received Octets Received --------- -------------------- -------------------- -------------------- ------------------- 0/0 1016 0 76 120240 0/1 491 0 26 43790 0/2 0 0 0 0 0/3 0 0 0 0 0/4 0 0 0 0 0/5 1019 1 12 786754 <<< 0/6 0 0 0 0 0/7 0 0 0 0 0/8 0 0 0 0 0/9 0 0 0 0 0/10 0 0 0 0 0/11 0 0 0 0 0/12 0 0 0 0 0/13 0 0 0 0 0/14 17113 32 284 1324560 <<< 0/15 0 0 0 0 0/32 0 0 0 0 0/63 0 n/a n/a 0 Type <CR> to continue, Q<CR> to stop: Interface UC Sent MC Sent BRDC Sent Octets Sent --------- -------------------- -------------------- --------------------- -------------------- 0/0 28241 38 437 13457706 0/1 19812 26 323 9413770 0/2 0 0 0 0 0/3 0 0 0 0 0/4 0 0 0 0 0/5 695 1 12 54198 <<< 0/6 0 0 0 0 0/7 0 0 0 0 0/8 0 0 0 0 0/9 0 0 0 0 0/10 0 0 0 0 0/11 0 0 0 0 0/12 0 0 0 0 0/13 0 0 0 0 0/14 32006 32 474 21298284 <<< 0/15 0 0 0 0 0/32 0 0 0 0 0/63 n/a n/a n/a n/a
2. Check HA port 0 status in the shell through i2c: 0x0c means enabled, 0x12 means disabled
root@SRX:/var/log # i2csc fpc0 0x60 fpc0 (group 0x11) Register offset Value HA0 CTL 0x60 0x0c
3. To use the VTY to check the counters. This one is cumulative.
SPC3-1(SRX vty)# show esw statistics 0 5 Good Octets Received: 1118349053 Bad Octets Received: 0 MAC Transmit Error: 0 BRDC Packets Received: 2517064 MC Packets Received: 210139 Size 64: 21934104 Size 65 to 127: 208581648 Size 128 to 255: 6006605 Size 256 to 511: 466728 Size 512 to 1023: 31806900 Size 1024 to 1518: 0 Size 1519 to Max: 0 Good Octets Sent: -1443087622 Excessive Collision: 0 MC Packets Sent: 210052 BRDC Packets Sent: 2520088 FC Sent: 0 Good FC Received: 0 Drop Events: 0 Undersize Packets: 0 Fragments Packets: 0 Oversize Packets: 0 Jabber Packets: 0 MAC RX Error Packets Received: 0 Bad CRC: 0 Collisions: 0 Late Collision: 0 FC Received: 0 Good UC Packets Received: 162446990 Good UC Packets Sent: 181035995 Multiple Packets Sent: 0 Deferred Packets Sent: 0 SPC3-1(SRX vty)# show esw statistics 0 14 Good Octets Received: 973448334 Bad Octets Received: 0 MAC Transmit Error: 0 BRDC Packets Received: 1893401 MC Packets Received: 210503 Size 64: 21971767 Size 65 to 127: 208922583 Size 128 to 255: 6018306 Size 256 to 511: 467490 Size 512 to 1023: 31854308 Size 1024 to 1518: 0 Size 1519 to Max: 0 Good Octets Sent: -1054930372 Excessive Collision: 0 MC Packets Sent: 210517 BRDC Packets Sent: 3153791 FC Sent: 0 Good FC Received: 0 Drop Events: 0 Undersize Packets: 0 Fragments Packets: 0 Oversize Packets: 0 Jabber Packets: 0 MAC RX Error Packets Received: 0 Bad CRC: 0 Collisions: 0 Late Collision: 0 FC Received: 0 Good UC Packets Received: 161633093 Good UC Packets Sent: 182395390 Multiple Packets Sent: 0 Deferred Packets Sent: 0
There's another way to read the HA link status as shown below. This one has additional port mapping.
root@SRX:~ # vty node0.fpc0 SPC platform (2300Mhz Intel(R) Xeon-Broadwell(TM) CPU processor, 3168MB memory, 8192KB flash) SPC3-0(SRX vty)# show esw ports Dev/Port Mode Link Speed Duplex Loopback Mode Port-map --------- ---------------- ----- ----- ------ ------------- --------- 0/0 1000_BaseX Up 1G Full None PMB-ETH0 0/1 1000_BaseX Up 1G Full None PMB-ETH1 0/2 n/a Down n/a Full N/A N/C 0/3 n/a Down n/a Full N/A N/C 0/4 KR Down 10G Full None HA-1 0/5 KR Up 10G Full None HA-0 0/6 n/a Down n/a Full N/A N/C 0/7 n/a Down n/a Full N/A N/C 0/8 n/a Down n/a Full N/A N/C 0/9 n/a Down n/a Full N/A N/C 0/10 n/a Down n/a Full N/A N/C 0/11 n/a Down n/a Full N/A N/C 0/12 n/a Down n/a Full N/A N/C 0/13 n/a Down n/a Full N/A N/C 0/14 1000_BaseX Up 1G Full None CB0 0/15 1000_BaseX Down 1G Full None CB1
From log we can see that fpc0 ha port 0 are enabled on both nodes (0x0c means enabled, 0x12 means disabled)
root@SRX:~ # i2csc fpc0 0x60 fpc0 (group 0x10) Register offset Value HA0 CTL 0x60 0x0c root@SRX:~ # i2csc fpc0 0x60 fpc0 (group 0x10) Register offset Value HA0 CTL 0x60 0x0c Console# show interfaces status all Dev/Port Mode Link Speed Duplex Loopback Mode^M --------- ---------------- ----- ----- ------ -------------^M ^M 0/0 1000_BaseX Up 1G Full None ^M 0/1 1000_BaseX Up 1G Full None ^M 0/2 n/a Down n/a Full N/A ^M 0/3 n/a Down n/a Full N/A ^M 0/4 KR Down 10G Full None ^M 0/5 KR Down 10G Full None ^M <<< 0/6 n/a Down n/a Full N/A ^M 0/7 n/a Down n/a Full N/A ^M 0/8 n/a Down n/a Full N/A ^M 0/9 n/a Down n/a Full N/A ^M 0/10 n/a Down n/a Full N/A ^M 0/11 n/a Down n/a Full N/A ^M 0/12 n/a Down n/a Full N/A ^M 0/13 n/a Down n/a Full N/A ^M 0/14 1000_BaseX Up 1G Full None ^M 0/15 1000_BaseX Up 1G Full None ^M 0/32 n/a Down n/a Full N/A ^M Console# show interfaces status all Dev/Port Mode Link Speed Duplex Loopback Mode^M --------- ---------------- ----- ----- ------ -------------^M 0/0 1000_BaseX Up 1G Full None ^M 0/1 1000_BaseX Up 1G Full None ^M 0/2 n/a Down n/a Full N/A ^M 0/3 n/a Down n/a Full N/A ^M 0/4 KR Down 10G Full None ^M 0/5 KR Down 10G Full None ^M <<< 0/6 n/a Down n/a Full N/A ^M 0/7 n/a Down n/a Full N/A ^M 0/8 n/a Down n/a Full N/A ^M 0/9 n/a Down n/a Full N/A ^M 0/10 n/a Down n/a Full N/A ^M 0/11 n/a Down n/a Full N/A ^M 0/12 n/a Down n/a Full N/A ^M 0/13 n/a Down n/a Full N/A ^M 0/14 1000_BaseX Up 1G Full None ^M 0/15 1000_BaseX Up 1G Full None ^M 0/32 n/a Down n/a Full N/A ^M
This seems to be caused by a connection issue.
Further tests are needed to isolate the issue further, Since currently the secondary control link is fine, we can move the SFP and cable from fpc 1 to fpc0. These tests are very specific to the outputs/test results noted in the above steps
Steps to check if the issue is with the SPC card.
Steps to identify if the cable/SFP is bad.
Test Results:
node0 fpc0 port0 <--[LINK DOWN]--> port 0 fpc12 node1 node0 fpc1 port0 <--[LINK UP]----> port 0 fpc13 node1 node0 fpc0 port0 <--[LINK UP]----> port 0 fpc13 node1 node0 fpc1 port0 <--[LINK DOWN]--> port 0 fpc12 node1
Based on the above tests SPC card in Node 1 had to be replaced.