This article is about SRX Series branch devices (SRX100, SRX210, SRX220, SRX240, SRX550and SRX650) which have code with 10.4 or later. The existing virtual LAN (VLAN) tag, which is used for control-link traffic, will be replaced with the use of experimental Ether type 0x88b5.However, backward compatibility is also supported for devices, which have already been deployed the chassis cluster with VLAN tagging.
This KB applies if you have a chassis cluster up and running with some code prior to 10.4 and done a RMA for an RE or entire node of a cluster.New devices come with code later than 10.4. So to have a cluster, you either have to upgrade the code in existing devices to the latest version or downgrade the new device to the existing cluster code. Now, after performing the upgrade/downgrade, you will see a split brain condition in the cluster.
{primary:node0} root@node0> show chassis cluster status Cluster ID: 1 Node Priority Status Preempt Manual failover Redundancy group: 0 , Failover count: 1 node0 200 primary no no node1 0 lost n/a n/a Redundancy group: 1 , Failover count: 1 node0 200 primary no no node1 0 lost n/a n/a {primary:node1} root@node1> show chassis cluster status Cluster ID: 1 Node Priority Status Preempt Manual failover Redundancy group: 0 , Failover count: 1 node0 0 lost n/a n/a node1 100 primary no no Redundancy group: 1 , Failover count: 1 node0 0 lost n/a n/a node1 0 primary no no
With codes prior to 10.4, by default, the control port tagging was enabled and it used the 4094 VLAN. For 10.4 and later codes, by default, it is disabled.So, the upgrade/downgrade makes one node of the control port as tagged and the other node as untagged; so this causes control packets to drop, which in turn causes the Split Brain condition.You can check if the control ports are enabled or disabled, by running the show chassis cluster information detail command via the CLI. The chopped output of the command is as follows:
{primary:node0} root@node0> show chassis cluster information detail node0: -------------------------------------------------------------------------- Control link statistics: Control link 0: Heartbeat packets sent: 259 Heartbeat packets received: 0 Heartbeat packet errors: 0 Duplicate heartbeat packets received: 0 Control recovery packet count: 0 Sequence number of last heartbeat packet sent: 254 Sequence number of last heartbeat packet received: 0 Fabric link statistics: Probes sent: 254 Probes received: 0 Probe errors: 0 Probes not processed: 0 Probes dropped due to control link down: 0 Probes dropped due to fabric link down: 0 Sequence number of last probe sent: 254 Sequence number of last probe received: 0 Chassis cluster LED information: Current LED color: Red Last LED change reason: Peer node: node1 is not present Control port tagging: Disabled {primary:node1} root@node1> show chassis cluster information detail node1: -------------------------------------------------------------------------- Control link statistics: Control link 0: Heartbeat packets sent: 779 Heartbeat packets received: 0 Heartbeat packet errors: 0 Duplicate heartbeat packets received: 0 Control recovery packet count: 0 Sequence number of last heartbeat packet sent: 1175 Sequence number of last heartbeat packet received: 0 Fabric link statistics: Probes sent: 779 Probes received: 0 Probe errors: 0 Probes not processed: 0 Probes dropped due to control link down: 284 Probes dropped due to fabric link down: 0f Sequence number of last probe sent: 1175 Sequence number of last probe received: 0 Chassis cluster LED information: Current LED color: Red Last LED change reason: Peer node: node0 is not present Control port tagging: Enabled
to avoid the split brain condition, set both sides of the control-link either as tagged or untagged, by using the following command via the CLI:
root> set chassis cluster control-link-vlan enable/disable
{primary:node0}root@node0> set chassis cluster control-link-vlan enablewarning: A reboot is required for control-link-vlan to be enabled{primary:node0}root@node0> request system rebootReboot the system ? [yes,no] (no) yes
{primary:node1}root@node1> set chassis cluster control-link-vlan disablewarning: A reboot is required for control-link-vlan to be disabled{primary:node1}root@node1> request system rebootReboot the system ? [yes,no] (no) yes
{primary:node0}root@node0> show chassis cluster statusCluster ID: 1Node Priority Status Preempt Manual failoverRedundancy group: 0 , Failover count: 1node0 200 primary no nonode1 100 secondary no noRedundancy group: 1 , Failover count: 1node0 200 primary no nonode1 100 secondary no no{primary:node0}root@node0> show chassis cluster information detailnode0:-------------------------------------------------Control link statistics:Control link 0:Heartbeat packets sent: 623Heartbeat packets received: 600Heartbeat packet errors: 0Duplicate heartbeat packets received: 0Control recovery packet count: 0Sequence number of last heartbeat packet sent: 617Sequence number of last heartbeat packet received: 637Fabric link statistics:Probes sent: 617Probes received: 522Probe errors: 0Probes not processed: 254Probes dropped due to control link down: 0Probes dropped due to fabric link down: 3Sequence number of last probe sent: 617Sequence number of last probe received: 637Chassis cluster LED information:Current LED color: GreenLast LED change reason: No failuresControl port tagging:Disablednode1:-------------------------------------------Control link statistics:Control link 0:Heartbeat packets sent: 639Heartbeat packets received: 603Heartbeat packet errors: 0Duplicate heartbeat packets received: 0Control recovery packet count: 0Sequence number of last heartbeat packet sent: 637Sequence number of last heartbeat packet received: 617Fabric link statistics:Probes sent: 637Probes received: 520Probe errors: 0Probes not processed: 253Probes dropped due to control link down: 0Probes dropped due to fabric link down: 3Sequence number of last probe sent: 637Sequence number of last probe received: 617Chassis cluster LED information:Current LED color: GreenLast LED change reason: No failuresControl port tagging:Disabled