Description

During a maintenance activity involving a node replacement or cluster member rejoin, an unexpected restart of services or processing cards may occur if the cluster control and fabric port configuration is applied in an incorrect sequence.

 

In a chassis cluster environment, both nodes must have consistent control and fabric port configurations before the replacement node is brought back online. If a node rejoins the cluster before the peer node has been updated with the corresponding configuration changes, a synchronization event may be triggered. In certain scenarios, this can result in service interruptions while the cluster reconciles the configuration state across both nodes.

Symptoms

Expected Procedure (Minimal-Downtime Maintenance)

 

For a node replacement or upgrade operation, the recommended sequence is:

  1. Configure temporary (dummy) control and fabric ports as required.
  2. Replace the affected node.
  3. Configure the correct control and fabric ports on the replacement node.
  4. Halt or keep the replacement node offline.
  5. Configure the correct control and fabric ports on the peer node and commit the changes.
  6. Verify that the configuration is successfully applied on the active node.
  7. Power on the replacement node and allow it to rejoin the cluster.

 

Following this sequence ensures that both nodes have consistent configurations before cluster synchronization occurs.

 

Observed Behavior

 

The following sequence occurred instead:

  1. The correct control and fabric port configuration was applied to the replacement node.
  2. The replacement node was halted.
  3. The replacement node was powered on before the peer node had been updated with the corresponding control and fabric port configuration.
  4. The replacement node successfully rejoined the cluster.
  5. Configuration changes were subsequently committed on the peer node while both nodes were online.

 

As a result, the cluster detected a configuration mismatch and initiated a synchronization process. During this synchronization, service cards or processing modules on the peer node restarted, resulting in temporary service impact.

Solution

This behavior is expected when cluster control and fabric port configuration changes are committed while both nodes are online and operating with inconsistent cluster interface settings.

 

Once the configuration changes were committed on the peer node, the cluster initiated a synchronization process to align the operational state across both members. As part of this process, service cards or processing modules restarted to apply the updated configuration and restore cluster consistency.

 

No hardware failure was involved; the restart occurred as part of the cluster synchronization mechanism.

Modification History

2025-12-22 : Article Created