This article explains the procedures and processes that run behind the scene when a secondary node join the cluster.
From the console, we can see the secondary node booting is completed. However, FPCs will not be powered up. We will see all the data interfaces missing as well since the FPCs did not come up. Control and Fab links will also appear as down.
Chassisd logs from the Secondary node show below log messages after the bootup.
Sep 11 10:40:04 LCC: send: lcc 1 sending swversion CHASSISD release 19.4R3.11 built by builder on 2020-10-08 22:09:37 UTC Sep 11 10:40:04 LCC: ch_cluster_lcc_donate_mac_pool_to_scc: sending mac addresses to the SCC chassis Sep 11 10:40:04 LCC: send: lcc 1 sending mac addresses Sep 11 10:40:04 LCC: ch_lcc_dispatch: SCC closed connection, LCC quits too
Chassisd logs from the primary node show below log messages.
Sep 11 10:40:02 SCC: ipc pipe 0x10d2700 created Sep 11 10:40:04 SCC: ch_ipc_dispatch: IDL decode failed. Magic 5022 error -1 (multi-chassis, 105) Sep 11 10:40:04 CHASSISD_IPC_MSG_FRU_NOT_FOUND: ch_ipc_dispatch: unable to locate FRU for message with type 1026, subtype 146, opcode 1 Sep 11 10:40:04 SCC: ch_connection_shutdown: Destroying the IPC pipe
These messages can be observed repeatedly on both nodes.
We need to understand SCC and LCC importance in chassis cluster and how that can be impacted for the cluster boot up.
In an SRX cluster, the SCC runs only on the Primary Node irrespective of which node is acting as primary. LCC runs on both primary and secondary nodes. LCC0 runs on node0 and LCC1 runs on Node1. During the the cluster booting process LCCx tries to connect with the SCC process. This connectivity has to be successful for further process completion. When both nodes are rebooted at the same time the unit that comes up first tries to detect the SCC if it detects it as the Primary node the SCC will be started and then the local LCC will be started. Once LCC is online it will try to power on the Line-cards and send the ready messages to SCC, then LCC will send an online message and online ACK to finish the Lince-card online process.
In case there is already a node acting as primary then the node that comes up next will try to start the LCC and try to connect the SCC. If this connectivity fails then no cards will be brought up. As per the error message below SCC was not able to understand the message that was sent by the new node that was trying to join the cluster. These kinds of issues can be observed if the customer upgrades the nodes by not following the proper upgrade path.
Sep 11 10:40:04 SCC: ch_ipc_dispatch: IDL decode failed. Magic 5022 error -1 (multi-chassis, 105) Sep 11 10:40:04 CHASSISD_IPC_MSG_FRU_NOT_FOUND: ch_ipc_dispatch: unable to locate FRU for message with type 1026, subtype 146, opcode 1 Sep 11 10:40:04 SCC: ch_connection_shutdown: Destroying the IPC pipe
Follow a proper upgrade path to avoid such issues.