This article discusses the impact of replacing a QFX3600-I InterConnect (IC) device on a QFX3000-M QFabric system.

The tests described in this article were carried out in a lab environment using a IXIA traffic generator.  A lab environment is generally less complex than a production environment. This article aims to quantify the amount of packet loss, but will not look into how services might be impacted. Service impact depends on the design, the protocols used, and how they handle and recover from packet loss.

Note: The impact of an IC crash or replacement on a QFabric QFX3000-G is different.

A possible symptom is that a fan fails, and even after replacing it, the new fan also does not work. After troubleshooting, it is determined that the slot is faulty and not the fan.

The  QFX3600-I  chassis might need to be replaced or rebooted following troubleshooting or RMA.

Impact of IC Replacement by Different Scenarios

Scenario 0 - power-off the IC:

The recommended procedure can be found in the following technical document:
Adding or Replacing an Interconnect Device in a QFX3000-M QFabric System

Additionally, tests were conducted in the lab pushing 20Gbps through the IC with minimal packet loss. Powering off the device or halting the device exhibit the same results. In the lab, traffic is gracefully failed over to the other IC device within 10 to 15 seconds, observing no packet loss.

Note: This was a lab with a clean install. You should expect and plan for some packet drops in a live working scenario.

The following commands were used on the IC, which exhibit the same results:

• request system power-off in 0
• request system halt in 0
• request system reboot in 0​

Scenario 1 - Disable the interfaces on the node devices 1 by 1

This may complicate and/or extend your maintenance window.
Once the IC chassis is replaced, you need to go back to each node device and enable the interfaces 1 by 1.

Scenario 2 - Disable the interfaces on the IC device 1 by 1

This may extend your maintenance window.
You need to log into the IC device as root and then to the CLI to be able to access the configuration from outside the DG.
This is not the recommended method of replacing the IC device.
If you do this, you would have several moments where traffic is impacted (1 per interface during each interface failover

This would be comparable to powering off the device, but can give you a quick way to restore traffic back in case the service impact is greater than expected.

Impact of IC Replacement by Disabling Interfaces

• Frame drops are observed for L2 unicast/multicast traffic when the fte* interfaces from the interconnect device are disabled.
• These frame drops stop when the traffic is failed over to the remaining interconnect device.
• No frame drops are observed when the interfaces are enabled again, placing traffic back on the IC device.
• Total frame drops for a given frame size is inversely proportional to the frame size.
• The percentage of dropped frames during failover as compared to the total frames transmitted in a 60s interval is constant, regardless of frame size. Which suggests that the number of dropped frames is a function of frame size.
• By measuring the total frame count transmitted in a 60s interval with a constant load, one can calculate the amount of frames per second going through the IC device.
• Failover time is calculated to be around 14ms, based on the fps number and the total Dropped Frame Count.

One can observe these amounts of drops during the moment traffic is failed over, under the following conditions:

• ≈6s commit time (± 1s error) on the IC device
• Constant load of 5 Gbps Tx and 5 Gbps Rx going through the IC device (aggregated 10G of traffic)
• A Total Frame Count span of ≈60s (± 1s error) , during which all fte* interfaces in the IC device are disabled at the ≈30s mark (± 1s error)