This article provides some useful commands that can be used to indicate the need to tune ECN thresholds.
In designing the network infrastructure for an AI/ML cluster, the key objectives are to provide maximum throughput, minimal latency, minimal network interference for AI/ML traffic flows, and a lossless fabric. Today, most AI clusters operate most efficiently over lossless networks for optimum training completion times. Packet loss can significantly impact training completion times.
The following congestion parameters can be used to indicate the need for tuning ECN thresholds:
Monitor CLI command: "show interfaces <int> extensive"
Review “Output Errors” and “ECN Marked Packets”. This indicates that the buffer is utilized to the fill level value, and the switch requests senders to reduce the transmit rate. If this prematurely impacts application performance, tune as indicated in the Action field. Increase the drop profile fill level value until PFC occurs. Then, reduce by decrements of 5 until PFC stops.
ECN marking occurs later in the buffer utilization, reducing the frequency of traffic throttling.
Monitor CLI command: "show interfaces <int> extensive"—--> Review “Priority Flow Control Statistics” Indicates traffic is coming in at a rate greater than the shared input buffer.
Reduce drop profile fill level value until PFC no longer occurs. ECN occurs earlier in buffer utilization to mitigate the Pause Frame and interruption of traffic transmission.
CLI commands:
show interfaces <int> extensive
show interfaces queue <int>
show interfaces queue buffer-occupancy <int>
Review “Egress Queues” and “Dropped Packets” Indicates packets are being dropped based on WRED profile.
Reduce the drop profile fill level values until ECN occurs before drops/PFC. Traffic reduction should happen earlier, allowing queues to clear without drops.
show interfaces priority-group <int> buffer-occupancy
Review “Drops” and “Resource Errors” The ingress shared buffer is being exceeded and unable to store the incoming packets. This event is common when PFCs are generated.
Increase shared buffer > ingress > bufferpartition <%> For "Resource Errors", Increase the "ingress buffer-partition lossless-headroom" percentage. If the ingress shared buffer partition is at the expected value, Reduce the fill level values until ECN occurs before drops/PFC. More memory is allocated to the specified buffer if needed. No or minimal input drops were experienced.