This document explains steps to troubleshoot queue drops observed on AE bundle.
Customer reported that video traffic is impacted and observed queue drops.
Please collect below command outputs from router:show interfaces ae* extensiveshow interfaces queue ae*show configuration class-of-service interfacesshow configuration class-of-service schedulers <scheduler name>Step 1. Execute command "show interfaces ae* extensive" and check if drops are incrementing. Also check the utilization and confirm that link is not overutilized.
Egress queues: 8 supported, 4 in use Queue counters: Queued packets Transmitted packets Dropped packets 0 725116998840 725107997846 9000994 >>>>>>>> drops are incrementing 1 61303989175 61303989175 0 2 2480086933 2480086933 0 3 124854596 124854596 0
Below schedulers are configured :
Queue number: Mapped forwarding classes 0 BEST_EFFORT_2_CLASS,BEST_EFFORT_1_CLASS 1 GOLD_SILVER_CLASS,BRONZE_MGMT_CLASS 2 VOICE_CLASS,VIDEO_CLASS 3 PROTOCOLS_CLASS
Step 2. Check which queue is dropping the packets :
Lab_PTX10001-36MR> show interfaces queue ae15 Physical interface: ae15, up, Physical link is Up Interface index: 4889, SNMP ifIndex: 1067 Description: Forwarding classes: 16 supported, 7 in use Egress queues: 8 supported, 4 in use Queue: 0, Forwarding classes: BEST_EFFORT_2_CLASS,BEST_EFFORT_1_CLASS Queued: Packets : 2648517003654 984446 pps Bytes : 3515610120111375 9818867808 bps Transmitted: Packets : 2647741522645 984342 pps Bytes : 3514565211574352 9817703520 bps Tail-dropped packets : 775074848 104 pps >>>>>>>>>>>>> Tail drops observed only on Best Effort queue Tail-dropped bytes : 1044459154368 1164288 bps RL-dropped packets : 0 0 pps RL-dropped bytes : 0 0 bps RED-dropped packets : 406161 0 pps RED-dropped bytes : 449382655 0 bps Low : 0 0 pps Medium-low : 406161 0 pps Medium-high : 406161 0 pps High : 406161 0 pps Low : 0 0 bps Medium-low : 449382655 0 bps Medium-high : 449382655 0 bps High : 449382655 0 bps Queue-depth bytes : Average : 0 Current : 0 Peak : 0 Maximum : 0 Queue: 1, Forwarding classes: GOLD_SILVER_CLASS,BRONZE_MGMT_CLASS Queued: Packets : 103249338078 186844 pps Bytes : 113356740214642 1702522288 bps Transmitted: Packets : 103249338078 186844 pps Bytes : 113356740214642 1702522288 bps Tail-dropped packets : 0 0 pps Tail-dropped bytes : 0 0 bps RL-dropped packets : 0 0 pps RL-dropped bytes : 0 0 bps RED-dropped packets : 0 0 pps RED-dropped bytes : 0 0 bps Low : 0 0 pps Medium-low : 0 0 pps Medium-high : 0 0 pps High : 0 0 pps Low : 0 0 bps Medium-low : 0 0 bps Medium-high : 0 0 bps High : 0 0 bps Queue-depth bytes : Average : 0 Current : 0 Peak : 0 Maximum : 0 Queue: 2, Forwarding classes: VOICE_CLASS,VIDEO_CLASS Queued: Packets : 4939372470 4229 pps Bytes : 6835557019757 39446456 bps Transmitted: Packets : 4939372470 4229 pps Bytes : 6835557019757 39446456 bps Tail-dropped packets : 0 0 pps Tail-dropped bytes : 0 0 bps RL-dropped packets : 0 0 pps RL-dropped bytes : 0 0 bps RED-dropped packets : 0 0 pps RED-dropped bytes : 0 0 bps Low : 0 0 pps Medium-low : 0 0 pps Medium-high : 0 0 pps High : 0 0 pps Low : 0 0 bps Medium-low : 0 0 bps Medium-high : 0 0 bps High : 0 0 bps Queue-depth bytes : Average : 0 Current : 0 Peak : 0 Maximum : 0 Queue: 3, Forwarding classes: PROTOCOLS_CLASS Queued: Packets : 387331807 270 pps Bytes : 55667537232 289544 bps Transmitted: Packets : 387331807 270 pps Bytes : 55667537232 289544 bps Tail-dropped packets : 0 0 pps Tail-dropped bytes : 0 0 bps RL-dropped packets : 0 0 pps RL-dropped bytes : 0 0 bps RED-dropped packets : 0 0 pps RED-dropped bytes : 0 0 bps Low : 0 0 pps Medium-low : 0 0 pps Medium-high : 0 0 pps High : 0 0 pps Low : 0 0 bps Medium-low : 0 0 bps Medium-high : 0 0 bps High : 0 0 bps Queue-depth bytes : Average : 0 Current : 0 Peak : 0 Maximum : 0
We checked that packet drops observed only in Best Effort queue. No drops observed in other queues.
When you have packets enqueued which is more than you could send, you might see RED drop. If the enqueueing packets come too fast and it even can't be enqueued (ie. the current queue size is reaching max), you will see tail drop.
Step 3. Check configuration for class-of-service applied on the interface level. Check configuration for schedulers.
Lab_PTX10001-36MR> show configuration class-of-service scheduler-maps PTX10000_CORE_OUT_MAP forwarding-class BEST_EFFORT_2_CLASS scheduler BEST_EFFORT_QUEUE; forwarding-class BEST_EFFORT_1_CLASS scheduler BEST_EFFORT_QUEUE; forwarding-class BRONZE_MGMT_CLASS scheduler MGMT_CMRC_QUEUE; forwarding-class GOLD_SILVER_CLASS scheduler MGMT_CMRC_QUEUE; forwarding-class VIDEO_CLASS scheduler VIDEO_VOICE_QUEUE; forwarding-class VOICE_CLASS scheduler VIDEO_VOICE_QUEUE; forwarding-class PROTOCOLS_CLASS scheduler PROTOCOLS_QUEUE;
schedulers { BEST_EFFORT_QUEUE { transmit-rate percent 1; buffer-size percent 13; priority low; drop-profile-map loss-priority any protocol any drop-profile WRED_DROP; } MGMT_CMRC_QUEUE { transmit-rate percent 45; buffer-size percent 40; priority low; drop-profile-map loss-priority any protocol any drop-profile WRED_DROP; } VIDEO_VOICE_QUEUE { transmit-rate percent 45; buffer-size percent 40; priority high; } PROTOCOLS_QUEUE { transmit-rate { percent 7; rate-limit; } buffer-size percent 7; priority strict-high; } }
As per customer, Video Traffic is impacted. Video traffic in untagged and thus taking best effort queue.
Step 4. Check transmit-rate and buffer-size for the scheduler.
In this case, best-effort queue was more utilized as incoming video traffic was not marked.
We checked that transmit-rate and buffer-size is very small for best-effort queue.
Lab_PTX10001-36MR> show configuration class-of-service schedulers BEST_EFFORT_QUEUE transmit-rate percent 1; buffer-size percent 13; priority low; drop-profile-map loss-priority any protocol any drop-profile WRED_DROP;
We can increase the transmit-rate and buffer-size for best effort queue but we need to make sure that total percent of all the schedulers is 100.
It is difficult to predict and to test the bandwidth allocation for different class.
Customer design team should check for appropriate bandwidth allocation for each class as per usage.
Step 5. Modify transmit-rate and buffer-size according to traffic utilization.
In this case, modified the best effort and video-voice queue configuration to resolve the issue.
BEST_EFFORT_QUEUE { transmit-rate percent 37; buffer-size percent 35; priority high; drop-profile-map loss-priority any protocol any drop-profile WRED_DROP; } VIDEO_VOICE_QUEUE { transmit-rate percent 10; buffer-size percent 10; priority high;