This article explains a scenario where the traceroute response is not coming from a intermediate router.
While running taceroute a PTX in the path not generating or sending with time exceeded message hence not showing up in the trace route path.
testuser@labrouter> traceroute 103.251.241.5
traceroute to 103.251.241.5 (103.251.241.5) from 1.1.1.1, 30 hops max, 52 byte packets
1 101.25.65.73.839 ms 1.662 ms 1.203 ms
2 * * * <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< incomplete
3 33.44.3.1 1.780 ms
MPLS Label=10478 CoS=0 TTL=1 S=1
4 52.82.194.75 0.987 ms
5 103.251.241.5 1.763 ms
This issue can be due to the hostpath bandwidth getting oversubscribed so check for the following outputs.
Login to fpc shell using the below command.
start shell pfe network fpc0
Once in shell collect the below outputs and share.
debug cos halp all_stats
debug cos halp all_stats local-only non-zero
debug cos halp ifd Remote_Host_IF:pPFE0 drop_stats summary
debug cos halp ifd Remote_Host_IF:pPFE1 drop_stats summary
debug cos halp ifd Remote_Host_IF:pPFE2 drop_stats summary
debug cos halp show sp_ifd chip 0 >>>> find the Remote Host IF index number in decimal
#debug cos halp ifd 2147483652
#debug cos halp ifd 2147483652 txstats
#debug cos halp ifd 2147483652 drop_stats
#debug cos halp ifd 2147483652 drop_stats summary
# show icmp statistics
# show ttp statistics
# show shim jnh punt stats
# show ddos policer all stats
Once you review the above outputs you would notice the drop stats being high as the capacity of the host queue is 10 Mbps however the volume being received on the host queue is hugh.
FPC0(labrouter vty)# debug cos halp all_stats local-only non-zero
-----------|--------------------|---------------|---------------|----------|
IFD | IFD | Transmit | Drop | Local |
IDX | Name | bps | bps | OQ |
2147483650| Loopback_Direct0:pP| 275.664 Kbps| 0| 96|
2147483652| Remote_Host_IF:pPFE| 9.766 Mbps| 1.672 Gbps| 128|
2147483654| Host_IF1:pPFE0| 712.000 bps| 0| 120|
2147549186| Loopback_Direct0:pP| 481.376 Kbps| 0| 96|
2147549188| Remote_Host_IF:pPFE| 9.784 Mbps| 2.167 Gbps| 128|
2147549190| Host_IF1:pPFE1| 712.000 bps| 0| 120|
2147614722| Loopback_Direct0:pP| 390.048 Kbps| 0| 96|
2147614724| Remote_Host_IF:pPFE| 9.781 Mbps| 1.754 Gbps| 128|
2147614726| Host_IF1:pPFE2| 712.000 bps| 0| 120|
2147680258| Loopback_Direct0:pP| 113.536 Kbps| 0| 96|
2147680260| Remote_Host_IF:pPFE| 240.960 Kbps| 0| 128|
2147680262| Host_IF1:pPFE3| 712.000 bps| 0| 120|
2147745794| Loopback_Direct0:pP| 156.240 Kbps| 0| 96|
2147745796| Remote_Host_IF:pPFE| 624.448 Kbps| 0| 128|
2147745798| Host_IF1:pPFE4| 712.000 bps| 0| 120|
2147811330| Loopback_Direct0:pP| 116.984 Kbps| 0| 96|
2147811332| Remote_Host_IF:pPFE| 101.528 Kbps| 0| 128|
2147811334| Host_IF1:pPFE5| 712.000 bps| 0| 120|
You would noticed here that the volume being received in queue 5. Queue 5 caters to sampling and exceptions (TTL/MTU exceeded, etc. ) packets. So a drop in queue 5 would impact the icmp time exceeded messages as well leading to the traceroute response getting dropped here.
FPC0(labrouter vty)# ...ote_Host_IF:pPFE0 drop_stats summary
Getting Summary IFD Remote_Host_IF:pPFE0: Start VOQ 160 num_voqs 8
Summary VOQ Drop Stats for FPC0: voq 160 thru 167
----|------|---|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|-------------|-------------|
| VOQ Drop Statistics |
pPFE|vPFE12| OQ| Tail Byte Count | RED Byte Count | Tail Bit Rate | RED Bit Rate | Tail Pkt Count | RED Pkt Count |Tail Pkt Rate| RED Pkt Rate|
0| 1| 32| 0| 0| 0.000 bps| 0.000 bps| 0| 0| 0.000 pps| 0.000 pps|
0| 1| 33| 0| 0| 0.000 bps| 0.000 bps| 0| 0| 0.000 pps| 0.000 pps|
0| 1| 34| 0| 0| 0.000 bps| 0.000 bps| 0| 0| 0.000 pps| 0.000 pps|
0| 1| 35| 0| 0| 0.000 bps| 0.000 bps| 0| 0| 0.000 pps| 0.000 pps|
0| 1| 36| 0| 0| 0.000 bps| 0.000 bps| 0| 0| 0.000 pps| 0.000 pps|
0| 1| 37| 1301867771743124| 0| 1.474 Gbps| 0.000 bps| 997550323999| 0| 151.607 Kpps| 0.000 pps|
0| 1| 38| 0| 0| 0.000 bps| 0.000 bps| 0| 0| 0.000 pps| 0.000 pps|
0| 1| 39| 0| 0| 0.000 bps| 0.000 bps| 0| 0| 0.000 pps| 0.000 pps|
====|======|===|====================|====================|====================|====================|====================|====================|=============|=============|
0| 1|Tot| 1301867771743124| 0| 1.474 Gbps| 0.000 bps| 997550323999| 0| 151.607 Kpps| 0.000 pps|
FPC0(labrouter vty)# debug cos halp show sp_ifd chip 0
Special Path IFD's for pPFE ID 0 chip 0 local=TRUE
IFD | IFD Index | |
Name | Hex | Decimal | Type |
------------------------------|----------|----------|--------------------|
LCPU:pPFE0 |0x80000001|2147483649| LCPU |
Loopback_Direct0:pPFE0 |0x80000002|2147483650| Loopback_Direct0 |
Loopback_Direct1:pPFE0 |0x80000003|2147483651| Loopback_Direct1 |
Remote_Host_IF:pPFE0 |0x80000004|2147483652| Remote_Host_IF |
Host_IF0:pPFE0 |0x80000005|2147483653| Host_IF0 |
Host_IF1:pPFE0 |0x80000006|2147483654| Host_IF1 |
FPC0(labrouter vty)# debug cos halp ifd 2147483652 drop_stats
Getting IFD Remote_Host_IF:pPFE0: Start VOQ 160 num_voqs 8
Drop Stats on FPC0 Chip 0 for voq 160 thru 167
0| 1| 37| 1301886288142055| 0| 2.604 Gbps| 0.000 bps| 997564599709| 0| 239.278 Kpps| 0.000 pps|
0| 1|Tot| 1301886288142055| 0| 2.604 Gbps| 0.000 bps| 997564599709| 0| 239.278 Kpps| 0.000 pps|
As you can see here issue is clearly due to oversubscription of host queue. A permanent solution to this would be to idenity what so much of host bound traffic is comming to hostpath and stop it as this could be some ddos or other traffic that is not supposed to come.
2024-06-28 : Article Created
2025-7-31 : Replaced hostname
2025-8-12 : Filled in Modification History