Description

This article summarizes the behavior of Junos platforms when drops are seen associated with rapid pings to the Junos device itself.

Symptoms

Some ping requests are dropped by the Junos platform when testing with rapid pings to the same. 

In any juniper devices when we ping jumbo packet size along with rapid count these drops will be seen even if its direct Point to point connectivity. 

Solution

 

Sometimes it is possible to observe ICMP (Internet Control Message Protocol) packet drops when a rapid ping is used to the remote side of a directly connected interface while using the default class-of-service configuration. The following discussion shows how to check various ICMP-related counters on the router to verify if such packets are being dropped due to a rate limit.

Topology

SRX1(192.168.1.2)-----------------------------(192.168.1.5)SRX2

SRX1> ping 192.168.1.5 rapid count 100000

PING 192.168.1.5 (192.168.1.5): 56 data bytes

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
^C
--- 192.168.1.5 ping statistics ---
53063 packets transmitted, 53000 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.255/0.553/53.249/1.420 ms

 

It can be seen that there is a loss of 63 packets during the test.  Let us check the ICMP statistics on SRX2 :

SRX2> show system statistics icmp
icmp:
63 drops due to rate limit
0 calls to icmp_error
0 errors not generated because old message was icmp
Output Histogram
53063 echo reply
0 messages with bad code fields
0 messages less than the minimum length
0 messages with bad checksum
0 messages with bad source address
0 messages with bad length
0 echo drops with broadcast or multicast destination address
0 timestamp drops with broadcast or multicast destination address
Input Histogram
1 destination unreachable
53063 echo
53063 message responses generated



This is not a software bug, nor a problem with the link. Instead, the drops occurred on the SRX2 is because of the default rate-limiting of ICMP packets on Junos platforms. SRX2 is receiving all of the ICMP request packets and at the same time generating ICMP replies to send back to SRX1. However, during this process, it is hitting the default rate limit of ICMP within the kernel, which is on the routing-engine. The default ICMP rate limit on the system can be checked using the start shell command.

NOTE  : Do not run the shell commands in production without JTAC supervision.

SRX2> start shell
% sysctl -a | grep "inet.icmp"
net.inet.icmp.maskrepl: 0
net.inet.icmp.bucketsize: 5
net.inet.icmp.tokenrate: 1000
net.inet.icmp.drop_redirect: 0
net.inet.icmp.log_redirect: 0
net.inet.icmp.bmcastecho: 1
% exit

Notice in the output above that the ICMP rate is set to 1000; this is the default system value.

ICMP is not the best way to test the delay or capacity on the link. One of the best ways to test the delay or capacity is to attach a host or traffic generator to both sides. Then traffic can be sent through, or transit to,  the devices. By taking a transit path through the router, there is not any default rate limiting that would limit the traffic.

NOTE  : The below icmp token rate is same for all juniper devices (MX, ACX, PTX, SRX, EX & QFX series )

Modification History

Updated symptom & solutions , same behaviour is seen in all juniper device.