csim@BX-scapa-re0> show network-agent statistics detail 2 | find object
Sensor Path : /network-instances/network-instance/mpls/signaling-protocols/rsvp-te/sessions/session/explicit-route-objects/
Reporting Interval : 30
Component(s) : rpd
Component ID : 65535
Average iLatency (ms) : 4298494
Average Circular Buffer Used (%) : 0
Bytes Sent : 1126483
Packets Sent : 66
Drops : 0
Initial Sync Bytes Sent : 0
Initial Sync Packets Sent : 0
Initial Sync Drops : 0
Initial Sync Average iLatency (ms) : 0
Initial Sync Average Circular Buffer Used (%) : 0
Upon further analysis of the logs and debug session outputs, we found that the problem was observed on both the collectors. The root cause is identified to be the low JSD send socket buffer size which was set to 512KB instead of the default value of 8 MB. The default value of 8 MB is the recommended value for JUNOS EVO.
user@router-name# show system services
extension-service {
request-response {
grpc {
clear-text {
port 10162;
}
max-connections 8;
send-socket-buffer-size 512000;
Smaller send socket buffer size can lead to high Latency in data transfer if there is a sudden spike in RTT and packet drop.
With the send-socket-buffer-size set to 512KB, Juniper could replicate the issue in an in-house lab when the network had a high round-trip time (RTT), such as 250ms to the collector. A very high sensor latency spike was observed.
[root@jtimon]# ping inet6 2001:caff:0:c:0:5254:7d:80ec
PING 2001:caff:0:c:0:5254:7d:80ec(2001:caff:0:c:0:5254:7d:80ec) 56 data bytes
64 bytes from 2001:caff:0:c:0:5254:7d:80ec: icmp_seq=1 ttl=62 time=251 ms
64 bytes from 2001:caff:0:c:0:5254:7d:80ec: icmp_seq=2 ttl=62 time=250 ms
64 bytes from 2001:caff:0:c:0:5254:7d:80ec: icmp_seq=3 ttl=62 time=250 ms
64 bytes from 2001:caff:0:c:0:5254:7d:80ec: icmp_seq=4 ttl=62 time=250 ms
The average iLatency value returned to normal when the send-socket-buffer-size was reset to the default of 8MB, even with the same RTT of 250ms.
[edit system services extension-service request-response grpc]
- send-socket-buffer-size 512000;
[root@jtimon]# ping 10.85.183.237
PING 10.85.183.237 (10.85.183.237) 56(84) bytes of data.
64 bytes from 10.85.183.237: icmp_seq=1 ttl=61 time=250 ms
64 bytes from 10.85.183.237: icmp_seq=2 ttl=61 time=250 ms
64 bytes from 10.85.183.237: icmp_seq=3 ttl=61 time=250 ms
^C
Sensor Path : /network-instances/network-instance/mpls/signaling-protocols/rsvp-te/sessions/session/explicit-route-objects/ Reporting Interval : 30 Component(s) : rpd Component ID : 65535 Average iLatency (ms) : 0 Average Circular Buffer Used (%) : 0 Bytes Sent : 579562 Packets Sent : 34 Drops : 0 Initial Sync Bytes Sent : 0 Initial Sync Packets Sent : 0 Initial Sync Drops : 0 Initial Sync Average iLatency (ms) : 0 Initial Sync Average Circular Buffer Used (%) : 0 Due to its different software architecture, JUNOS EVO has better resources to handle streaming data with a larger JSD send socket buffer size than JUNOS. The default send-socket-buffer-size value of 8MB has been proven effective. Hence, it is recommended to use the default value of 8MB for send-socket-buffer size. (by removing the “send-socket-buffer-size” configuration for JUNOS EVO).Another recommendation was - Collector GRPC HTTP initial window size of 5 MB for all pre-gNMI sessions for all EVO devices.