Issue of GNMI statistics showing high ilatency for multiple sensors except BGP.

+ observed high ilatency for parent as well as child sensors. which includes interfaces, system, components.

+ these issue started on MX10003 after upgrading to version 24.2S5 and you have other MX10003 with 24.2S4 with same configuration and there didn't observe the issue.

+ You have mentioned that no service impact observed due to these issue, only issue is rendering on the monitoring tool and sometimes graph is not getting generated.

Sensor Statistics :

    Sensor Path                  : /interfaces/

    Reporting Interval              : 10

    Node-name                   : jro-2-inet,jro-2-inet,jro-2-inet

    Component(s)                 : PFE,PFE,PFE,PFE,xmlproxyd_TM_Thread_1,rpd,l2aldTM,na-grpcd,mib2d,dcd,l2ald,dot1xd

    GNMI Sub Mode                 : SAMPLE

    Bytes Sent                  : 5175386109

    Packets Sent                 : 7592393

    Drops                     : 6032909

     Average iLatency (ms)             : 258189 <<<

    Average Circular Buffer Used (%)       : 80

    Initial Sync Bytes Sent            : 526970

    Initial Sync Packets Sent           : 936

    Initial Sync Drops              : 0

    Initial Sync Average iLatency (ms)      : 9

    Initial Sync Average Circular Buffer Used (%) : 80

As per Engineering, the issue could be related with internal PR 1906921, which is fixed via PR1907660.

And since issue is seen after upgrade to 23.4R2-S5 & not seen in 23.4R2-S4, they have confirmed the above possibility.

Their ask is to upgrade to 23.4R2-S6, which has the fix.

If the issue is not resolved with above upgrade, we need to set window size of 8MB at collector:

+++++++++++++++++++++++++

With 23.4R2S6 don't see any issue which could cause latency. On 23.x we have JSD as gateway grpc server on the device and post 24.2 we have nginx as gateway server. With JSD (< 24.2 versions), it’s expected that collector uses appropriate http window size (~8 MB) set from the collector side. Please check what is the window size set by the collector side. If its not set, can we set to 8 MB and see. Post 24.2 where nginx is the GW, we don't have this requirement of having window size at the collector.

You may refer below KB for window size:

https://supportportal.juniper.net/s/article/Telemetry-Optimization-higher-Latency-observed