Description

Number of simultaneous GRPC telemetry subscriptions to the QFX5100

Symptoms

If there are multiple paths from the multiple collectors subscribed to the fully loaded QFX5100 device,

  1. May notice the High system CPU
  2. No data to the some of the subscribed path etc.

Solution

If the above symptoms are noticed, please verify the number of paths and number of collectors.

Its noticed that in a fully loaded qfx device, If Mutiple paths have been subscribed from more than 2 collectors, high CPU utilization can be noticed.

 

<-- Please note the 4 collectors are polling the device

 

user@qfx> show network-agent statistics detail | match client  


May 17 12:10:04


  Client IP              : unix::21       


  Client IP              : unix::22       


  Client IP              : ipv6:::ffff:10.85.184.5:39782


  Client IP              : ipv6:::ffff:10.85.184.5:39784


  Client IP              : ipv6:::ffff:10.85.184.5:39786


  Client IP              : ipv6:::ffff:10.85.184.5:39788




<-- 4 TCP sessions to the qfx5100 established.


user@qfx> show system connections | match 11123         


May 17 12:10:07


tcp4    0   0 10.85.17.172.11123              10.85.184.5.39788               ESTABLISHED


tcp4    0   0 10.85.17.172.11123              10.85.184.5.39786               ESTABLISHED


tcp4    0   17 10.85.17.172.11123              10.85.184.5.39784               ESTABLISHED


tcp4    0   0 10.85.17.172.11123              10.85.184.5.39782               ESTABLISHED


tcp46   0   0 *.11123                    *.*                      LISTEN



The following are the list of the subscription paths used when the issue happened. The issue here is specific to this profile, so may or not be common to every other customers.


/junos/system/cmerror/configuration/ 

 /junos/system/cmerror/counters/ 

 /junos/system/linecard/transceiver/

 /components/     

 /junos/events/event[id='CHASSISD_SNMP_TRAP7']/ 

 /interfaces/interface/subinterfaces/subinterface/state/ 

 /junos/system/linecard/interface/logical/usage/ 

 /interfaces/interface/subinterfaces/subinterface/state/ 

 /interfaces/interface/subinterfaces/subinterface/state/

 /junos/system/linecard/transceiver/

 /junos/task-memory-information/task-memory-total-bytes-percent/

 /junos/rsvp-interface-information/

 /network-instances/network-instance/protocols/protocol/isis/levels/level/link-state-database/lsp/tlvs/tlv/hostname/state/

 /interfaces/interface/state/

 /arp-information/ipv4/neighbors/neighbor/state/

 /components/component/properties/property/state/value/

 /components/component/state/

 /junos/services/label-switched-path/usage/

 /junos/system/linecard/fabric/

 /junos/system/linecard/firewall/

 /junos/system/linecard/optics/

 /junos/system/linecard/packet/usage/

 /junos/task-memory-information/task-memory-overall-report/

 /lacp/interfaces/interface/members/member/state/

 /lldp/interfaces/interface/neighbors/neighbor/state/

 /lldp/interfaces/interface/state/

 /lldp/state/     

 /nd6-information/ipv6/neighbors/neighbor/state/

 /network-instances/network-instance/mpls/lsps/constrained-path/tunnels/tunnel/

 /network-instances/network-instance/mpls/signaling-protocols/rsvp-te/global/state/

 /network-instances/network-instance/mpls/signaling-protocols/rsvp-te/interface-attributes/interface/

 /network-instances/network-instance/mpls/signaling-protocols/rsvp-te/sessions/session/state/session-name/

 /network-instances/network-instance/mpls/signaling-protocols/rsvp-te/sessions/session/explicit-route-objects/

 /network-instances/network-instance/protocols/protocol/bgp/neighbors/

 /system/processes/process/state/


Modification History

2024-07-17 : Article Created