This KB article provides a general data collection checklist to troubleshoot IPFIX issue on QFX platform.

Important Notes:

• Use these commands only as directed by Juniper Networks Technical Assistance Center (JTAC) or engineering teams.

For additional assistance, please contact Juniper Networks JTAC.

Routing Engine:

• ddos-protection statistics
• SRRD state (memory/cpu)
• RPD state (memory/CPU)
• Check Jflow config
• List of commands [Collect these 5 times]

set cli timestamp

show services accounting flow inline-jflow fpc_slot <slot_num>

show services accounting status inline-jflow fpc-slot <slot_num>

show services accounting errors inline-jflow fpc-slot <slot_num>

show chassis fpc

show route summary

show firewall (for filter with sample action)

FPC:

• Login to fpc prompt using 'rsh -Ji fpc<slot_num>' and entering root password:

Capture ‘top’ command output

#top

Capture ‘ps’ command output.  This command will display CPU% utilization by different processes running on FPC

#ps aux | sort -k3 -nr

• Login to 'vty fpc<slot_num>':

Here we need to capture ‘show threads cpu’ output.

Execute in below sequence:

show threads cpu

show sched

set parser security 10

clear sched

show threads cpu

show threads <pid> 

use first row pid of 'show threads cpu' output in 'show threads <pid> command'

• FPC VTY CLI (vty fpc(N) : N is the FPC slot no)

show flow-table stats

show ukern_trace memory-composition

show heap

show threads cpu

show pechip <pechip_no> trapstats

• MSPMAND VTY (vty -s mspmand fpc(N) : N is the FPC slot no)

plugin jflow show statistics (only this command output updated every 10 seconds)

show msp plugins pkt-cntrs

show msp pkt-cntrs terse

show msp offload aggregate-counters

show msp offload state-counters

show msp objcache summary

show msp process detail

show msp sessions summary

show asic jflow stats

• monitor the interface where collector is connected.

Following shell script need to be run for 5 iterations[Please type it. Don’t copy & paste]:

FPC <run on all FPCs>

while true; do

date

ifconfig eth_jflow

sleep 10

done

  Shell (Do this for all FPC slots):

Script 1:

while true; do

date

vty -s mspmand -S -c "show asic jflow stats" fpc<n>

date

vty -s mspmand -S -c "plugin jflow show statistics" fpc<n>

date

vty -s mspmand -S -c "show msp plugins pkt-cntrs" fpc<n>

date

vty -s mspmand -S -c " show msp pkt-cntrs terse" fpc<n>

date

vty -s mspmand -S -c " show msp offload aggregate-counters" fpc<n>

date

vty -s mspmand -S -c "show msp offload state-counters" fpc<n>

date

vty -s mspmand -S -c "show msp objcache summary" fpc<n>

date

vty -s mspmand -S -c "show msp process detail" fpc<n>

date

vty -s mspmand -S -c "show msp sessions summary" fpc<n>

date

sleep 1

done

Script2:

while true; do

date

vty -s mspmand -S -c "plugin jflow show statistics" fpc<n>

date

sleep 10

done

Script3                 [Do it for all the FPC slots]:

while true; do

   date

cprod -A fpc<n> -c “show flow-table stats”

cprod -A fpc<n> -c “show ukern_trace memory-composition”

               cprod -A fpc<n> -c “show heap”

cprod -A fpc<n> -c “show threads cpu”

               cprod -A fpc<n> -c “show pechip trapstats”

             cprod -A fpc<n> -c “show expr-if host stats”

               cprod -A fpc<n> -c “show sched”

               cprod -A fpc<n> -c “set parser security 10”

               cprod -A fpc<n> -c “clear sched”

               cprod -A fpc<n> -c “show threads cpu”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe12  txstats”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe13 txstats”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe14 txstats”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe15 txstats”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe16 txstats”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe18 txstats”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe19 txstats”

               cprod -A fpc<n> -c “debug cos halp ifd remote_host_if:ppfe15 txstats”

          sleep 1

      done

2023-5-15: Initial Publication.