Description

 

Data Collection and Troubleshooting Guides can help with issue investigation as well as reduce time to resolve. Each problem/issue could require a different set of data to collect. This article contains a list of data to collect as well as pointers to Resolution Guides and references on how to collect the data.

 

Symptoms

 

  • What information should I collect to assist in troubleshooting prior to opening a case?

    • The goal of this document is to reduce the time spent on initial data collection and reduce time to resolve by providing a comprehensive list of what to collect or gather to troubleshoot an issue.

 

Solution

 

This section contains the following:

 

Caveats and tips:

  • traceoptions requires additional system resources to gather and store data:

    • Ensure that you have enough disk space when enabling traceoptions .

    • Gauge current system utilization before enabling traceoptions .

      • “ show chassis routing-engine ”

      • “ show security monitoring fpc <spc-slot> ” (use slot 0 for branch)

    • Instead of using “ flag all ,” you can flag specific areas of interest.

    • Delete all traceoptions that are not needed for immediate debugging.

    • Do not forget to remove traceoptions after data collection is completed. This can be done by deactivating or deleting the traceoptions configuration stanza that you previously added to activate traceoptions .

      For example, assume that you enabled traceoptions using the following configuration: 

      set chassis cluster traceoptions file cluster.tr size 5m files 5 world-readable
set chassis cluster traceoptions flag all

      To disable this traceoptions configuration, you can issue one of the following two commands (not both) and commit the changes: 

      deactivate chassis cluster traceoptions

      OR 

      delete chassis cluster traceoptions

  • To deactivate paging (output that stops at each page and requires you to press the Space bar), you can:

    • Run “ set cli screen-length 0 ” to apply for all commands for your sessions.

    • Add the “ | no-more ” option at the end of a command.

 

Data to Collect for all configurations:

Regardless of configuration, all cases will benefit from attaching session captures, request information output, and logs when you initially open the case. If you need to investigate an intermittent concern (for example slow transfers at peak hours), be sure to collect this data at the time of the problem.

All Configurations Background information

  1. Provide all SSH / Telnet session captures.

  2. Provide any available topology information.

  3. Provide a summary of how the device is being used (production, lab system, co-location, etc).

  4. Provide a summary of device history (new install, production for X months/years, other recent cases, etc).

  5. Provide a summary of any recent changes in the network or on the device.

Request support info

  1. Enter: request support information | save /var/log/rsi1.log .

  2. After step #1 completes, wait enough time to ensure that the condition you wish to address continues/appears before proceeding to the next step.

  3. Enter: request support information | save /var/log/rsi2.log .

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure the /var/log/ directory was properly archived, check the file size using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

 

See References section for the following:

 

Additional Data to Collect

In addition, collect the data shown below for the following issues:

Jump to:

Chassis Cluster
Traffic failing for a specific host / application
High CPU
OSPF
BGP
Multicast
ALG
UTM - Anti-Virus
UTM - Anti-Spam
UTM - Web Filtering
UTM - Content Filtering
IPSec - Route Based
IPSec - Policy Based
IPSec - Dynamic VPNM
IPSec- NCP Exclusive Remove Access Client Connections
IDP - Security Package Update
IDP - Attack Detection
ATP - Advanced-Anti-Malware File Inspection
ATP - Security-Intelligence

Chassis Cluster Show commands: 
set cli timestamp
show chassis fpc pic-status
show chassis cluster status
show chassis cluster interfaces
show chassis cluster statistics
show chassis cluster information
show chassis cluster ip-monitoring status
Logs -Each node: 
/var/log/messages
/var/log/jsrpd
/var/log/chassisd
Traceoptions:
Caveats 
set chassis cluster traceoptions file cluster.tr size 5m files 5 world-readable
set chassis cluster traceoptions flag all
Known Issues: PR Search
[ Back to Top ]

Traffic failing for a specific host / application: Show commands: 
show security flow session summary
show security flow session {source-prefix | destination-prefix | source-port |
destination-port} <ip-prefix> extensive
show security flow session session-identifier <session-id> (same output as above)

show security flow cp-session summary
show interface extensive
show arp no-resolve (for locally connected hosts)
traceroute <ip-prefix> (for failing host)
Logs:

  • None by default.

  • If security policy logs are enabled, check the configured log file for policy RT_FLOW events

Traceoptions:

Caveats 
set security flow traceoptions file flow.trace
set security flow traceoptions file size 5m
set security flow traceoptions file files 5
set security flow traceoptions flag basic-datapath
set security flow traceoptions flag packet-drops
set security flow traceoptions packet-filter hostinit source-prefix a.a.a.a/32
set security flow traceoptions packet-filter hostinit destination-prefix b.b.b.b/32
set security flow traceoptions packet-filter hostresp source-prefix b.b.b.b/32
set security flow traceoptions packet-filter hostresp destination-prefix c.c.c.c/32

 

  • a.a.a.a - source address for initial traffic (use "inside/local" address if using source NAT)

  • b.b.b.b - destination address for initial traffic and source address for return traffic

  • c.c.c.c - destination address for return traffic (use "outside/global" address if using SRC NAT)

Notes:

See the ALG sections for more application-specific details.

[ Back to Top ]

High CPU CLI commands: 
set cli timestamp
show chassis routing-engine
show system processes extensive
show system users
show system connections
show system statistics
show chassis forwarding
show security monitor fpc pic <SPC-slot> (use 0 for Branch platforms)
show security monitor performance spu
show security monitor performance sess
Logs:

None

Traceoptions:

None

[ Back to Top ]

OSPF
Show commands:

(If OSPF is running in a routing instance, specify which instance where applicable) 

set cli timestamp
show ospf overview
show ospf database
show ospf neighbor detail
show ospf route
show ospf statistics
show ospf interface
show ospf log
show route protocol ospf
show route <x.x.x.x> extensive
show ospf database extensive
Logs: 
/var/log/messages
Traceoptions:

Caveats

(use below for inet.0 default instance) 

set protocols ospf traceoptions file ospf.tr
set protocols ospf traceoptions file size 5m
set protocols ospf traceoptions file files 5
set protocols ospf traceoptions flag all

(use below for routing instances) 

set routing-instances ospf-vr protocols ospf traceoptions file ospf-vr.tr
set routing-instances ospf-vr protocols ospf traceoptions file size 5m
set routing-instances ospf-vr protocols ospf traceoptions file files 5
set routing-instances ospf-vr protocols ospf traceoptions flag all
Known Issues: PR Search
[ Back to Top ]

BGP Show commands: 
set cli timestamp
show bgp summary
show bgp neighbor
show route advertising-protocol bgp <neighbor-address> extensive
show route receive-protocol bgp <neighbor-address>
show route forwarding-table
show route resolution unresolved
Logs: 
/var/log/messages
Traceoptions:

Caveats 
set protocols bgp traceoptions file bgp.tr
set protocols bgp traceoptions file size 5m
set protocols bgp traceoptions file files 5
set protocols bgp traceoptions flag all
set routing-instances bgp-vr protocols bgp traceoptions file bgp.tr
set routing-instances bgp-vr protocols bgp traceoptions file size 5m
set routing-instances bgp-vr protocols bgp traceoptions file files 5
set routing-instances bgp-vr protocols bgp traceoptions flag all
Known Issues: PR Search
[ Back to Top ]

Multicast Show commands: 
show multicast route
show multicast statistics
show multicast sessions
show multicast usage
show multicast interface
show multicast next-hops
show multicast rpf summary
show interface <if-name> extensive
show igmp group detail
show igmp statistics
show igmp interface detail
show pim statistics
show pim neighbors
show pim rps detail
show pim join extensive
show pim bootstrap
show msdp source-active
show msdp detail
show msdp statistics
show route
Logs: 
/var/log/messages
Traceoptions:

Caveats 
set routing-options multicast traceoptions file mcast.tr
set routing-options multicast traceoptions file size 5m
set routing-options multicast traceoptions file files 5
set routing-options multicast traceoptions flag all
Known Issues: PR Search
[ Back to Top ]
ALG CLI commands: 
set cli timestamp
show security alg status
show security alg <alg-name>

   [obtain all sub-commands for the ALG in question, use “?” to view options] 

show security resource-manager summary
show security resource-manager resource active
show security resource-manager resource active <number>
show security resource-manager group active
show security resource-manager group active <number>
show security flow gate
Logs: 
none
Traceoptions:

Caveats 
set security traceoptions file alg-sec.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all
set security alg <alg-type> traceoptions flag all
set security flow traceoptions file alg-flow.tr
set security flow traceoptions file size 5m
set security flow traceoptions file files 5
set security flow traceoptions file world-readable
set security flow traceoptions flag basic-datapath
set security flow traceoptions flag packet-drops
set security flow traceoptions packet-filter alginit source-prefix a.a.a.a/32
set security flow traceoptions packet-filter alginit destination-prefix b.b.b.b/32
set security flow traceoptions packet-filter algresp source-prefix b.b.b.b/32
set security flow traceoptions packet-filter algresp destination-prefix c.c.c.c/32
Notes:

See "Traffic failing for a specific host/application" for an example of packet filters for flow traceoptions.

Known Issues: PR Search
[ Back to Top ]

UTM - Anti-Virus CLI commands: 
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm anti-virus status detail
show security utm anti-virus statistics
show chassis routing-engine
show system processes extensive
Updating Full AV database:
request security utm anti-virus kaspersky-lab-engine pattern-update
Updating Express AV database:
request security utm anti-virus juniper-express-engine pattern-update
Updating Sophos AV database:
request security utm anti-virus sophos-engine pattern update
Logs: 
/var/log/utmd
/var/log/utmd-av
Traceoptions:

Caveats 
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile anti-virus traceoptions flag all
set security traceoptions file av.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all
Known Issues: PR Search
[ Back to Top ]

UTM - Anti-Spam CLI commands: 
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm anti-spam status
show security utm anti-spam statistics
show chassis routing-engine
show system processes extensive
Logs: 
/var/log/utmd
/var/log/utmd-as
Traceoptions:

Caveats 
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile anti-spam traceoptions flag all
set security traceoptions file as.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all
Known Issues: PR Search
[ Back to Top ]

UTM - Web Filtering CLI commands: 
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm web-filtering status
show security utm web-filtering statistics
show chassis routing-engine
show system processes extensive
Logs: 
/var/log/utmd
/var/log/utmd-wf
Traceoptions:

Caveats 
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile web-filtering traceoptions flag all
set security traceoptions file wf.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all
Known Issues: PR Search
[ Back to Top ]

UTM - Content Filtering CLI commands: 
set cli timestamp
show system licenses
show security utm status
show security utm session
show security utm content-filtering statistics
Logs: 
/var/log/utmd
Traceoptions:

Caveats 
set security utm traceoptions flag all
set security utm application-proxy traceoptions flag all
set security utm feature-profile content-filtering traceoptions flag all
set security traceoptions file cf.tr
set security traceoptions file size 5m
set security traceoptions file files 5
set security traceoptions file world-readable
set security traceoptions flag all
Known Issues: PR Search
[ Back to Top ]

IPsec VPN -
Route-Based 		Show commands: 
show security ike security-association
show security ike security-association index <#> detail
show security ipsec security-association
show security ipsec security-association index <#> detail
show security ipsec statistics
show security ipsec statistics index <#>
show security ipsec next-hop-tunnels
monitor interface st0.x
show interfaces extensive st0.x
show security flow session tunnel
show route
show security pki local-cert detail
show security pki ca-cert detail
show security pki crl detail
Logs: 
/var/log/kmd*

/var/tmp/kmd* (SRX 1400 and higher)
Traceoptions:

Caveats 
set security ike traceoptions file vpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing
Notes:

If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and setup packet filters for outer ESP traffic as well as inner application/host traffic.

[ Back to Top ]

IPsec VPN - Policy-Based Show commands: 
show security ike security-association
show security ike security-association index <#> detail
show security ipsec security-association
show security ipsec security-association index <#> detail
show security ipsec statistics
show security ipsec statistics index <#>
show security ipsec next-hop-tunnels
show security flow session tunnel
IF PKI certs are used:
show security pki local-cert detail
show security pki ca-cert detail
show security pki crl detail
show security policies detail
show log /var/etc/policy.id
Logs: 
/var/log/kmd*
/var/tmp/kmd* (SRX 1400 and higher)
Traceoptions:

Caveats 
set security ike traceoptions file vpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing
Notes:

If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and setup packet filters for outer ESP traffic as well as inner application/host traffic.

[ Back to Top ]

IPsec - Dynamic VPN Show commands: 
show security ike security-association
show security ike security-association index <number> detail
show security ike active-peer
show security ipsec security-association
show security ipsec security-association index <id>
show security ipsec statistics
show security dynamic-vpn client version
show security dynamic-vpn users detail
show system license
Logs: SRX: 
/var/log/kmd
/var/log/httpd.log
/var/log/authd

Pulse client:  
File > Logs > Log level > (detailed / normal)
File > Logs > Save as > <filename>

Traceoptions:

Caveats 
set system processes general-authentication-servic traceoptions file dynvpn-auth.tr
set system processes general-authentication-servic traceoptions file size 5m
set system processes general-authentication-servic traceoptions file files 5
set system processes general-authentication-servic traceoptions file world-readable
set system processes general-authentication-servic traceoptions flag all
set security ike traceoptions file dynvpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing
Notes:

If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and setup packet filters for outer ESP traffic as well as inner application/host traffic.

The Pulse client version is also helpful for troubleshooting. For more info on how to get it, see KB22857 - [juniper.net] [SRX] How to find the version information of the Junos Pulse Desktop client used for SRX Dynamic VPN connections .

Known Issues: PR Search
[ Back to Top ]

IPSec VPN -
NCP Exclusive Remote Access Client Connections 		Show commands: 
show security ike active-peer
show security ike security-association
show security ike security-association index <#> detail
show security ipsec security-association
show security ipsec security-association index <#> detail
show security ipsec tunnel-events-statistics
show security ipsec statistics
show security ipsec statistics index <#>
show interfaces extensive st0.x
show security flow session tunnel
show route
show security pki local-cert detail
show security pki ca-cert detail
show security pki crl detail
show network-access requests statistics
show system license
Logs: 
/var/log/kmd*
Traceoptions:

Caveats 
set security ike traceoptions file vpn.tr size 5m files 5 world-readable
set security ike traceoptions flag ike
set security ike traceoptions flag general
set security ipsec traceoptions flag security-associations
set security ipsec traceoptions flag packet-drops
set security ipsec traceoptions flag packet-processing
Notes:

If tunnels are up but traffic not passing, see section “Traffic failing for a specific host/application” and setup packet filters for outer ESP traffic as well as inner application/host traffic.

[ Back to Top ]

IDP - Security Package Update Show commands: 
show security idp security-package-version
show security idp status
show security idp memory
request security idp security-package download
request security idp security-package download status
request security idp security-package install
request security idp security-package install status
Logs: 
show log messages
show log idpd
show log idp-traceoptions
Traceoptions:

Caveats 
edit security idp traceoptions
set file idp-traceoptions
set flag all
set level all
edit security flow traceoptions
set file flow-trace
set flag basic-datapath
set flag packet-drops
set packet-filter 1 …
ALWAYS CONFIGURE PACKET-FILTERS
[ Back to Top ]
 
IDP - Attack Detection Show commands: 
show security idp security-package-version
show security idp status
show security idp counters flow
show security idp counters application-identification
show security idp counters flow
show security idp counters log
show security idp counters packet
show security idp memory
show security idp application-statistics
show security idp attack table
Notes:

Latency/Performance:

Change IDP policy to one of the predefined template IDP policies, like Recommended Policy, to verify if this is a customer IDP policy issue.

Datasheet benchmarks are based on IDP Recommended Policy.

False Positives/Negatives:

Gather:

  1. Packet capture of the False Positive/Negative

  2. IDP signature that is causing issue

  3. show security idp security-package-version

  4. show configuration security idp | display set

Contact [email protected] with above information.

[ Back to Top ]

ATP - Advanced-Anti-Malware File Inspection CLI commands: 
set cli timestamp
show services advanced-anti-malware status
request services advanced-anti-malware diagnostic <url> detail
request services advanced-anti-malware data-connection test start <packet-size>
request services advanced-anti-malware data-connection test status
show security pki local-certificate detail
show security pki ca-certificate detail
show services advanced-anti-malware statistics
show services advanced-anti-malware profile
show services advanced-anti-malware policy
show services ssl proxy statistics
Logs: 
/var/log/messages
/var/log/aamw_traceoptions
Traceoptions:

Caveats 
edit services advanced-anti-malware traceoptions
set file size 20m
set file files 10
set file aamw_traceoptions
set flag all
set level all
Notes:

Ensure the clock is correct or configure NTP.

vSRX requires a Sky ATP license installed on device for ATP Cloud enrollment.

[ Back to Top ]

ATP - Security-Intelligence CLI commands: 
set cli timestamp
request services security-intelligence download status
show configuration services security-intelligence url
show services advanced-anti-malware status
show services security-intelligence update status
show services security-intelligence statistics
show services security-intelligence category summary
show services security-intelligence statistics
show security pki local-certificate detail
show security pki ca-certificate detail
show services ssl proxy statistics
Logs: 
/var/log/messages
/var/log/secintel_traceoptions
Traceoptions:

Caveats 
edit services security-intelligence traceoptions
set file size 20m
set file files 10
set file secintel_traceoptions
set flag all
set level all
Notes:

Ensure the clock is correct or configure NTP.

vSRX requires a Sky ATP license installed on device for ATP Cloud enrollment.

[ Back to Top ]
 

References:

How to:

 

Resolution Guides and Troubleshooting Checklists:

JTAC Certified step-by-step troubleshooting flowcharts and articles

Resolution Guides and Articles - SRX - NAT UTM (Unified Threat Management) Basic Troubleshooting Checklist for SRX
[SRX] Troubleshooting Checklist - DHCP
[SRX] Troubleshooting Checklist - RADIUS
 

Technical Bulletins:   SRX Series (log in to see more)

For more information on Technical Bulletins, see KB9890 - Subscribe to email notifications for Technical Bulletins (TSB), Security Advisories (JSA), Problem Reports (PR), Knowledge Base (KB) articles and more [juniper.net] .

 

Modification History

  • 2021-05-18: Article checked for accuracy; article found valid; broken links corrected

  • 2020-07-02: Added ATP Advanced Anti Malware File Inspection and Sec-Intel

  • 2020-03-30: Removed reference to outdated J-Series article.

  • 2017-05-01: Added-IPsec VPN -NCP Exclusive Remote Access Client Connection.

Related Information