Description

Data Collection and Troubleshooting Guides can help with issue investigation as well as reducing resolution time. Each issue might require a different set of data to collect. This article contains a list of data to collect as well as pointers to Resolution Guides and references on how to collect the data.

Symptoms

On an EX Series Switch or a QFX Series Switch, what information should I collect to assist in troubleshooting, before opening a case?

The goal of this document is to reduce the time spent on initial data collection and reduce resolution time by providing a comprehensive list of what to collect or gather to troubleshoot an issue.

Note: Your feedback is welcome on the format of the Data Collection Checklist.

Solution

This section contains the following:

Caveats and tips:

  • Trace options require additional system resources to gather and store data:

    • Please ensure you have enough disk space when enabling trace options
    • Gauge current system utilization before enabling trace options
      • "show chassis routing-engine "
    • Instead of using "flag all" you can flag specific areas of interest
    • Delete all trace options that are not needed for immediate debugging
    • Don't forget to remove trace options after data collection is completed

  • To deactivate paging (output stopping each page requiring you to press the space bar) you can:

    • Run "set cli screen-length 0" for this to apply for all commands for your sessions
    • Add the "| no-more" option at the end of a command

Step 1. Data to Collect for all configurations:

Regardless of configuration, all cases will benefit by attaching the session captures, request information output, and logs when initially opening the case. If you need to investigate an intermittent concern (for example slow transfers at peak hours) please be sure to collect this data at the time of the problem.

All Configurations Background information
  1. Provide all SSH / Telnet session captures
  2. Provide any available topology information
  3. Summary of how the device is being used (production, lab system, co-location, etc)
  4. Summary of device history (new install, production for X months/years, other recent cases, etc)
  5. Summary of any recent changes in the network or on the device
Request support info
  1. Enter: request support information | save /var/log/rsi1.log
  2. Once step #1 is completed, wait enough time to ensure that the condition you wish to address continues/appears before proceeding to the next step.
  3. Enter: request support information | save /var/log/rsi2.log
LogsArchive the /var/log/ contents:
 
file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure the /var/log/ directory was properly archived, check the file size using the command: file list /var/tmp/CURRENT-DATE.tgz detail

See the References section for the following:

Step 2. Additional Data to Collect:

In addition, collect the data shown below for the following issues:

Jump to:

Aggregated - LACP
BFD
BGP
CoS
DHCP
dot1X
ERP (protection-group)
EX8200 Fabric Plane
Firewall Filters
High CPU
IPv6
ISIS
L2 VPN
L3 VPN
MPLS
Multicast
NTP
OAM
OSPF
PoE
RIP
SNMP
Spanning Tree
Virtual Chassis
VLAN
VRRP


Aggregated link
(LACP) 		CLI commands:show interfaces terse | match ae
show interfaces statistics
show ethernet-switching table | match xx-x/x/x
show lacp interfaces
show lacp statistics interfaces xx-x/x/x
show interface extensive
Logs:/var/log/messages
Traceoptions:set protocols lacp traceoptions file lacp.tr
set protocols lacp traceoptions file size 5m
set protocols lacp traceoptions file files 5
set protocols lacp traceoptions flag all
Questions:- What device is connected at the other end?
- Is link protection enabled?
- What are the troubleshooting steps so far?
- Did the issue start after any kind of device or config change?

[ Back to Top ]

BFD CLI commands:show bfd session brief
show bfd session
show bfd session summary
show bfd session detail
show bfd session discriminator brief
show bfd session discriminator detail address
Logs:/var/log/messages
Traceoptions:set protocols bfd traceoptions file bfd.tr
set protocols bfd traceoptions file size 5m
set protocols bfd traceoptions file files 5
set protocols bfd traceoptions flag all

[ Back to Top ]

BGP Show commands:set cli timestamp
show bgp summary
show bgp group
show bgp neighbor
show route next-hop
show route protocol bgp all
show route receive-protocol bgp < peer address> extensive
show route advertising-protocol bgp extensive
show route forwarding-table
show route resolution unresolved
show route community
show route damping [history|decayed|surpressed]
show route aspath-regex
monitor traffic interface XX-X/X/X size 4096 detail
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols bgp traceoptions file bgp.tr
set protocols bgp traceoptions file size 5m
set protocols bgp traceoptions file files 5
set protocols bgp traceoptions flag all
set routing-instances bgp-ri protocols bgp traceoptions file bgp.tr
set routing-instances bgp-ri protocols bgp traceoptions file size 5m
set routing-instances bgp-ri protocols bgp traceoptions file files 5
set routing-instances bgp-ri protocols bgp traceoptions flag all

[ Back to Top ]

CoS CLI commands:show interface xx-x/x/x detail
show interface queue xx-x/x/x both-ingress-egress
show interfaces XX-X/X/X extensive

show class-of-service interfaces XX-X/X/X
show class-of-service code-point-aliases
show class-of-service code-point-aliases dscp
show class-of-service code-point-aliases inet-prec
show class-of-service code-point-aliases exp
show class-of-service forwarding-class
show class-of-service classifier [name]
show class-of-service scheduler-map
show class-of-service rewrite-rule
show class-of-service drop-profile
show class-of-service forwarding-table
show class-of-service forwarding-table classifier mapping
show class-of-service forwarding-table scheduler-map
Logs:/var/log/messages
Traceoptions:set class-of-service traceoptions file class-of-service.tr
set class-of-service traceoptions file size 5m
set class-of-service traceoptions file files 5
set class-of-service traceoptions flag all

[ Back to Top ]

DHCP CLI commands:show configuration | display set | match dhcp
show system services dhcp binding detail
show system services dhcp client
show system services dhcp pool detail
show interfaces terse
show dhcp snooping binding vlan
show dhcp snooping binding interface xx-x/x/x
monitor traffic interface xx-x/x/x
Logs:/var/log/messages
Traceoptions:set system processes dhcp-service traceoptions file DHCP-trace-logs.txt
set system processes dhcp-service traceoptions flag all
set system processes dhcp-service traceoptions level all
Questions:- Any troubleshooting steps so far?
- Confirm whether the discover packet is reaching the switch.
- Is the switch replying with an offer packet?
- Any packet captures taken at the interface ends?

[ Back to Top ]

dot1X CLI commands:show dot1x authentication-bypassed-users
show dot1x authentication-failed-users
show dot1x static-mac-address
show dot1x statistics
show dot1x interface brief
show dot1x interface detail
Logs:/var/log/messages
Traceoptions:set protocols dot1x traceoptions file dot1x.tr
set protocols dot1x traceoptions file size 5m
set protocols dot1x traceoptions file files 5
set protocols dot1x traceoptions flag all

[ Back to Top ]

ERP (protection-group) CLI commands:show configuration protocols protection-group
show ethernet-switching interfaces xx-x/x/x
show protection-group ethernet-ring interface brief
show protection-group ethernet-ring interface detail
show protection-group ethernet-ring node-state
show protection-group ethernet-ring statistics
show protection-group ethernet-ring configuration
Logs:/var/log/messages
Traceoptions:set protocols protection-group traceoptions file protection-group.tr
set protocols protection-group traceoptions file size 5m
set protocols protection-group traceoptions file files 5
set protocols protection-group traceoptions flag all

[ Back to Top ]

EX 8200
Fabric
Plane

 		CLI commands:General:
show chassis environment cb
show chassis alarms

Fabric related:
show chassis hardware
show chassis fabric summary
show chassis fabric map
show chassis fabric fpc
show chassis fabric plane
show class-of-service fabric statistics (take multiple outputs to see counters incrementing)
show pfe statistics traffic (take multiple outputs to see counters incrementing)
 
Logs:show log messages
show log chassisd
Hidden Commands:

Caveats 		NOTE: These commands are not officially supported, and are hidden, albeit are good for troubleshooting Fabric Plane issues.

Please exercise caution when executing this command, as incorrect syntax may lead to problems.

show chassis fabric statistics detail
show chassis fabric asic-configuration plane

 

Questions:- Have the SCBs for which the alarms have been generated had any previous history of any issues?
- What kind of troubleshooting has already been done to mitigate the problem? Please provide detailed steps.
- Were there any recent changes to the configuration?
- Were there any changes to the hardware setup, or was new hardware introduced?
- Were there any events that occurred just before the log message that may have triggered it? This may include events such as routing engine switchover etc.
- Was a software upgrade/downgrade done recently?

[ Back to Top ]

Firewall Filters CLI commands:show interfaces filters
show firewall
show firewall filter
show firewall log
show policy conditions
Logs:/var/log/messages
Traceoptions:None

[ Back to Top ]

High CPU CLI commands:set cli timestamp
show chassis routing-engine (multiple snapshots, atleast 5)
show system processes extensive (multiple snapshots atleast 5)
show system users
show system connections
show system statistics
show task memory detail
show task memory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state
Logs:/var/log/messages
Traceoptions:user@router# show routing-options
traceoptions {
file routing-trace size 10m files 20 world-readable;
flag task;
flag state;
flag timer;
}

[ Back to Top ]

IPv6 CLI commands:show interface terse
show route table inet6
show ipv6 neighbor
show interface lo0 extensive | display xml
Logs:/var/log/messages
Traceoptions:None

[ Back to Top ]

ISIS Show commands:set cli timestamp
show isis adjacency
show isis adjacency extensive
show isis interface
show isis hostname
show isis spflog
show isis statistics
show isis route
show route protocol isis
show isis database
show isis database detail
show bfd session
show route forwarding-table
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols isis traceoptions file rip.tr
set protocols isis traceoptions file size 5m
set protocols isis traceoptions file files 5
set protocols isis traceoptions flag all
set routing-instances isis-vr protocols isis traceoptions file isis.tr
set routing-instances isis-vr protocols isis traceoptions file size 5m
set routing-instances isis-vr protocols isis traceoptions file files 5
set routing-instances isis-vr protocols isis traceoptions flag all

[ Back to Top ]

L2 VPN CLI commands:show l2vpn connections extensive
show route table
show route advertising-protocol bgp x.x.x.x
show route receive-protocol bgp x.x.x.x
show route table mpls.o
show route forwarding-table family mpls
Logs:/var/log/messages
Traceoptions:None

[ Back to Top ]

L3 VPN CLI commands:show route table
show route table hidden
show route forwarding-table vpn

ping x.x.x.x routing-instance
traceroute x.x.x.x routing-instance
telnet x.x.x.x routing-instance
ping mpls l3vpn prefix

show route advertising-protocol bgp x.x.x.x
show route receive-protocol bgp x.x.x.x
show arp

OSPF VPN commands
show ospf interface instance
show ospf neighbor instance
show ospf database instance
Logs:/var/log/messages
Traceoptions:None

[ Back to Top ]

MPLS Show commands:show mpls interface
show mpls lsp
show mpls lsp extensive
show mpls lsp ingress
show mpls lsp transit
show ted database
RSVP
show rsvp interface
show rsvp neighbor
show rsvp session
show rsvp session ingress
show rsvp version
show route table mpls.0
show route table inet.3
LDP
show ldp neighbors
show ldp session
show ldp database [session peer]
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols mpls|ldp|rsvp traceoptions file mpls.tr
set protocols mpls|ldp|rsvp traceoptions file size 5m
set protocols mpls|ldp|rsvp traceoptions file files 5
set protocols mpls|ldp|rsvp traceoptions flag all

set protocols mpls label-switched-path traceoptions file mpls-lsp.tr
set protocols mpls label-switched-path traceoptions file size 5m
set protocols mpls label-switched-path traceoptions file files 5
set protocols mpls label-switched-path traceoptions flag all

[ Back to Top ]

Multicast Show commands:General
show multicast route
show multicast sessions
show multicast usage
show multicast interface
show multicast next-hops
show multicast rpf summary
show interface extensive
IGMP
show igmp group detail
show igmp statistics
show igmp interface detail
PIM
show pim statistics
show pim neighbors
show pim rps detail
show pim join extensive
show pim bootstrap
show pim source detail
MSDP
show msdp source-active
show msdp detail
show msdp statistics
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols pim|igmp|msdp traceoptions file multicast-proto.tr
set protocols pim|igmp|msdp traceoptions file size 5m
set protocols pim|igmp|msdp traceoptions file files 5
set protocols pim|igmp|msdp traceoptions flag all

enable traceoptions for either of PIM, IGMP, MSDP

[ Back to Top ]

NTP CLI commands:show ntp associations
show ntp status
show version
show system uptime
Logs:/var/log/messages
Traceoptions:none

[ Back to Top ]

OAM CLI commands:show configuration protocols oam
show oam ethernet link-fault-management brief
show oam ethernet link-fault-management detail
show oam ethernet link-fault-management vlan detail

show oam ethernet connectivity-fault-management adjacencies
show oam ethernet connectivity-fault-management interfaces brief
show oam ethernet connectivity-fault-management interfaces detail
show oam ethernet connectivity-fault-management interfaces vlan
Logs:/var/log/messages
Traceoptions:set protocol oam ethernet link-fault-management traceoptions file link-fault.tr
set protocol oam ethernet link-fault-management traceoptions file size 5m
set protocol oam ethernet link-fault-management traceoptions file files 5
set protocol oam ethernet link-fault-management traceoptions flag all

set protocol oam ethernet connectivity-fault-management traceoptions file connect-fault.tr
set protocol oam ethernet connectivity-fault-management traceoptions file size 5m
set protocol oam ethernet connectivity-fault-management traceoptions file files 5
set protocol oam ethernet connectivity-fault-management traceoptions flag all

[ Back to Top ]

OSPF
Show commands:

(If OSPF is running in a routing instance, specify which instance where applicable)

set cli timestamp
show ospf overview
show ospf database
show ospf neighbor detail
show ospf route
show ospf statistics
show ospf interface (brief | detail | extensive)
show ospf log
show route protocol ospf
show route extensive
show ospf database summary
show ospf database extensive
show bfd session

Logs:/var/log/messages
Traceoptions:

Caveats 		(use below for inet.0 default instance)
set protocols ospf traceoptions file ospf.tr
set protocols ospf traceoptions file size 5m
set protocols ospf traceoptions file files 5
set protocols ospf traceoptions flag all

(use below for routing instances)
set routing-instances ospf-vr protocols ospf traceoptions file ospf-vr.tr
set routing-instances ospf-vr protocols ospf traceoptions file size 5m
set routing-instances ospf-vr protocols ospf traceoptions file files 5
set routing-instances ospf-vr protocols ospf traceoptions flag all
Known Issues:NA

[ Back to Top ]

PoE CLI commands:show configuration poe
show poe controller
show poe interface
show interfaces terse
show poe notification-control
show poe telemetries interface xx-x/x/x all
Logs:/var/log/messages
Questions:- Did the device face any power outage?
- Are the poe devices currently connected? If so, in which ports?
- Details of the POE devices connected?

[ Back to Top ]

RIP Show commands:set cli timestamp
show rip neighbor
show route protocol rip
show route advertising-protocol rip extensive
show route receive-protocol rip extensive
show rip statistics
show route forwarding-table
show route resolution unresolved
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols rip traceoptions file rip.tr
set protocols rip traceoptions file size 5m
set protocols rip traceoptions file files 5
set protocols rip traceoptions flag all
set routing-instances rip-vr protocols rip traceoptions file rip.tr
set routing-instances rip-vr protocols rip traceoptions file size 5m
set routing-instances rip-vr protocols rip traceoptions file files 5
set routing-instances rip-vr protocols rip traceoptions flag all

[ Back to Top ]

SNMP CLI commands:set cli timestamp
show configuration snmp
show snmp statistics # Multiple time with polling activity.
show snmp mib walk
show interface lo0 extensive
show system processes extensive | no-more # Multiple time with polling activity.
show chassis routing-engine | no-more # Multiple time with polling activity.
show system commit | no-more
show system statistics # Multiple time with polling activity.
show system buffers # Multiple time with polling activity.
show system core-dumps
show route
ping source count 500 rapid
show pfe statistics traffic # Multiple time with polling activity.
show interface extensive | no-more
Logs:/var/log/messages
Traceoptions:set snmp traceoptions file snmp files 10 size 50m
set snmp traceoptions flag all
 Questions:What troubleshooting steps have you done?
Were there any recent changes? Upgrade, topology, config
Are there any specific SNMP MIBs that are not responding, or are all the SNMP MIBs?
Have you taken any pcaps to see if the switch is sending the corresponding traffic at all?

[ Back to Top ]

STP
(Spanning
Tree
Protocol)
 		CLI commands:show configuration protocols
show spanning-tree bridge brief
show spanning-tree bridge detail
show spanning-tree statistics
show spanning-tree interface
show ethernet-switching mac-learning-log
Logs:/var/log/messages
Traceoptions:set protocols stp traceoptions file stp.tr
set protocols stp traceoptions file size 5m
set protocols stp traceoptions file files 5
set protocols stp traceoptions flag all
Questions:- What kind of spanning tree flavor is running?
- Is the customer using other vendor devices within the same spanning tree?
- Is the customer using the same spanning tree flavor on all the switches?
- Any recent configuration changes related to cost, bridge priority, storm control, or BPDU block?
- Is there any hub connected to any of the switches?

[ Back to Top ]

VLAN CLI commands:show configuration vlans
show vlans brief
show vlans
show vlans summary
show interfaces terse | match in
Logs:/var/log/messages
Traceoptions:none
Questions:Did the issue start after any configuration/ device change?
What are the devices connected?

[ Back to Top ]


VRRP CLI commands:show configuration protocols vrrp
show vrrp brief
show vrrp detail
show vrrp interface xx-x/x/x
show vrrp track summary
show vrrp track detail
Logs:/var/log/messages
Traceoptions:set protocol vrrp traceoptions file vrrp.tr
set protocol vrrp traceoptions file size 5m
set protocol vrrp traceoptions file files 5
set protocol vrrp traceoptions flag all
Questions:- Any new device was added to the network?
- What are the troubleshooting steps so far?
- Any recent configuration changes?
- Any spanning tree-related issues?
- Are the hello packets being received on the other end?

[ Back to Top ]

Virtual Chassis CLI commands:show version
show configuration virtual-chassis
show virtual-chassis status
show virtual-chassis vc-port all-members
show virtual-chassis mode
show virtual-chassis protocol interface
show virtual-chassis protocol statistics
Logs:/var/log/messages
Traceoptions:set virtual-chassis traceoptions file virtual-chassis.tr
set virtual-chassis traceoptions file size 5m
set virtual-chassis traceoptions file files 5
set virtual-chassis traceoptions flag all
Questions:- Any troubleshooting steps so far?
- Did the issue begin after any device or config change?
- Is the issue happening again and again?


[ Back to Top ]

References:

How to:

Resolution Guides : KB9936 - JTAC Certified step-by-step troubleshooting flowcharts and articles [juniper.net]

Technical Bulletins:   EX Series (login to see more)

For more information on Technical Bulletins, see KB9890 - How do I subscribe to a technical bulletin so I can I get email alerts regarding product issues, new product release announcements and security or safety issues [juniper.net]?

Modification History

02/14/2020: Article reviewed for accuracy; minor, non-technical edits.
09/10/2024: Corrected minor grammar errors | Added SNMP questions to get some background

10/20/2024: Added trace options for DHCP

Related Information