Description

Data Collection and Troubleshooting Guides can help with issue investigation as well as reduce resolution time. Each issue may require a different set of data to collect. This article contains a list of what to collect, as well as pointers to Resolution Guides and references on how to collect the data.


 

Symptoms

What information should I collect to assist in troubleshooting, prior to opening a case?

The goal of this document is to reduce the time spent on initial data collection and reduce resolution time by providing a comprehensive list of what to collect or gather to troubleshoot an issue.

 

Solution

 

This section contains the following:


Caveats and tips:
  • Traceoptions require additional system resources to gather and store data:
    • Please ensure you have enough disk space when enabling traceoptions
    • Gauge current system utilization before enabling traceoptions
      • “ show chassis routing- engine ”
    • Instead of using “flag all” you can flag specific areas of interest
    • Delete all traceoptions that are not needed for immediate debugging
    • Don’t forget to remove traceoptions after data collection is completed
  • To deactivate paging (output stopping each page requiring you to press space bar) you can:
    • Run “set cli screen-length 0” for this to apply for all commands for your sessions
    • Add the “| no-more” option at the end of a command

 

Step 1.  Data to Collect for all configurations:

Regardless of configuration, all cases will benefit by attaching the session captures, request information output, and logs when initially opening the case. If you need to investigate an intermittent concern (for example slow transfers at peak hours) please be sure to collect this data at the time of the problem.

 

All Configurations Background information
  1. Provide all SSH / Telnet session captures
  2. Provide any available topology information
  3. Summary of how the device is being used (production, lab system, co-location, etc)
  4. Summary of device history (new install, production for X months/years, other recent cases, etc)
  5. Summary of any recent changes in the network or on the device
Request support info
  1. Enter: request support information | save /var/log/rsi1.log
  2. Once step #1 completes, wait enough time to ensure that the condition you wish to address continues/appears before proceeding to the next step.
  3. Enter: request support information | save /var/log/rsi2.log
LogsArchive the /var/log/ contents:
 
file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure the /var/log/ directory was properly archived, check the file size using the command: file list /var/tmp/CURRENT-DATE.tgz detail

See References section for the following:  


Step 2.  Additional Data to Collect:

In addition, collect the data shown below for the following issues:

Jump to:
BGP
OSPF
Multicast
MPLS
L3 VPN
L2 VPN
CoS
Firewall Filters
SNMP
High CPU
1588
SyncE

BGP Show commands:set cli timestamp
show bgp summary
show bgp group 
show bgp neighbor
show route next-hop database
show route protocol bgp
show route protocol bgp terse
show route receive-protocol bgp
show route advertising-protocol bgp
show route advertising-protocol bgp <neighbor-address> extensive
show route receive-protocol bgp <neighbor-address>
show route forwarding-table
show route resolution unresolved
show route hidden
show route community <reg-ex>
show route terse community-name <name>
show route damping
show route aspath-regex "regex"
show route damping [history|decayed|surpressed]
monitor traffic interface XX-X/X/X size 4096 detail
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols bgp traceoptions file bgp.tr
set protocols bgp traceoptions file size 5m
set protocols bgp traceoptions file files 5
set protocols bgp traceoptions flag all
set routing-instances bgp-ri protocols bgp traceoptions file bgp.tr
set routing-instances bgp-ri protocols bgp traceoptions file size 5m
set routing-instances bgp-ri protocols bgp traceoptions file files 5
set routing-instances bgp-ri protocols bgp traceoptions flag all
[ Back to Top ]

OSPF
Show commands:

(If OSPF is running in a routing instance, specify which instance where applicable)

set cli timestamp
show ospf overview
show ospf database
show ospf neighbor detail
show ospf route
show ospf statistics
show ospf interface
show ospf log
show route protocol ospf
show route <x.x.x.x> extensive
show ospf database summary 
show ospf database extensive
show bfd session 

Logs:/var/log/messages
Traceoptions:

Caveats 		(use below for inet.0 default instance)
set protocols ospf traceoptions file ospf.tr
set protocols ospf traceoptions file size 5m
set protocols ospf traceoptions file files 5
set protocols ospf traceoptions flag all

(use below for routing instances)
set routing-instances ospf-vr protocols ospf traceoptions file ospf-vr.tr
set routing-instances ospf-vr protocols ospf traceoptions file size 5m
set routing-instances ospf-vr protocols ospf traceoptions file files 5
set routing-instances ospf-vr protocols ospf traceoptions flag all
[ Back to Top ]

Multicast Show commands:General
show multicast route   
show multicast statistics
show multicast sessions
show multicast usage
show multicast interface
show multicast next-hops
show multicast rpf summary
show interface <if-name> extensive
IGMP
show igmp group detail
show igmp statistics
show igmp interface detail
PIM
show pim statistics
show pim neighbors
show pim rps detail
show pim join extensive
show pim bootstrap
show pim source detail
MSDP
show msdp source-active
show msdp detail
show msdp statistics
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols pim|igmp|msdp traceoptions file multicast-proto.tr
set protocols pim|igmp|msdp traceoptions file size 5m
set protocols pim|igmp|msdp traceoptions file files 5
set protocols pim|igmp|msdp traceoptions flag all

enable traceoptions for either of PIM, IGMP, MSDP
[ Back to Top ]

MPLS Show commands:show mpls interface 
show mpls lsp
show mpls lsp extensive
show mpls lsp ingress
show mpls lsp transit
show ted database
RSVP
show rsvp interface
show rsvp neighbor
show rsvp session
show rsvp session ingress
show rsvp version
show route table mpls.0
show route table inet.3
LDP
show ldp neighbors
show ldp session
show ldp database [session peer]
Logs:/var/log/messages
Traceoptions:

Caveats 		set protocols mpls|ldp|rsvp traceoptions file mpls.tr
set protocols mpls|ldp|rsvp traceoptions file size 5m
set protocols mpls|ldp|rsvp traceoptions file files 5
set protocols mpls|ldp|rsvp traceoptions flag all

set protocols mpls label-switched-path <lsp-name> traceoptions file mpls-lsp.tr
set protocols mpls label-switched-path <lsp-name> traceoptions file size 5m 
set protocols mpls label-switched-path <lsp-name> traceoptions file files 5
set protocols mpls label-switched-path <lsp-name> traceoptions flag all
[ Back to Top ]


L3 VPN CLI commands:show route table <vpn-a>  
show route table <vpn-a> hidden
show route forwarding-table vpn <vpn-a>

ping x.x.x.x routing-instance <vpn-a>
traceroute x.x.x.x routing-instance <vpn-a>
telnet x.x.x.x routing-instance <vpn-a>
ping mpls l3vpn <vpn-a> prefix <prefix>

show route table bgp.l3vpn.0 
show route advertising-protocol bgp x.x.x.x
show route receive-protocol bgp x.x.x.x
show arp

OSPF VPN commands
show ospf interface instance <vpn-a>
show ospf neighbor instance <vpn-a>
show ospf database instance <vpn-a>
Logs:None
Traceoptions:None
[ Back to Top ]

L2 VPN CLI commands:Martini
show l2circuit connections extensive
show ldp neighbor detail
show ldp database detail
Logs:None
Traceoptions:None
[ Back to Top ]

CoS CLI commands:show interface xx-x/x/x detail   
show interface queue xx-x/x/x both-ingress-egress
show interfaces XX-X/X/X extensive

show class-of-service interfaces XX-X/X/X 
show class-of-service code-point-aliases
show class-of-service code-point-aliases dscp
show class-of-service code-point-aliases inet-prec
show class-of-service code-point-aliases exp
show class-of-service forwarding-class
show class-of-service classifier [name]
show class-of-service scheduler-map
show class-of-service rewrite-rule
show class-of-service drop-profile
show class-of-service forwarding-table
show class-of-service forwarding-table classifier mapping
show class-of-service forwarding-table scheduler-map
Logs:None
Traceoptions:None
[ Back to Top ]

Firewall Filters CLI commands:show interfaces filters
show firewall
show firewall filter <filter-name>
show firewall filter <name> prefix-action <psa-name> from 1 to 8
show firewall log
show log <log-file-name>
show policer
show interface policer XX-X/X/X
Logs:show log /var/tmp/ <sample file>
show log sampled
Traceoptions:None
[ Back to Top ]

IPv6 CLI commands:set show interface terse
show route table inet6
show ipv6 neighbor
show interface lo0 extensive | display xml
Logs:None
Traceoptions:None
[ Back to Top ]

SNMP CLI commands:set cli timestamp
show configuration snmp
show snmp statistics extensive # Multiple time with polling activity.
show snmp mib walk <oid>
show snmp mib walk ipfragfails
show interface lo0 extensive
show system processes extensive | no-more # Multiple time with polling activity.
show chassis routing-engine | no-more # Multiple time with polling activity.
show system commit | no-more
show system statistics # Multiple time with polling activity.
show system buffers # Multiple time with polling activity.
show system core-dumps
show route <ip of NMS server>
ping <ip of NMS server> source <snmp source ip> count 500 rapid show pfe statistics traffic # Multiple time with polling activity.
show interface extensive | no-more
show system process extensive | no-more
Logs:None
Traceoptions:set snmp traceoptions file snmp files 10 size 50m
set snmp traceoptions flag all
[ Back to Top ]

High CPU CLI commands:set cli timestamp
show chassis routing-engine (multiple snapshots, atleast 5)
show system processes extensive (multiple snapshots atleast 5)
show system users
show system connections
show system statistics


Turn on task accounting and collect the task accounting detail output 
(three times with a gap of 30 seconds).
Don't forget to turn it off once done.

set task accounting on
show task accounting detail
set task accounting off

show task memory detail
show task memeory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state
 
Logs:None
Traceoptions:user@router# show routing-options
traceoptions {
file routing-trace size 10m files 20 world-readable;
flag task;
flag state;
flag timer;
}
[ Back to Top ]

1588 CLI commands:show ptp lock-status
show ptp clock
show ptp global-information
show ptp port detail
show ptp statistics detail
Logs:None
Traceoptions:None
[ Back to Top ]

SyncE CLI commands:show configuration interfaces
show synchronous-ethernet esmc
Show synchronous-ethernet global-infromation
Logs:show log clksyncd
Traceoptions:set protocols clock-synchronization traceoptions flag all
[ Back to Top ]

 

References:

How to:

 

Modification History

2024-08-22: Updated Data category

Related Information