Background Information

1. Provide all SSH / Telnet session captures.

2. Provide any available topology information.

3. Provide a summary of how the device is being used (production, lab system, co-location, and so on).

4. Provide device history details (new installation, production for X months/years, other recent cases, and so on).

5. Provide details of any recent changes in the network or on the device.

Request Support Information

1. Enter request support information | save /var/log/rsi1.log .

2. After step #1 completes, wait long enough to ensure that the condition that you wish to address continues/appears before proceeding to the next step.

3. Enter request support information | save /var/log/rsi2.log .

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Show Commands

set cli timestamp
show bgp summary
show bgp group 
show bgp neighbor
show route next-hop database
show route protocol bgp
show route protocol bgp terse
show route receive-protocol bgp
show route advertising-protocol bgp
show route advertising-protocol bgp <neighbor-address> extensive
show route receive-protocol bgp <neighbor-address>
show route forwarding-table
show route resolution unresolved
show route hidden
show route community <reg-ex>
show route terse community-name <name>
show route damping
show route aspath-regex "regex"
show route damping [history|decayed|suppressed]
monitor traffic interface XX-X/X/X size 4096 detail

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Caveats

set protocols bgp traceoptions file bgp.tr
set protocols bgp traceoptions file size 5m
set protocols bgp traceoptions file files 5
set protocols bgp traceoptions flag all
set routing-instances bgp-ri protocols bgp traceoptions file bgp.tr
set routing-instances bgp-ri protocols bgp traceoptions file size 5m
set routing-instances bgp-ri protocols bgp traceoptions file files 5
set routing-instances bgp-ri protocols bgp traceoptions flag all

Show Commands

If OSPF is running in a routing instance, specify which instance where applicable.
 

set cli timestamp
show ospf overview
show ospf database
show ospf neighbor detail
show ospf route
show ospf statistics
show ospf interface
show ospf log
show route protocol ospf
show route <x.x.x.x> extensive
show ospf database summary 
show ospf database extensive
show bfd session

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Caveats

Use the following for the inet.0 default instance:

set protocols ospf traceoptions file ospf.tr
set protocols ospf traceoptions file size 5m
set protocols ospf traceoptions file files 5
set protocols ospf traceoptions flag all

Use the following for routing instances:

set routing-instances ospf-vr protocols ospf traceoptions file ospf-vr.tr
set routing-instances ospf-vr protocols ospf traceoptions file size 5m
set routing-instances ospf-vr protocols ospf traceoptions file files 5
set routing-instances ospf-vr protocols ospf traceoptions flag all

Known Issues

N/A

Show Commands

set cli timestamp
show rip overview
show rip neighbor
show route protocol rip
show route advertising-protocol rip <neighbor-address> extensive
show route receive-protocol rip <neighbor-address>
show rip statistics
show route forwarding-table 
show route resolution unresolved

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Caveats

set protocols rip traceoptions file rip.tr
set protocols rip traceoptions file size 5m
set protocols rip traceoptions file files 5
set protocols rip traceoptions flag all

set routing-instances rip-vr protocols rip traceoptions file rip.tr
set routing-instances rip-vr protocols rip traceoptions file size 5m
set routing-instances rip-vr protocols rip traceoptions file files 5
set routing-instances rip-vr protocols rip traceoptions flag all

Show Commands

set cli timestamp
show isis adjacency
show isis adjacency extensive
show isis interface
show isis interface details
show isis hostname
show isis spflog
show isis statistics 
show isis route 
show route protocol isis
show isis database
show isis database detail
show bdf session
show route forwarding-table

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Caveats

set protocols isis traceoptions file rip.tr  
set protocols isis traceoptions file size 5m  
set protocols isis traceoptions file files 5 
set protocols isis traceoptions flag all 

set routing-instances isis-vr protocols isis traceoptions file isis.tr
set routing-instances isis-vr protocols isis traceoptions file size 5m
set routing-instances isis-vr protocols isis traceoptions file files 5
set routing-instances isis-vr protocols isis traceoptions flag all

Show Commands

General

show multicast route   
show multicast statistics
show multicast sessions
show multicast usage
show multicast interface
show multicast next-hops
show multicast rpf summary
show interface <if-name> extensive

IGMP

show igmp group detail
show igmp statistics
show igmp interface detail

PIM

show pim statistics
show pim neighbors
show pim rps detail
show pim join extensive
show pim bootstrap
show pim source detail

MSDP

show msdp source-active
show msdp detail
show msdp statistics

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Caveats

set protocols pim|igmp|msdp traceoptions file multicast-proto.tr
set protocols pim|igmp|msdp traceoptions file size 5m
set protocols pim|igmp|msdp traceoptions file files 5
set protocols pim|igmp|msdp traceoptions flag all

Enable traceoptions for PIM, IGMP, or MSDP.

Show Commands

show mpls interface 
show mpls lsp
show mpls lsp extensive
show mpls lsp ingress
show mpls lsp transit
show ted database

RSVP

show rsvp interface
show rsvp neighbor
show rsvp session
show rsvp statistics
show rsvp session ingress
show rsvp version
show route table mpls.0
show route table inet.3

LDP

show ldp neighbors
show ldp session
show ldp database [session peer]

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Caveats

set protocols mpls|ldp|rsvp traceoptions file mpls.tr
set protocols mpls|ldp|rsvp traceoptions file size 5m
set protocols mpls|ldp|rsvp traceoptions file files 5
set protocols mpls|ldp|rsvp traceoptions flag all

set protocols mpls label-switched-path <lsp-name> traceoptions file mpls-lsp.tr

set protocols mpls label-switched-path <lsp-name> traceoptions file size 5m

set protocols mpls label-switched-path <lsp-name> traceoptions file files 5

set protocols mpls label-switched-path <lsp-name> traceoptions flag all

Show Commands

show vpls connections extensive
show bgp summary
show route table <instance-name>.l2vpn.0 extensive
show route table bgp.l2vpn.0 extensive
show route table mpls.0 extensive
show route table inet.3 extensive
show vpls flood extensive
show vpls mac-table
show interfaces terse
show interfaces routing
show route forwarding table family mpls extensive
show route forwarding table family vpls extensive
show interface ge-x/y/z extensive | no-more
show l2-learning interface
show l2-learning mac-move-buffer extensive
show l2-learning l2alm-peers
show l2-learning debug-statistics events
show l2-learning debug-statistics ipc
show l2-learning debug-statistics mac-events
show l2-learning debug-statistics mac-messages
show l2-learning debug-statistics mac-processing
show l2-learning debug-statistics performance-counters
show l2-learning debug-statistics rtsock
show bridge statistics
show vpls statistics
show bridge mac-table instance <instance-name> extensive
show vpls mac-table instance <instance-name> extensive

Determine if traffic is getting to the VPLS interface. Apply the following firewall filter in the VPLS interface as input and output:

set firewall family vpls filter jtac-test term t1 from destination-mac-address <>
set firewall family vpls filter jtac-test term t1 from source-mac-address <>
set firewall family vpls filter jtac-test term t1 then count counter_in
set firewall family vpls filter jtac-test term t1 then accept
set firewall family vpls filter jtac-test term t2 then accept

root@root> show firewall filter jtac-test
 
Filter: jtac-test
Counters:
Name               Bytes              Packets
counter_in         0                  0

Test PE-to-PE connectivity over the VPLS PW:

user@PE1> ping vpls instance red destination-mac <*> source-ip <*>

For source-ip, configure IRB under the VPLS instance within the same address range within CPEs. Remember to have the loopback interface open for the source address.

For M/T Series

show route forwarding-table family vpls vpn <instance-name> extensive

Note: This command is used for M/T Series devices. It is equivalent to the previous command.

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Caveats

If allowed, turn on the following traceoptions for approximately five minutes while traffic is passing through. Then, turn it back off and collect the log files. See Caveats .

set routing-instances <instance-name> protocols vpls traceoptions flag all

This command will collect all VPLS-related traces.

set protocols l2-learning traceoptions file l2ald-trace size 100m;

set protocols l2-learning traceoptions file flag mac-learning;
set protocols l2-learning traceoptions file flag ipc;
set protocols l2-learning traceoptions file flag routing-socket;

CLI Commands

show route table <vpn-a>  
show route table <vpn-a> hidden
show route forwarding-table vpn <vpn-a>

ping x.x.x.x routing-instance <vpn-a>
traceroute x.x.x.x routing-instance <vpn-a>
telnet x.x.x.x routing-instance <vpn-a>
ping mpls l3vpn <vpn-a> prefix <prefix>

show route table bgp.l3vpn.0 
show route advertising-protocol bgp x.x.x.x
show route receive-protocol bgp x.x.x.x
show arp

OSPF VPN commands

show ospf interface instance <vpn-a>
show ospf neighbor instance <vpn-a>
show ospf database instance <vpn-a>

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

None

CLI Commands

Kompella

show l2vpn connections extensive     
show route table <vpn-a>
show route table bgp.l2vpn.0 
show route advertising-protocol bgp x.x.x.x
show route receive-protocol bgp x.x.x.x
show route table mpls.o
show route forwarding-table family mpls
show rsvp session extensive

Martini

show l2circuit connections extensive
show ldp neighbor detail
show ldp database detail
show route table mpls.o

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

Kompella

set routing-instances l2vpn protocols l2vpn traceoptions file l2vpn_trace
set routing-instances l2vpn protocols l2vpn traceoptions file size 50m
set routing-instances l2vpn protocols l2vpn traceoptions file files 5
set routing-instances l2vpn protocols l2vpn traceoptions flag all

Martini

set protocols l2circuit traceoptions file l2circuit_trace
set protocols l2circuit traceoptions file size 50m
set protocols l2circuit traceoptions file files 5
set protocols l2circuit traceoptions flag all

set protocols ldp traceoptions file ldp_trace
set protocols ldp traceoptions file size 50m
set protocols ldp traceoptions file files 5
set protocols ldp traceoptions flag all

CLI Commands

show interface xx-x/x/x detail   
show interface queue xx-x/x/x both-ingress-egress
show interfaces XX-X/X/X extensive

show class-of-service interfaces XX-X/X/X 
show class-of-service code-point-aliases
show class-of-service code-point-aliases dscp
show class-of-service code-point-aliases inet-prec
show class-of-service code-point-aliases exp
show class-of-service forwarding-class
show class-of-service classifier [name]
show class-of-service scheduler-map
show class-of-service rewrite-rule
show class-of-service drop-profile
show class-of-service forwarding-table
show class-of-service forwarding-table classifier mapping
show class-of-service forwarding-table scheduler-map

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

None

CLI Commands

show interfaces filters
show firewall
show firewall filter <filter-name>
show firewall filter <name> prefix-action <psa-name> from 1 to 8
show firewall log
show log <log-file-name>
show policer
show interface policer XX-X/X/X

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

show log /var/tmp/<sample file>
show log sampled

Traceoptions

None

CLI Commands

set show interface terse
show route table inet6
show ipv6 neighbor
show interface lo0 extensive | display xml

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

None

CLI Commands

set cli timestamp
show configuration snmp

show snmp statistics extensive # Multiple time with polling activity.

show snmp mib walk <oid>
show snmp mib walk ipfragfails
show interface lo0 extensive

show system processes extensive | no-more # Multiple time with polling activity.

show chassis routing-engine | no-more # Multiple time with polling activity.

show system commit | no-more
show system statistics # Multiple time with polling activity.
show system buffers # Multiple time with polling activity.
show system core-dumps
show route <ip of NMS server>
ping <ip of NMS server> source <snmp source ip> count 500 rapid

show pfe statistics traffic # Multiple time with polling activity.

show interface extensive | no-more
show system process extensive | no-more
show snmp statistics subagents | no-more
show snmp stats-response-statistics

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

set snmp traceoptions file snmp files 10 size 50m
set snmp traceoptions flag all

CLI Commands

set cli timestamp
show chassis routing-engine (multiple snapshots, atleast 5)
show system processes extensive (multiple snapshots atleast 5)
show system users
show system connections
show system statistics

Turn on task accounting and collect the task accounting detail output (three times with a gap of 30 seconds). Do not forget to turn it off after you have finished.
 

set task accounting on
show task accounting detail
set task accounting off

show task memory detail
show task memory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

user@router# show routing-options
traceoptions {
file routing-trace size 10m files 20 world-readable;
flag task;
flag state;
flag timer;
}

Show Commands

General

show chassis environment cb
show chassis alarms

Fabric-related

show chassis hardware
show chassis fabric summary
show chassis fabric map
show chassis fabric fpc
show chassis fabric plane

show class-of-service fabric statistics (take multiple outputs to see counters incrementing)

show pfe statistics traffic (take multiple outputs to see counters incrementing)

request pfe execute target <fpc_no> command "show hsl2 statistics detail"

request pfe execute target <fpc_no> command "show syslog messages"

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

show log messages
show log chassisd

Hidden Commands

Caveats

Note: These commands are not officially supported, and are hidden; however, they are good for troubleshooting Fabric Plane issues.

Exercise caution when executing these commands, because incorrect syntax may lead to problems. Junos OS Release 15.1R7 may reset the FPC, but Junos OS Release 18.4R2 is seen to be stable.

show chassis fabric asic-configuration

show chassis fabric statistics detail <plane_no> totals

Take multiple outputs to see the counters incrementing and from all the planes.

show chassis hsl channel asic-name CBXFY slot Z extensive

• X is the CB slot.

• Y is the Fabric Plane number.

• Z is the FPC slot no.

• CBXFY is in upper case.

For example:

show chassis hsl channel asic-name CB0F1 slot 3 extensive

Run this command for all the SCBs and for all the planes multiple times to see the incrementing counters.

Questions

• Did the SCBs for which the alarms were generated have any previous history of problems?

• What kind of troubleshooting has already been done to mitigate the problem. Provide detailed steps.

• Were there any recent changes to configuration?

• Were there any changes to the hardware setup, or was new hardware introduced?

• Did any events occur just before the log message that may have triggered it? This may include events such as Routing Engine switchover.

• Was a software upgrade/downgrade done recently?

Show Commands

show route table<instance-name>.inet.0 | no-more
show route table<instance-name>.mvpn.0 | no-more
show route table<instance-name>.inet.0 extensive | no-more
show route table<instance-name>.mvpn.0 extensive | no-more
show pim joininstance <instance-name> extensive | no-more
show multicast route instance <instance-name> extensive | no-more
show interface extensive |no-more
show pim rps instance <instance-name> extensive | no-more
show mpls lsp p2mp ingress |no-more
show mpls lsp p2mp ingress extensive |no-more
show mpls lsp p2mp | no-more
show mpls lsp | no-more
show bgp summary |no-more

Logs
(each node)

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

PIM detailed Traceoptions

RSVP detailed Traceoptions

BGP detailed Traceoptions

MVPN detailed Traceoptions

RTSOCKMON

Known Issues

PR Search

CGNAT

Show Commands 

show chassis pic fpc-slot x pic-slot x
show services service-sets cpu-usage
show services service-sets summary
show services service-sets statistics syslog
show services service-sets statistics packet-drops
show services service-sets memory-usage
show services service-sets memory-usage zone
show services stateful-firewall conversations
show services stateful-firewall conversations destination-prefix
show services stateful-firewall statistics extensive
show services stateful-firewall flow-analysis 
show services stateful-firewall flows extensive
show services nat mappings detail
show services nat pool detail
show services nat mappings endpoint-independent
show services nat mappings address-pooling-paired

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

None

Spanning
Tree

Show Commands 

show spanning-tree bridge detail
show spanning-tree bridge msti x detail
show spanning-tree interface detail
show spanning-tree mstp configuration detail
show spanning-tree statistics bridge
show spanning-tree statistics interface detail
show spanning-tree statistics message-queues
show spanning-tree statistics routing-instance detail
show spanning-tree stp-buffer see-all

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

set protocols rstp|mstp|vstp traceoptions file spanning-tree.trace
set protocols rstp|mstp|vstp traceoptions file size 20m
set protocols rstp|mstp|vstp traceoptions file files 5
set protocols rstp|mstp|vstp traceoptions flag all

BFD

Show Commands 

show ppm connections detail
show bfd session extensive | no-more
show ppm interfaces detail | no-more
show ppm adjacencies detail | no-more
show ppm transmissions detail | no-more
show pfe statistics traffic | no-more

Logs

Archive the /var/log/ contents:

file archive compress source /var/log/* destination /var/tmp/CURRENT-DATE.tgz

To ensure that the /var/log/ directory was properly archived, check the file size by using the command: file list /var/tmp/CURRENT-DATE.tgz detail .

Traceoptions

set routing-options ppm traceoptions file ppm
set routing-options ppm traceoptions file size 30m
set routing-options ppm traceoptions flag all
set protocols bfd traceoptions file bfd
set protocols bfd traceoptions file size 30m
set protocols bfd traceoptions flag all