This is an internal KB stating some cli and pfe commands to troubleshoot OSPF/OSPF3 adjacency issue.
OSPF/OSPFv3 adjacency went down/not coming up.
RSI and /var/log collected in the problem state
> request support information | save /var/tmp/rsi-prblm.txt
> file archive compress source /var/log/* destination /var/tmp/varlog-prblm.tgz
PPMD traces:
# set routing-options ppm traceoptions file ppm.log size 1g
# set routing-options ppm traceoptions flag all
Note: This below PPMD and RPD traces should only be enabled in the problem state when there is an active outage, otherwise we wouldn't recommend enabling the below on a production box carrying traffic
OSPF3 traces:
# set protocols ospf3 traceoptions file ospf_log
# set protocols ospf3 traceoptions file size 1g
# set protocols ospf3 traceoptions flag hello detail
# set protocols ospf3 traceoptions flag error detail
# set protocols ospf3 traceoptions flag event detail
OSPF traces:
# set protocols ospf traceoptions file ospf_log
# set protocols ospf traceoptions file size 1g
# set protocols ospf traceoptions flag hello detail
# set protocols ospf traceoptions flag error detail
# set protocols ospf traceoptions flag event detail
OSPF commands:
show ospf neighbor
show ospf neighbor extensive
show ospf neighbor area 0 extensive
show ospf neighbor brief
show ospf neighbor detail
show ospf neighbor interface <interface-name>
show ospf database
show ospf database extensive
show ospf database detail
show ospf database brief
show ospf interface detail
show ospf interface <interface-name>
show ospf interface <interface-name> extensive
show ospf route
show ospf interface
show ospf interface extensive
show route protocol ospf
show route protocol ospf detail
show route protocol ospf extensive
RPD traces:
# set routing-options traceoptions file rpd.log
# set routing-options traceoptions flag all
CLI:
> show ppm connections
> show ppm rpc-statistics
> show ppm event-statistics
> show ppm interfaces detail
> show ppm transmissions detail
> show ppm adjacencies detail
> show route forwarding-table destination
> show route summary
> show ospf3 statistics
> show ospf statistics
> show ddos-protection protocols
> show ddos-protection protocols statistics terse
> show system statistics | no-more
> show system statistics ip6
> show system virtual-memory
> show system buffers
> show system memory
> show task memory detail
> show task io
> show task io | match ospf
> set task accounting on
> show task accounting detail
> show task statistics
> show task summary
> show krt queue
> show krt state
TCPdump on couple of interfaces where OSPF/OSPF3 is down
Example:
> monitor traffic interface ae10 no-resolve extensive count 10 layer2-headers print-hex
Shell:
% nhinfo -a
% sysctl kern.ipc
% sysctl kern.ipc | grep nmb
% sysctl net.isr
% sysctl net.isr.drop
% sysctl -a | grep rpd
% top -SH | grep netisr
% rtsockmon -rnt
Gcore:
Note: gcores need to be collected in problem state and need root-access
Examples:
labroot@node> start shell user root
Password:
root@node:RE:0% ps -aux | grep ppmd
root 2067 0.0 0.5 12900 8696 ?? S Sun08PM 11:22.54 /usr/sbin/ppmd -N
root@node:RE:0% gcore -c /var/tmp/ppmd.gcore.0 2067
root@node:RE:0% ps -aux | grep rpd
root 2071 0.0 3.6 180176 68836 ?? S Sun08PM 18:21.54 /usr/sbin/rpd -N
root@node:RE:0% gcore -c /var/tmp/rpd.gcore.0 2071