User can see the MS-PIC may restart multiple times and no core dumps can be seen on the device.
Feb 26 10:49:38 se1_RE0 chassisd[23427]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power off (jnxFruContentsIndex 20, jnxFruL1Index 12, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName MIC: MS-MIC-16G @ 11/0\/*, jnxFruType 11, jnxFruSlot 11, jnxFruOfflineReason 7, jnxFruLastPowerOff 1588299736, jnxFruLastPowerOn 1579104599)
----------------------------------
root@se1_RE0> show chassis alarms no-forwarding
No alarms currently active
FPC 11 REV 16 750-038489 CACV4291 MPCE Type 1 3D
CPU REV 06 711-038484 CACR7286 MPCE PMB 2G
MIC 0 REV 17 750-043688 CACR2001 MS-MIC-16G
PIC 0 BUILTIN BUILTIN MS-MIC-16G
MIC 1 REV 18 750-049846 CALB3476 3D 20x 1GE(LAN)-E,SFP
PIC 2 BUILTIN BUILTIN 10x 1GE(LAN) -E SFP
> show chassis pic fpc-slot 11 pic-slot 0
FPC slot 11, PIC slot 0 information:
Type MS-MIC-16G
State Online
PIC version 1.17
CPU load average 5 percent
Interrupt load average 0 percent
Total DRAM size 16211 MB
Memory buffer utilization 0 percent
Memory heap utilization 13 percent
Uptime 1 hour, 19 minutes, 26 seconds
Package Extension-provider
Control Cores 1
Data Cores 7
Object Cache Size 12288
Policy DB Size 1024
Forwarding DB Size 256
Wired Process Mem Size 1024
Wired MAX Processes 1
root@se1_RE0> show system core-dumps no-forwarding
/var/crash/*core*: No such file or directory
/var/tmp/*core*: No such file or directory
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
Before the PIC flapped, we could see the below log messages:
Feb 25 09:16:50 se1.ty4_RE0 : %PFE-5: fpc11 Service-MIC(11/0): Transient flow-control asserted by XLP MAC over link 0 for 1 seconds
Feb 25 09:16:50 se1.ty4_RE0 : %PFE-5: fpc11 Service-MIC(11/0): Transient flow-control asserted by XLP MAC over link 1 for 1 seconds
Feb 25 09:16:51 se1.ty4_RE0 : %PFE-5: fpc11 Service-MIC(11/0): Transient flow-control asserted by XLP MAC over link 0 for 2 seconds
Feb 25 09:16:51 se1.ty4_RE0 : %PFE-5: fpc11 Service-MIC(11/0): Transient flow-control asserted by XLP MAC over link 1 for 2 seconds
Feb 25 09:16:52 se1.ty4_RE0 : %PFE-5: fpc11 Service-MIC(11/0): Transient flow-control asserted by XLP MAC over link 0 for 3 seconds
Feb 25 09:16:54 se1.ty4_RE0 : %PFE-5: fpc11 Service-MIC(11/0): Transient flow-control asserted by XLP MAC over link 1 for 5 seconds
Feb 25 09:16:54 se1.ty4_RE0 : %PFE-4: fpc11 Prolonged flow-control asserted by MAC on ms-11/0, bringing interface down
Feb 25 09:16:54 se1.ty4_RE0 : %PFE-3: fpc11 ms-11/0: MAC stuck, trigger reset
Feb 25 09:16:54 se1.ty4_RE0 spd[34938]: %DAEMON-5-SPD_AMS_NAT_RANGE_DELETE: Deleting AMS NAT Ranges and NHs for SERVICE-TY-116486-2037 on ams0.
PIC then restarted 5 seconds later:
Feb 25 09:16:59 se1.ty4_RE0 chassisd[23427]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power off (jnxFruContentsIndex 8, jnxFruL1Index 12, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: MS-MIC-16G @ 11/0/*, jnxFruType 11, jnxFruSlot 11, jnxFruOfflineReason 8, jnxFruLastPowerOff 1579103762, jnxFruLastPowerOn 17209)
Feb 25 09:16:59 se1.ty4_RE0 chassisd[23427]: %DAEMON-5-CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 12, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: MS-MIC-16G @ 11/0/*, jnxFruType 11, jnxFruSlot 11, jnxFruOfflineReason 2, jnxFruLastPowerOff 1579103762, jnxFruLastPowerOn 1579103762)
As we can see PIC recovers automatically.
Incase you still see the logs, proceed with reboot of the PIC/FPC to clear the issue.
JTAC would need the mspmand core files to better understand this, check if there is "dump-on-flow-control" knob configured.
Please configure dump-on-flow-control knob in case of any re-occurrence to generate a core-dump:
Enable this configuration for all the services card that is processing NAT traffic in a production network and plan is to continue to monitor.
> set interfaces <interface-name> multiservice-options flow-control-options dump-on-flow-control
For example:
se1_RE0# show | compare
[edit interfaces]
+ pc-0/0/0 {
+ multiservice-options {
+ flow-control-options {
+ dump-on-flow-control;
+ }
Please note this command does not work with ms- and mams- interfaces.
https://www.juniper.net/documentation/us/en/software/junos/flow-monitoring/topics/ref/statement/dump-on-flow-control-edit-interfaces.html