This KB describes one possibility that may cause FXPC core-dump after some configuration changes done on EX4400 switches.
>>> Exact changes the customer made: [edit system services] - dhcp-local-server { - group dhcp-server1 { - interface irb.1017; - } - } [edit interfaces irb] - unit 1017 { - family inet { - address X.X.X.X/25; - } - } [edit snmp] + community snmp_community { + authorization read-only; + clients { + X.X.X.X/24; + X.X.X.X/24; + X.X.X.X/32; + X.X.X.X/32; + } + } - community somestring { - authorization read-only; - clients { - X.X.X.X/32; - X.X.X.X/32; - } - } [edit vlans Test_VLAN] - vlan-id 1017; - interface ge-0/0/1.1017; - interface ge-0/0/2.1017; - interface ae0.1017; - l3-interface irb.1017; + interface ge-0/0/1.1017; + interface ge-0/0/2.1017; + interface ae0.1017;
>>> Logs at that time stamp: Aug 22 12:34:05 EX4400-48T sshd[54461]: Accepted password for ossqgn from 10.160.144.20 port 49282 ssh2 Aug 22 12:34:06 EX4400-48T mgd[54464]: UI_COMMIT: User 'ossqgn' requested 'commit' operation (comment: none) Aug 22 12:34:13 EX4400-48T mgd[54464]: UI_COMMIT: User 'ossqgn' requested 'commit' operation (comment: deployed by automation) Aug 22 12:34:16 EX4400-48T mgd[54464]: UI_COMMIT_CONFIRMED_REMINDER: 'commit confirmed' must be confirmed within 3 minutes Aug 22 12:34:16 EX4400-48T jlaunchd[19588]: dhcp-service (PID 54267) terminate signal 15 sent Aug 22 12:34:16 EX4400-48T jlaunchd[19588]: Registered PID 55062(commit-batch): exec_command Aug 22 12:34:16 EX4400-48T jlaunchd[19588]: commit-batch (PID 55062) started Aug 22 12:34:16 EX4400-48T jlaunchd[19588]: Registered PID 55062(commit-batch): new process Aug 22 12:34:16 EX4400-48T xntpd[3908]: ntpd 4.2.8p15-a Sat May 18 12:47:48 2024 (1): Starting Aug 22 12:34:16 EX4400-48T xntpd[3908]: Command line: /usr/sbin/xntpd -j -N -g Aug 22 12:34:16 EX4400-48T rpd[20385]: RPD_COMMIT_SYNC_STATUS: RPD commit sync completed on MASTER Aug 22 12:34:16 EX4400-48T rpd[20385]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs started on MASTER Aug 22 12:34:16 EX4400-48T rpd[20385]: RPD_COMMIT_JOBS_STATUS: RPD commit jobs completed on MASTER Aug 22 12:34:16 EX4400-48T cosd[20396]: COSD_CLASS_8021P_UNSUPPORTED: ieee-802.1 classifier is not valid on interface xe-0/2/2.16386 Aug 22 12:34:16 EX4400-48T cosd[20396]: COSD_CLASS_8021P_UNSUPPORTED: ieee-802.1 classifier is not valid on interface xe-0/2/3.16386 Aug 22 12:34:16 EX4400-48T dcd[20492]: DCD_INFO_MSG: MIXMODE : ifd(ae0), flags: is_valid 1, mix_rate_support 1 mix_configured 0 Aug 22 12:34:16 EX4400-48T jlaunchd[19588]: dhcp-service (PID 54267) exited with status=0 Normal Exit Aug 22 12:34:16 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 685, ifAdminStatus down(2), ifOperStatus down(2), ifName irb.1017 Aug 22 12:34:16 EX4400-48T mgd[54464]: UI_COMMIT_COMPLETED: : commit complete Aug 22 12:34:17 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 682, ifAdminStatus down(2), ifOperStatus down(2), ifName ae0.1017 Aug 22 12:34:17 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 684, ifAdminStatus down(2), ifOperStatus down(2), ifName ge-0/0/1.1017 Aug 22 12:34:17 EX4400-48T fpc0 PFE_BRCM_COS_HALP_ERR: BRCM_COS_HALP(brcm_cos_ng_access_rw_acl_delete:54):Invalide iff Aug 22 12:34:17 EX4400-48T fpc0 PFE_BRCM_COS_HALP_ERR: BRCM_COS_HALP(brcm_cos_ng_access_rw_acl_delete:54):Invalide iff Aug 22 12:34:17 EX4400-48T fpc0 LBCM-L2,pfe_bcm_l2_bridge_ifl_common_set(),956:bcm_vlan_control_port_set for egress vlan-xlat miss untagged drop on unit/port (0/4) is successful Aug 22 12:34:39 EX4400-48T kernel: peer_input_pending_internal: 6212: peer class: 0, type: 27, index: 0, vksid: 1, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE Aug 22 12:34:39 EX4400-48T kernel: peer_input_pending_internal: 6212: peer class: 0, type: 27, index: 0, vksid: 1, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE Aug 22 12:34:39 EX4400-48T kernel: peer_inputs: 6511: Closing connection peer class: 0, type: 27, index: 0, vksid: 1, state: 1, err 5 Aug 22 12:34:39 EX4400-48T kernel: peer_input_pending_internal: 6212: peer class: 0, type: 27, index: 0, vksid: 0, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE Aug 22 12:34:39 EX4400-48T kernel: peer_input_pending_internal: 6212: peer class: 0, type: 27, index: 0, vksid: 0, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE Aug 22 12:34:39 EX4400-48T kernel: peer_inputs: 6470: VKS0 closing connection peer class: 0, type: 27, index: 0, vksid: 0, state: 1, err 5 Aug 22 12:34:39 EX4400-48T lacpd[78388]: LACPD_TIMEOUT: xe-0/2/0: lacp current while timer expired current Receive State: CURRENT Aug 22 12:34:39 EX4400-48T lacpd[78388]: LACP_INTF_MUX_STATE_CHANGED: ae0: xe-0/2/0: Lacp state changed from COLLECTING_DISTRIBUTING to ATTACHED, actor port state : |EXP|-|-|-|IN_SYNC|AGG|SHORT|ACT|, partner port state : |-|-|DIS|COL|OUT_OF_SYNC|AGG|SHORT|ACT| Aug 22 12:34:39 EX4400-48T lacpd[78388]: LACPD_TIMEOUT: xe-0/2/1: lacp current while timer expired current Receive State: CURRENT Aug 22 12:34:39 EX4400-48T lacpd[78388]: LACP_INTF_DOWN: ae0: Interface marked down due to lacp timeout on member xe-0/2/1 Aug 22 12:34:39 EX4400-48T lacpd[78388]: LACP_INTF_MUX_STATE_CHANGED: ae0: xe-0/2/1: Lacp state changed from COLLECTING_DISTRIBUTING to ATTACHED, actor port state : |EXP|-|-|-|IN_SYNC|AGG|SHORT|ACT|, partner port state : |-|-|DIS|COL|OUT_OF_SYNC|AGG|SHORT|ACT| Aug 22 12:34:39 EX4400-48T chassisd[19921]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 0 Aug 22 12:34:39 EX4400-48T chassisd[19921]: CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(0) Aug 22 12:34:39 EX4400-48T lacpd[78388]: LACP_INTF_MUX_STATE_CHANGED: ae0: xe-0/2/0: Lacp state changed from ATTACHED to DETACHED, actor port state : |EXP|DEF|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|, partner port state : |-|DEF|-|-|OUT_OF_SYNC|AGG|SHORT|PAS| Aug 22 12:34:39 EX4400-48T lacpd[78388]: LACP_INTF_MUX_STATE_CHANGED: ae0: xe-0/2/1: Lacp state changed from ATTACHED to DETACHED, actor port state : |EXP|DEF|-|-|OUT_OF_SYNC|AGG|SHORT|ACT|, partner port state : |-|DEF|-|-|OUT_OF_SYNC|AGG|SHORT|PAS| Aug 22 12:34:39 EX4400-48T kernel: lag_bundlestate_ifd_change: bundle ae0: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 10000000000 Number of links (Current : Required) 0 : 1 Aug 22 12:34:39 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperDisabled (entPhysicalIndex 120, entStateAdmin 3, entStateAlarm 128) Aug 22 12:34:39 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChanged Aug 22 12:34:39 EX4400-48T kernel: stp_change_idx: stp_forw_state not found for stp_index 1 Aug 22 12:34:39 EX4400-48T kernel: stp_change_idx: stp_forw_state not found for stp_index 2 Aug 22 12:34:39 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4400-48T @ 0/*/*, jnxFruType 3, jnxFruSlot 0) Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter Sock_tx init delayed Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter Sock_rx init delayed Aug 22 12:34:39 EX4400-48T fpc0 PFEBM: BEM0 Aug 22 12:34:39 EX4400-48T fpc0 BME0 created Aug 22 12:34:39 EX4400-48T fpc0 PFE-BCM-AS,brcm_asec_module_init(),64:brcm_is_as_ipsg_supported: 1 Aug 22 12:34:39 EX4400-48T fpc0 PIC: 0 S/W Init... Aug 22 12:34:39 EX4400-48T fpc0 PIC: 1 S/W Init... Aug 22 12:34:39 EX4400-48T fpc0 CLKSYNC: master RE connection made Aug 22 12:34:39 EX4400-48T fpc0 PIC: 2 S/W Init... Aug 22 12:34:39 EX4400-48T fpc0 CMLC: Command opcode 351 already present Aug 22 12:34:39 EX4400-48T fpc0 localtcp_socket_connect: bind failed, errno = 49 Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter Rt_add init delayed Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter Rt_chg init delayed Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter Rt_del init delayed Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter NH_add init delayed Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter NH_chg init delayed Aug 22 12:34:39 EX4400-48T fpc0 Ukern event counter NH_del init delayed Aug 22 12:34:39 EX4400-48T fpc0 L2ALM master RE connection made Aug 22 12:34:39 EX4400-48T fpc0 PORTDEV: Optic/IFD [0/0] missing for port: 0/1/0. Not converting to VCP port. Aug 22 12:34:39 EX4400-48T fpc0 PORTDEV: Optic/IFD [0/0] missing for port: 0/1/1. Not converting to VCP port. Aug 22 12:34:39 EX4400-48T fpc0 PORTDEV: Optic/IFD [0/0] missing for port: 0/1/2. Not converting to VCP port. Aug 22 12:34:39 EX4400-48T fpc0 PORTDEV: Optic/IFD [0/0] missing for port: 0/1/3. Not converting to VCP port. Aug 22 12:34:40 EX4400-48T fpc0 CMQFX: Unkown msg->setting 100036 Aug 22 12:34:40 EX4400-48T fpc0 info: multi hash input:0, configured:3 (3) Aug 22 12:34:40 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 2, jnxFruL1Index 1, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName Power Supply 0 @ 0/0/*, jnxFruType 7, jnxFruSlot 0) Aug 22 12:34:40 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 2, jnxFruL1Index 1, jnxFruL2Index 2, jnxFruL3Index 0, jnxFruName Power Supply 1 @ 0/1/*, jnxFruType 7, jnxFruSlot 1) Aug 22 12:34:40 EX4400-48T alarmd[19924]: Alarm set: PS SFXPC id=150995019, color=YELLOW, class=CHASSIS, reason=FPC 0 PEM 0 is not powered Aug 22 12:34:40 EX4400-48T alarmd[19924]: Alarm set: PS SFXPC id=167772491, color=YELLOW, class=CHASSIS, reason=FPC 0 PEM 1 is not powered Aug 22 12:34:40 EX4400-48T craftd[20379]: Receive FX craftd set alarm message: color: 2 class: 100 object: 75 slot: 0 silent: 0 short_reason=FPC 0 PEM 0 No Power long_reason=FPC 0 PEM 0 is not powered id=150995019 reason=150994944 Aug 22 12:34:40 EX4400-48T craftd[20379]: Minor alarm set, FPC 0 PEM 0 is not powered Aug 22 12:34:40 EX4400-48T craftd[20379]: Receive FX craftd set alarm message: color: 2 class: 100 object: 75 slot: 1 silent: 0 short_reason=FPC 0 PEM 1 No Power long_reason=FPC 0 PEM 1 is not powered id=167772491 reason=167772160 Aug 22 12:34:40 EX4400-48T craftd[20379]: Minor alarm set, FPC 0 PEM 1 is not powered Aug 22 12:34:43 EX4400-48T dc-pfe[55068]: HEAP malloc(0) detected! Aug 22 12:34:43 EX4400-48T dc-pfe[55068]: HEAP malloc(0) detected! Aug 22 12:34:43 EX4400-48T fpc0 HEAP malloc(0) detected! Aug 22 12:34:43 EX4400-48T fpc0 HEAP malloc(0) detected! Aug 22 12:34:45 EX4400-48T dc-pfe[55068]: SCHED: Thread 50 (cmqfx_pseudo) ran for 2085 ms without yielding Aug 22 12:34:45 EX4400-48T dc-pfe[55068]: Scheduler Oinker Aug 22 12:34:45 EX4400-48T dc-pfe[55068]: Frame 0: sp = 0x1948ca4c, pc = 0x232c27a Aug 22 12:34:45 EX4400-48T dc-pfe[55068]: Frame 1: sp = 0x1948ca90, pc = 0x232dd93 Aug 22 12:34:45 EX4400-48T dc-pfe[55068]: Frame 2: sp = 0x1948caa4, pc = 0x303dd73 Aug 22 12:34:45 EX4400-48T dc-pfe[55068]: Frame 3: sp = 0x1948cad8, pc = 0x232dfa0 Aug 22 12:34:45 EX4400-48T dc-pfe[55068]: Frame 4: sp = 0x1948cae0, pc = 0x0 Aug 22 12:34:46 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperEnabled (entPhysicalIndex 120, entStateAdmin 4, entStateAlarm 2) Aug 22 12:34:46 EX4400-48T chassisd[19921]: CHASSISD_IFDEV_CREATE_NOTICE: fpc_create_devs_from_queue: created interface device for pfh-0/0/0 Aug 22 12:34:46 EX4400-48T chassisd[19921]: CHASSISD_IFDEV_CREATE_NOTICE: fpc_create_devs_from_queue: created interface device for pfe-0/0/0 Aug 22 12:34:46 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 7, jnxFruL1Index 1, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4400-48T @ 0/*/*, jnxFruType 3, jnxFruSlot 0) Aug 22 12:34:46 EX4400-48T dc-pfe[55068]: DEVRT_CLIENT_SELF_MEMBER_CHANGE: nmember 0 omember -1 Aug 22 12:34:46 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 620, ifAdminStatus up(1), ifOperStatus down(2), ifName ae0 Aug 22 12:34:46 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 516, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/0 Aug 22 12:34:46 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 517, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/1 Aug 22 12:34:46 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 612, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/2/0 Aug 22 12:34:46 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 613, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/2/1 Aug 22 12:34:46 EX4400-48T mib2d[20382]: SNMP_TRAP_LINK_DOWN: ifIndex 621, ifAdminStatus up(1), ifOperStatus down(2), ifName irb.1016 Aug 22 12:34:52 EX4400-48T fpc0 SCHED: Thread 50 (cmqfx_pseudo) ran for 2085 ms without yielding Aug 22 12:34:52 EX4400-48T fpc0 Scheduler Oinker Aug 22 12:34:52 EX4400-48T fpc0 Frame 0: sp = 0x1948ca4c, pc = 0x232c27a Aug 22 12:34:52 EX4400-48T fpc0 Frame 1: sp = 0x1948ca90, pc = 0x232dd93 Aug 22 12:34:52 EX4400-48T fpc0 Frame 2: sp = 0x1948caa4, pc = 0x303dd73 Aug 22 12:34:52 EX4400-48T fpc0 Frame 3: sp = 0x1948cad8, pc = 0x232dfa0 Aug 22 12:34:52 EX4400-48T fpc0 Frame 4: sp = 0x1948cae0, pc = 0x0 Aug 22 12:34:52 EX4400-48T fpc0 CMLC: No handler for subtype 673 Aug 22 12:34:52 EX4400-48T fpc0 CMLC: No handler for subtype 425 Aug 22 12:34:52 EX4400-48T fpc0 CMQFX: waiting for pfeman resync to complete Aug 22 12:34:52 EX4400-48T last message repeated 2 times Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/0 link 0 SFP laser bias current low alarm set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/0 link 0 SFP output power low alarm set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/0 link 0 SFP laser bias current low warning set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/0 link 0 SFP output power low warning set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/1 link 1 SFP laser bias current low alarm set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/1 link 1 SFP output power low alarm set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/1 link 1 SFP laser bias current low warning set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/1 link 1 SFP output power low warning set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/1 link 1 SFP receive power high warning set Aug 22 12:34:52 EX4400-48T fpc0 CMQFX: waiting for pfeman resync to complete Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/2 link 2 SFP laser bias current low alarm set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/2 link 2 SFP output power low alarm set Aug 22 12:34:52 EX4400-48T fpc0 sfp-0/2/2 link 2 SFP receive power low alarm set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/2 link 2 SFP laser bias current low warning set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/2 link 2 SFP output power low warning set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/2 link 2 SFP receive power low warning set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/3 link 3 SFP laser bias current low alarm set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/3 link 3 SFP output power low alarm set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/3 link 3 SFP receive power low alarm set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/3 link 3 SFP laser bias current low warning set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/3 link 3 SFP output power low warning set Aug 22 12:34:53 EX4400-48T fpc0 sfp-0/2/3 link 3 SFP receive power low warning set Aug 22 12:34:53 EX4400-48T fpc0 CMQFX: waiting for pfeman resync to complete Aug 22 12:34:55 EX4400-48T last message repeated 11 times Aug 22 12:34:55 EX4400-48T kernel: fxpc_peer_init:overriding peertype_module Aug 22 12:34:56 EX4400-48T fpc0 Probe exists for str bulkget_dfw Aug 22 12:34:56 EX4400-48T fpc0 CMQFX: waiting for pfeman resync to complete Aug 22 12:34:56 EX4400-48T fpc0 RT-HAL,rt_mesh_group_add_check,1398: Mesh-group table not found: proto 35 rtt 1) Aug 22 12:34:56 EX4400-48T fpc0 RT-HAL,rt_mesh_group_add_check,1398: Mesh-group table not found: proto 35 rtt 1) Aug 22 12:34:56 EX4400-48T fpc0 PFE-BCM-AS,brcm_as_get_keepalive_count(),1106:keepalive count set to 67 Aug 22 12:34:57 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 1, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: @ 0/0/*, jnxFruType 11, jnxFruSlot 0, jnxFruOfflineReason 2, jnxFruLastPowerOff 320773439, jnxFruLastPowerOn 320775195) Aug 22 12:34:57 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperEnabled (entPhysicalIndex 190, entStateAdmin 4, entStateAlarm 128) Aug 22 12:34:57 EX4400-48T chassisd[19921]: CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChanged >>> FXPC core-dump found on the time stamp when committed was done. root@EX4400-48T> show system core-dumps no-forwarding /var/crash/*core*: No such file or directory -rw------- 1 root wheel 11136288 Aug 22 12:34 /var/tmp/fxpc.core.0.gz <<<<<<<<<< /var/tmp/pics/*core*: No such file or directory /var/crash/kernel.*: No such file or directory /var/jails/rest-api/tmp/*core*: No such file or directory /tftpboot/corefiles/*core*: No such file or directory total files: 1
Upgrade to any one of the fixed versions of PR1831770.
External view of the PR:
https://prsearch.juniper.net/problemreport/PR1831770