Description

If chassis reports FXPC core-dumps, FPC restarts and the below Sio0 errors are seen.  The issue is caused by the noise/interrupts on the console port:

/kernel: sio0: 215 more interrupt-level buffer overflows (total 215)
/kernel: sio0: 1520 more interrupt-level buffer overflows (total 1735)

Symptoms

FXPC process crashes are seen continuously and the FPC restart reason shows as '0x2000:hypervisor reboot'.
If it is a VC setup, the FPC disconnects from the VC each time there is a crash.

> show system core-dumps no-forwarding
-rw-rw----  1 root  field    6504036 Jul 28 15:10 /var/tmp/fxpc.core.0.gz
-rw-rw----  1 root  field    5119402 Jul 28 18:39 /var/tmp/fxpc.core.1.gz
...

/kernel: sio0: 1441 more interrupt-level buffer overflows (total 1387167)
/kernel: sio0: 20008 more interrupt-level buffer overflows (total 1407175)
dc-pfe: SCHED: Thread 7 (SFP) ran for 1223 ms without yielding
dc-pfe: Scheduler Oinker
dc-pfe: Frame 0: sp = 0x2023ca50, pc = 0x8048ada
dc-pfe: Frame 1: sp = 0x2023ca60, pc = 0x8055b82
...

/kernel: BAD_PAGE_FAULT: pid 1513 (fxpc), uid 0: pc 0x807984e got a read fault at 0xdeadbf13, x86 fault flags = 0x4
/kernel: Trapframe Register Dump:
/kernel: eax: 00000003 ecx: deadbeef edx: 243698a0 ebx: 00000000
/kernel: esp: 234a51ec ebp: 234a51f0 esi: 23e48538 edi: 00000000
/kernel: eip: 0807984e eflags: 00010297
/kernel: cs: 0033 ss: 003b ds: 2049003b es: ae1003b
/kernel: fs: 201b003b trapno: 0000000c err: 00000004
/kernel: Page table info for PC address 0x807984e: PDE = 0x40780067, PTE = 3fa80425
/kernel: Dumping 16 bytes starting at PC address 0x807984e:
/kernel:     8b 49 24 5b 5d ff e1 0f b6 82 fe 00 00 00 88 03 
...

/kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration
/kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration
..

/kernel: pfe_listener_disconnect: conn dropped: issu_master = 0,listener idx=0, tnpaddr=0x11000080, reason: socket error
init: packet-forwarding-engine (PID 1513) terminated by signal number 11. Core dumped!
init: packet-forwarding-engine (PID 57986) started
...

vccpd[1512]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to 204e.71b3.3900 on vcp-255/0/102.32768, 
vccpd[1512]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to 204e.71b3.3900 on vcp-255/0/99.32768, 

Solution

If the chassis reports the following Sio0 errors, it indicates excessive interrupts/noise at the console:

/kernel: sio0: 215 more interrupt-level buffer overflows (total 215)
/kernel: sio0: 1520 more interrupt-level buffer overflows (total 1735)

Physical isolation at the console such as replacing the console cable or moving to a different port on terminal server or moving the console connection to another terminal server would fix the issue.