On M/MX/PTX devices, an FPC crash may result in a coredump and report the "Data TLB Miss" error.

This article outlines the cause for the FPC to crash with the "Data TLB Miss" error and the measures to address it.

The Flexible PIC Concentrators (FPC) on these routers may crash even though no network or configuration changes were made, and generate a coredump. The decoded backtrace may be generic and no anomalies may be found in the syslog for the event.

However, the reason string reported in the nvram command output of the FPC points to "Data TLB Miss."

The Translation Look-aside Buffer (TLB) cache is a CPU on-chip memory that is responsible for translation between the virtual address and the physical address. "TLB miss" might happen if the hash calculation could not find any matching address entry or due to null pointer access. This normally indicates a bit flip within the memory or CPU, and is mostly a transient issue if it is an isolated incident.

A snippet of the logs when the problem is seen is provided for reference:

root@jtac-mx960-r2002-re0> show system core-dumps no-forwarding
-rw-r--r--  1 root  wheel  994857776 May 30 09:06 /var/crash/core-SFPC4.core.0
total files: 1

root@jtac-mx960-r2002-re0> show chassis alarms no-forwarding
No alarms currently active

PFE NVRAM output

System Exception: Vector/Code 0x00e00, Signal 11
Event occurred at: May 30 09:04:43.846948

Juniper Embedded Microkernel Version 12.3R3-S4.10
Built by builder on 2015-05-27 00:39:30 UTC
Copyright (C) 1998-2015, Juniper Networks, Inc.
All rights reserved.
Reason string: "Data TLB Miss"
Context: Thread (PFE Manager)

Registers:
R00: 0x00000000 R01: 0x420a19c0 R02: 0x4209fa98 R03: 0x00080000
R04: 0x41750178 R05: 0x00000000 R06: 0x00000000 R07: 0x00000000
R08: 0x4096acc5 R09: 0x7c6c2cac R10: 0x7c6c2630 R11: 0xbd8a99df
R12: 0x48442082 R13: 0x4feecf1f R14: 0x41880000 R15: 0x417563a8
R16: 0x41756380 R17: 0x41756358 R18: 0x41756330 R19: 0x41756314
R20: 0x417562f4 R21: 0x417562d4 R22: 0x41880000 R23: 0x41880000
R24: 0x418f0000 R25: 0x40d6dd80 R26: 0x41992518 R27: 0x00000001
R28: 0x7c6c2630 R29: 0x00000000 R30: 0x40054494 R31: 0x4245f6f0
MSR: 0x00021200 CTR: 0x4039d378 Link: 0x400626e4 SP: 0x420a19c0
CR: 0x48442024 XER: 0x00000000 DEAR: 0x00080070 PC: 0x4006277c
ESR: 0x00000000 K_MSR: 0x00021200

Stack Traceback:
Frame 01: sp = 0x420a19c0, pc = 0x421ab118
Frame 02: sp = 0x420a19e0, pc = 0x40054494
Frame 03: sp = 0x420a19f8, pc = 0x400541c0
Frame 04: sp = 0x420a1a08, pc = 0x400541b4
Frame 05: sp = 0x420a1a18, pc = 0x40850a6c
Frame 06: sp = 0x420a1a98, pc = 0x40030fb0

[LOG] syslog called with interrupts off (caller pc:0x408d00d8)
[LOG] Coredump finished!
[LOG] CONV FPGA0 download start
[LOG] CONV FPGA0 download finished
[LOG] CONV FPGA0 configuring PCI interface start
[LOG] CONV FPGA0 configuring PCI interface done
[LOG] CONV FPGA1 download start
[LOG] CONV FPGA1 download finished
[LOG] CONV FPGA1 configuring PCI interface start
[LOG] CONV FPGA1 configuring PCI interface done
[LOG] IA FPGA download start
[LOG] IA FPGA download done
[LOG] IA FPGA PCI config start
[LOG] IA FPGA PCI config done
[LOG] Set the IP IRI for table #1 to 0x80000014
[LOG] IPV4 Init: Set the IP IRI to 0x80000014
[LOG]  otn_priodic ON
[LOG] RSMON rsmon_msg_thread_init
[LOG] creating ifstats trace buffer with compile-time requested size of 1048576

Syslog

May 30 09:02:31  jtac-mx960-r2002-re0 rpd[3501]: %DAEMON-4: bgp_read_message:2274: NOTIFICATION received from 172.30.17.121 (External AS 64725): code 6 (Cease) subcode 5 (Connection Rejected)
May 30 09:02:31  jtac-mx960-r2002-re0 rpd[3501]: %DAEMON-3: bgp_recv: read from peer 172.30.17.121 (External AS 64725) failed: Connection reset by peer
May 30 09:02:38  jtac-mx960-r2002-re0 rpd[3501]: %DAEMON-3: bgp_recv: peer 172.20.64.198 (External AS 65015): received unexpected EOF
May 30 09:03:26  jtac-mx960-r2002-re0 rpd[3501]: %DAEMON-3: bgp_recv: read from peer 172.20.65.54 (External AS 65015) failed: Connection reset by peer
May 30 09:04:45  jtac-mx960-r2002-re0 tnp.tftpd[5043]: %DAEMON-6-TFTPD_CONNECT_INFO: TFTP write from address 20 port 69 file core-SFPC4
May 30 09:04:45  jtac-mx960-r2002-re0 /kernel: %KERN-5: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration
May 30 09:04:47  jtac-mx960-r2002-re0 /kernel: %KERN-5: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration
May 30 09:04:49  jtac-mx960-r2002-re0 chassisd[1615]: %DAEMON-3-CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 4
May 30 09:04:49  jtac-mx960-r2002-re0 chassisd[1615]: %DAEMON-5-CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(4)
May 30 09:04:49  jtac-mx960-r2002-re0 /kernel: %KERN-5: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration

Chassisd logs

May 25 17:59:32 CHASSISD_PARSE_COMPLETE: Using new configuration
May 30 09:04:49  rcv: ch_ipc_dispatch() null ipc read for args 0x8e193a0 pipe 0x8e1bdc0, fru FPC 4 errno 60
May 30 09:04:49  ch_connection_shutdown: Destroying the IPC pipe
May 30 09:04:49  fpc_disconnect_generic: fpc 4 state Online cargs 0x8e193a0 clean_shutdown 0, offline_reason=None
May 30 09:04:49  -- FPC 4, last request 132, state Online

The "TLB miss" error does not generally repeat after a reset unless there is a permanent degradation in FPC memory. If the error is seen repeatedly, it may be due to potential hardware issues and may require a replacement of the card.

Contact Support  for assistance.