Description

On M-Series Routers acting as PE in L3VPN scenario ,L2RW error message seen when a VPN route is received from a remote PE reachable via RSVP LSP having first strict hop via GRE interface

Symptoms

In an L3VPN scenario when a VPN route to destination is received from remote PE reachable via RSVP LSP having GRE interface as first strict hop,the following error messages are seen on the receiving PE.

Jan 27 08:50:13 cfeb L2RW: install pgm called with encap length = 33
Jan 27 08:50:13 /kernel: RT_PFE: NH IPC op 1 (ADD NEXTHOP) failed, err 5 (Invalid)
Jan 27 08:50:13 /kernel: RT_PFE: NH details: idx 489 type 2 ifl 71
Jan 27 08:50:13 /kernel: RT_PFE: NH IPC op 28 (ADD INDIRECT NEXTHOP) failed, err 5 (Invalid)
Jan 27 08:50:13 cfeb L2RW: Fails to install L2 program
Jan 27 08:50:13 cfeb NH(nh_ucast_add): NULL l2rw_pgm_cptr
Jan 27 08:50:13 cfeb NH: Non-existant NH (489:Unicast, 2) in generic change path
Jan 27 08:50:13 cfeb NH: Indirect nh (262142) with unknown target nh (489)

Topology Used :
-----------------------

                 |----------------RSVP-LSP-------------|
CE1-----PE1--------------------P----------------PE2------CE2
                |----------GRE--------|

In this sample scenario,destination 50.50.50.0/24 and 192.192.192.192/32 are reachable via the VPN instance VPN-A as shown below :

lab> show route table VPN-A

VPN-A.inet.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

40.40.40.0/24 *[Direct/0] 00:01:10 > via fe-0/0/0.0
40.40.40.1/32 *[Local/0] 00:01:13 Local via fe-0/0/0.0
50.50.50.0/24 *[BGP/170] 00:04:20, localpref 100, from 181.181.181.181 AS path: I
> via gr-1/2/0.0, label-switched-path PE-PE
192.192.192.192/32 *[BGP/170] 00:04:20, localpref 100, from 181.181.181.181 AS path: I
> via gr-1/2/0.0, label-switched-path PE-PE

Ping is initiated from CE1 which is part of the routing instance VPN-A  to either 50.50.50.0/24 or 192.192.192.192/32 results in failure due to the error shown above.

lab@CE1> ping 192.192.192.192 rapid count 50
PING 192.192.192.192 (192.192.192.192): 56 data bytes
..................................................
--- 192.192.192.192 ping statistics ---
50 packets transmitted, 0 packets received, 100% packet loss

lab@CE1> ping 50.50.50.2 rapid count 50
PING 50.50.50.2 (50.50.50.2): 56 data bytes
..................................................
--- 50.50.50.2 ping statistics ---
50 packets transmitted, 0 packets received, 100% packet loss

All the packets are dropped at the CE1 facing interface of PE1.Field "aOutofRangeLengthField " shown in the following output indicates count of packets that are discarded.

CSBR0(noname vty)# show fe-pic 0 stats 0

PIC 0 Ether information:

Fast Ethernet PIC, 4 port
Periodics are enabled
Normal interrupts are enabled, total count is 0
IFD count is 4

ETHER-FPGA Statistic (port 0):
rx packet count = 100
rx error count = 0
rx DA reject count = 0
rx SA reject count = 0
tx packet count = 0
tx packet err cnt = 0

SQ84302 statistics information:
etherStatsDropEvents = 0
etherStatsCRCAlignErrors = 0
etherStatsUndersizePkts = 0
etherStatsOversizePkts = 0
etherStatsFragments = 0
etherStatsJabber = 0
etherStatsOctets = 10200
etherStatsPkts = 100
etherStatsBroadcastPkts = 0
etherStatsMulticastPkts = 0
etherStatsPkts64Octets = 0
etherStatsPkts65to127Octets = 100
etherStatsPkts128to255Octets = 0
etherStatsPkts256to511Octets = 0
etherStatsPkts512to1023Octets = 0
etherStatsPkts1024to1518Octets = 0
etherStatsOctets_TX = 0
etherStatsPkts_TX = 0
etherStatsBroadcastPkts_TX = 0
etherStatsMulticastPkts_TX = 0
etherStatsPkts64Octets_TX = 0
etherStatsPkts65to127Octets_TX = 0
etherStatsPkts128to255Octets_TX = 0
etherStatsPkts256to511Octets_TX = 0
etherStatsPkts512to1023Octets_TX = 0
etherStatsPkts1024to1518Octets_TX = 0
etherStatsCollisions = 0
ifOutDiscards = 0
ifOutErrors = 0
ifInOctets = 11000
ifOutOctets = 0
ifInUcastPkts = 100
ifOutUcastPkts = 0
aSingleCollisionFrames = 0
aMultipleCollisionFrames = 0
aFrameCheckSequenceErrors = 0
aAlignmentErrors = 0
aOctetsTransmittedOK = 0
aFramesWithDeferredXmissions = 0
aLateCollisions = 0
aFramesAbortedDuetoXSCollisions = 0
aCarrierSenseErrors = 0
aOctetsReceivedOK = 10200
aFramesWithExcessiveDefferal = 0
aInRangeLengthErrors = 0
aOutofRangeLengthField = 100
aSymbolErrorDuringCarrier = 0
aPauseMACCtrlFramesTransmitted = 0
aPauseMACCtrlFramesReceived = 0
aUnsupportedOpcodesReceived = 0
aSQETESTErrors = 0

CSBR0(noname vty)#

Configuration :

CE1 :
-----
lab@CE1# show configuration
interfaces
fe-0/0/1 {
unit 0 {
family inet {
address 40.40.40.2/24;
}
}
}
routing-options
static {
route 0.0.0.0/0 next-hop 40.40.40.1;
}


PE1 :
-----
lab@PE1 # show
interfaces {
fe-0/0/0 {
unit 0 {
family inet {
address 40.40.40.1/24;
}
}
}
ge-0/1/0 {
mtu 9192;
unit 0 {
family inet {
address 10.1.1.1/24;
}
family mpls;
}
}
gr-1/2/0 {
unit 0 {
tunnel {
source 10.1.1.1;
destination 10.1.1.2;
}
family inet {
address 10.2.1.18/30;
}
family mpls;
}
}
lo0 {
unit 0 {
family inet {
address 101.101.101.101/32;
}
}
}
}
routing-options {
static {
route 0.0.0.0/0 next-hop 10.209.75.254;
}
autonomous-system 100;
}
protocols {
rsvp {
interface ge-0/1/0.0;
interface gr-1/2/0.0;
interface lo0.0;
}
mpls {
no-cspf;
label-switched-path PE-PE {
from 101.101.101.101;
to 181.181.181.181;
primary force-gre;
}
path force-gre {
10.2.1.17 strict;
}
interface ge-0/1/0.0;
interface gr-1/2/0.0;
}
bgp {
group internal {
type internal;
local-address 101.101.101.101;
family inet {
unicast;
}
family inet-vpn {
unicast;
}
neighbor 181.181.181.181;
}
}
ospf {
traffic-engineering;
area 0.0.0.0 {
interface ge-0/1/0.0 {
interface-type p2p;
}
interface lo0.0;
interface gr-1/2/0.0 {
interface-type p2p;
metric 5000;
hello-interval 1;
dead-interval 3;
}
}
}
}
routing-instances {
VPN-A {
instance-type vrf;
interface fe-0/0/0.0;
route-distinguisher 101.101.101.101:1;
vrf-target {
import target:100:100;
export target:100:100;
}
vrf-table-label;
}
}


P
--
lab@P# show
interfaces {
ge-0/2/0 {
mtu 9192;
unit 0 {
family inet {
address 10.1.1.2/24;
}
family mpls;
}
}
gr-1/2/0 {
unit 0 {
tunnel {
source 10.1.1.2;
destination 10.1.1.1;
}
family inet {
address 10.2.1.17/30;
}
family mpls;
}
}
ge-1/3/0 {
mtu 9192;
unit 0 {
family inet {
address 20.1.1.1/24;
}
family mpls;
}
}
lo0 {
unit 0 {
family inet {
address 202.202.202.202/32;
}
}
}
}
protocols {
rsvp {
interface all;
}
mpls {
interface all;
}
ospf {
traffic-engineering;
area 0.0.0.0 {
interface ge-0/2/0.0 {
interface-type p2p;
}
interface ge-1/3/0.0 {
interface-type p2p;
}
interface lo0.0;
interface gr-1/2/0.0 {
interface-type p2p;
metric 5000;
hello-interval 1;
dead-interval 3;
}
}
}
}

PE2
------
lab@PE2 # show
interfaces {
ge-0/3/0 {
mtu 9192;
unit 0 {
family inet {
address 20.1.1.2/24;
}
family mpls;
}
}
fe-1/0/0 {
unit 0 {
family inet {
address 50.50.50.1/24;
}
}
}
lo0 {
unit 0 {
family inet {
address 181.181.181.181/32;
}
}
unit 100 {
family inet {
address 192.192.192.192/32;
}
}
}
}
routing-options {
autonomous-system 100;
}
protocols {
rsvp {
interface lo0.0;
interface ge-0/3/0.0;
}
mpls {
label-switched-path PE-PE {
from 181.181.181.181;
to 101.101.101.101;
primary physical;
}
path physical {
20.1.1.1 strict;
}
interface ge-0/3/0.0;
}
bgp {
group internal {
type internal;
local-address 181.181.181.181;
family inet {
unicast;
}
family inet-vpn {
unicast;
}
neighbor 101.101.101.101;
}
}
ospf {
traffic-engineering;
area 0.0.0.0 {
interface lo0.0;
interface ge-0/3/0.0 {
interface-type p2p;
}
}
}
}
routing-instances {
VPN-A {
instance-type vrf;
interface fe-1/0/0.0;
interface lo0.100;
route-distinguisher 181.181.181.181:1;
vrf-target {
import target:100:100;
export target:100:100;
}
vrf-table-label;
}
}


CE2 :
-----
lab@CE2# show
interfaces {
fe-0/3/1 {
unit 0 {
family inet {
address 50.50.50.2/24;
}
}
}
routing-options
static {
route 0.0.0.0/0 next-hop 50.50.50.1;
}

Solution

The Layer-2 (L2) Program memory on the Packet Forwarding Engine (PFE) is overutilized, due to the additional amount of overhead being added to the L2 Descriptor. The L2 Descriptor uses words as a unit of measurement.

The hardware limitations of the I/O Manager ASIC  have been exceeded in this particular scenario when a L3VPN route becomes reachable via RSVP LSP having first strict hop as GRE interface.I/O Manager ASIC(Version 3) allows for a maximum of 8 words(32 bytes).

This limitation is specific to the following platforms that share the common PFE architecture :

1.M40e
2.M7i/M10i with CFEB
3.M20
4.M40

There is no configurable workaround for this limitation.

Note :

1.Generation of the syslog message indicating this hardware limitation was not happening after some code changes in JUNOS version 9.0 and above which resulted in silent discard of the packets.PR501318 has been raised to get this error message back to help users realize the limitation.

2.This limitation does not impact the Routing-engine generated traffic to the destinations reachable via VPN instance because Routing-engine forms its own L2 header while sending out the packets and does not rely on the I/O Manager ASIC of the PFE to rewrite the L2 header.

Following snapshot confirms that :

lab@PE1> ping 192.192.192.192 routing-instance VPN-A
PING 192.192.192.192 (192.192.192.192): 56 data bytes
64 bytes from 192.192.192.192: icmp_seq=0 ttl=64 time=1.163 ms
64 bytes from 192.192.192.192: icmp_seq=1 ttl=64 time=1.184 ms
64 bytes from 192.192.192.192: icmp_seq=2 ttl=64 time=1.187 ms
64 bytes from 192.192.192.192: icmp_seq=3 ttl=64 time=1.153 ms
64 bytes from 192.192.192.192: icmp_seq=4 ttl=64 time=1.028 ms
64 bytes from 192.192.192.192: icmp_seq=5 ttl=64 time=1.181 ms
^C
--- 192.192.192.192 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.028/1.149/1.187/0.056 ms
lab@PE1>

lab@PE1> ping 50.50.50.2 routing-instance VPN-A
PING 50.50.50.2 (50.50.50.2): 56 data bytes
64 bytes from 50.50.50.2: icmp_seq=0 ttl=63 time=1.125 ms
64 bytes from 50.50.50.2: icmp_seq=1 ttl=63 time=1.120 ms
64 bytes from 50.50.50.2: icmp_seq=2 ttl=63 time=1.124 ms
64 bytes from 50.50.50.2: icmp_seq=3 ttl=63 time=1.145 ms
64 bytes from 50.50.50.2: icmp_seq=4 ttl=63 time=1.634 ms
64 bytes from 50.50.50.2: icmp_seq=5 ttl=63 time=1.117 ms
^C
--- 50.50.50.2 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.117/1.211/1.634/0.189 ms

lab@PE1>