In a scaled PWHT environment, the number of locally terminated circuits might exceed the size of the MAC-address pool associated with a logical tunnel interface. In this case, the excess circuits will remain unusable.
In lt-style configuration of PWHT, there are two types of sub-interfaces (IFLs) configured under a logical tunnel interfaces (lt-):
interfaces { lt-1/0/0 { unit 100 { /* Service IFL */ encapsulation vlan; vlan-id 100; peer-unit 1100; family inet { address 198.19.0.1/30; } } unit 1100 { /* Transport IFL */ encapsulation vlan-ccc; vlan-id 100; peer-unit 100; } } } protocols { l2circuit { neighbor 198.18.255.2 { interface lt-1/0/0.1100 { virtual-circuit-id 1100; } } } }
To make use of access devices, which are incapable of qualified (per-VLAN) MAC-learning, JUNOS allocates a unique MAC address for every "service" IFL, e.g.:
> show interfaces lt-1/0/0 terse | match "lt-1/0/0.10[0-3]" lt-1/0/0.100 up up inet 198.19.0.1/30 lt-1/0/0.101 up up inet 198.19.0.5/30 lt-1/0/0.102 up up inet 198.19.0.9/30 lt-1/0/0.103 up up inet 198.19.0.13/30
> show interfaces lt-1/0/0 | match "(lt-1/0/0.10[0-3]|MAC)" Logical interface lt-1/0/0.100 (Index 332) (SNMP ifIndex 655) MAC: 02:24:dc:78:69: 4a Logical interface lt-1/0/0.101 (Index 334) (SNMP ifIndex 659) MAC: 02:24:dc:78:69: 4b Logical interface lt-1/0/0.102 (Index 336) (SNMP ifIndex 3900) MAC: 02:24:dc:78:69: 4c Logical interface lt-1/0/0.103 (Index 338) (SNMP ifIndex 3901) MAC: 02:24:dc:78:69: 4d
On the other hand, a typical number of MAC-addresses which are allocated per lt- interface from the global (chassis) pool is 64. Thus, if the number of locally terminated PHWT circuits crosses the threshold, the newly provisioned interfaces will remain inactive:
<-- Syslog facility = kernel May 11 12:43:37.823 jtac-MX480-r182 kernel: if_rtb_default_ifl_bitmap_set() rtb ifl bitmap itable op 1 done. rtb id 0 ifl idx 476 iff 0xfffff80403739e80. May 11 12:43:37.823 jtac-MX480-r182 kernel: RLT:lt_ifl_peer_update:lt-1/0/0.1164 lt-1/0/0.164 op 1 May 11 12:43:37.823 jtac-MX480-r182 kernel: RLT:lt_ifl_peer_update:OLD {lt-1/0/0.1164-lt-1/0/0.164} cooki{0 130} May 11 12:43:37.823 jtac-MX480-r182 kernel: RLT:lt_ifl_peer_update:NEW {lt-1/0/0.1164-lt-1/0/0.164} cooki{130 130} May 11 12:43:37.823 jtac-MX480-r182 kernel: lt_ifl_alloc_mac_addr(530), IS_MASTER_RE: 1, error: 28 , errmsg: free index unavailable for mac address May 11 12:43:37.954 jtac-MX480-r182 kernel: RLT:lt_ifl_peer_update:lt-1/0/0.1164 lt-1/0/0.164 op 2
Note errno 28 stands for "No space left on device".
> show interfaces lt-1/0/0 terse | match "lt-1/0/0.16[0-7]" lt-1/0/0.160 up up inet 198.19.0.241/30 lt-1/0/0.161 up up inet 198.19.0.245/30 lt-1/0/0.162 up up inet 198.19.0.249/30 lt-1/0/0.163 up up inet 198.19.0.253/30 /* 64th circuit */ lt-1/0/0.164 up down inet 198.19.1.1/30 /* 65th circuit */ lt-1/0/0.165 up down inet 198.19.1.5/30 lt-1/0/0.166 up down inet 198.19.1.9/30 lt-1/0/0.167 up down inet 198.19.1.13/30
The service IFL is down:
> show interfaces lt-1/0/0.164 Logical interface lt-1/0/0.164 (Index 453) (SNMP ifIndex 4008) Flags: Hardware-Down Up SNMP-Traps 0x4000 VLAN-Tag [ 0x8100.164 ] Encapsulation: LT-VLAN Input packets : 0 Output packets: 0 Protocol inet, MTU: 9174 Max nh cache: 0, New hold nh limit: 0, Curr nh cnt: 0, Curr new hold cnt: 0, NH drop cnt: 0 Flags: Sendbcast-pkt-to-re Addresses, Flags: Dest-route-down Is-Preferred Is-Primary Destination: 198.19.1.0/30, Local: 198.19.1.1, Broadcast: 198.19.1.3
The interface state is propagated from the service IFL to the transport IFL (the peer unit):
> show interfaces lt-1/0/0.1164 Logical interface lt-1/0/0.1164 (Index 454) (SNMP ifIndex 4009) Flags: Up Link-Layer-Down SNMP-Traps 0x4000 VLAN-Tag [ 0x8100.164 ] Encapsulation: VLAN-CCC Input packets : 0 Output packets: 0 Protocol ccc, MTU: 9192
The corresponding PW is inactive:
> show l2circuit connections interface lt-1/0/0.1164 Layer-2 Circuit Connections: Legend for connection status (St) EI -- encapsulation invalid NP -- interface h/w not present MM -- mtu mismatch Dn -- down EM -- encapsulation mismatch VC-Dn -- Virtual circuit Down CM -- control-word mismatch Up -- operational VM -- vlan id mismatch CF -- Call admission control failure OL -- no outgoing label IB -- TDM incompatible bitrate NC -- intf encaps not CCC/TCC TM -- TDM misconfiguration BK -- Backup Connection ST -- Standby Connection CB -- rcvd cell-bundle size bad SP -- Static Pseudowire LD -- local site signaled down RS -- remote site standby RD -- remote site signaled down HS -- Hot-standby Connection XX -- unknown Legend for interface status Up -- operational Dn -- down Neighbor: 198.18.255.2 Interface Type St Time last up # Up trans lt-1/0/0.1164(vc 1164) rmt LD
To overcome this limit, change the allocation mode from "per-IFL" to "per-interface (IFD)".There will be only a single MAC-address used across all service IFLs:
[edit interfaces lt-1/0/0] + logical-tunnel-options { + per-unit-mac-disable; + }
The service IFLs are all active now:
> show interfaces lt-1/0/0 terse | match "lt-1/0/0.16[0-7]" lt-1/0/0.160 up up inet 198.19.0.241/30 lt-1/0/0.161 up up inet 198.19.0.245/30 lt-1/0/0.162 up up inet 198.19.0.249/30 lt-1/0/0.163 up up inet 198.19.0.253/30 lt-1/0/0.164 up up inet 198.19.1.1/30 lt-1/0/0.165 up up inet 198.19.1.5/30 lt-1/0/0.166 up up inet 198.19.1.9/30 lt-1/0/0.167 up up inet 198.19.1.13/30
All the IFLs share the same MAC-address:
> show interfaces lt-1/0/0.16[0-7] | match "(Logical|MAC)" Logical interface lt-1/0/0.160 (Index 436) (SNMP ifIndex 4000) MAC: 02:24:dc:78:69:4a Logical interface lt-1/0/0.161 (Index 438) (SNMP ifIndex 4002) MAC: 02:24:dc:78:69:4a Logical interface lt-1/0/0.162 (Index 440) (SNMP ifIndex 4004) MAC: 02:24:dc:78:69:4a Logical interface lt-1/0/0.163 (Index 442) (SNMP ifIndex 4006) MAC: 02:24:dc:78:69:4a Logical interface lt-1/0/0.164 (Index 444) (SNMP ifIndex 4008) MAC: 02:24:dc:78:69:4a Logical interface lt-1/0/0.165 (Index 446) (SNMP ifIndex 4010) MAC: 02:24:dc:78:69:4a Logical interface lt-1/0/0.166 (Index 448) (SNMP ifIndex 4012) MAC: 02:24:dc:78:69:4a Logical interface lt-1/0/0.167 (Index 450) (SNMP ifIndex 4020) MAC: 02:24:dc:78:69:4a