MPC11 MACSec issues
MX2010-with MPC11 configured for AE and MACSEC. AE0 has 4 legs and 2 of the legs are not going into CD state but stay in a default/detached state. Removing MACSEC or configuring MACSEC to excluded LACP for encryptions brings up all legs into a CD state. From looking at the data, seems like port et-7/2/1 is not sending encrypted packets at all while the MX304 et-0/1/5 is. Everything points to the MPC11 not TX encrypted packets.
Topology:
MX2010 AE0 MX304
-et-0/2/1---100g----et-0/0/5
-et-2/2/1---100g----et-0/0/13
-et-7/2/1---100g----et-0/1/5 ****default/attached
-et-0/5/1---100g----et-0/1/13****default/attached
Configuration:
ae0 {
apply-groups INTERFACES_BACKBONE;
description x
aggregated-ether-options {
lacp {
active;
}
unit 0 {
family inet {
address 64.4.244.188/31;
family inet6 {
address 2620:110:f00f:ffff::64.4.244.188/127;
security {
authentication-key-chains {
key-chain macsec-keys {
key 1 {
key-name x;
start-time x
macsec {
connectivity-association macsec-data {
cipher-suite gcm-aes-xpn-256;
security-mode static-cak;
mka {
should-secure;
pre-shared-key-chain macsec-keys;
interfaces {
et-0/1/1 {
connectivity-association macsec-data;
et-0/2/1 {
et-0/5/1 {
et-0/6/1 {
et-2/1/1 {
et-2/2/1 {
et-2/3/1 {
et-2/6/1 {
et-7/2/1 {
et-7/3/1 {
-----------------------------------------------------------------------------
AE0 is directly connected to a MX304. The MX2010 MPC11 seems not to be sending any encrypted packets.
Example:
MX2020: Focus is on et-7/2/1 but same issue seen on et-0/5/1.
ruvora@JNRP-re0> show lacp interfaces ae0
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
et-2/2/1 Actor No No Yes Yes Yes Yes Fast Active
et-2/2/1 Partner No No Yes Yes Yes Yes Fast Active
et-0/2/1 Actor No No Yes Yes Yes Yes Fast Active
et-0/2/1 Partner No No Yes Yes Yes Yes Fast Active
et-0/5/1 Actor No Yes No No No Yes Fast Active
et-0/5/1 Partner No Yes No No No Yes Fast Passive
et-7/2/1 Actor No Yes No No No Yes Fast Active
et-7/2/1 Partner No Yes No No No Yes Fast Passive
LACP protocol: Receive State Transmit State Mux State
et-2/2/1 Current Fast periodic Collecting distributing
et-0/2/1 Current Fast periodic Collecting distributing
et-0/5/1 Defaulted Fast periodic Detached
et-7/2/1 Defaulted Fast periodic Detached
ruvora@JNRP-re0> show security macsec connections interface et-7/2/1
CA name: macsec-data
Cipher suite: GCM-AES-XPN-256 Encryption: on
Key server offset: 0 Include SCI: no
Replay protect: off Replay window: 0
Outbound secure channels
SC Id: B4:F9:5D:C4:C0:97/1
Outgoing packet number: 1
Secure associations
AN: 1 Status: inuse Create time: 1d 06:52:09
Inbound secure channels
SC Id: B4:F9:5D:8A:45:12/1
uvora@JNRP-re0> show security macsec statistics interface et-7/2/1
Secure Channel transmitted
Encrypted packets: 0******NOT SENDING
Encrypted bytes: 0
Protected packets: 0
Protected bytes: 0
Secure Association transmitted
Encrypted packets: 0
Secure Channel received
Accepted packets: 0
Validated bytes: 0
Decrypted bytes: 0
Secure Association received
ruvora@JNRP-re0> show configuration | display set | match et-7/2/1
set security macsec interfaces et-7/2/1 connectivity-association macsec-data
set interfaces et-7/2/1 description x
set interfaces et-7/2/1 gigether-options 802.3ad ae0
ruvora@JNRP-re0> start shell pfe network fpc7
root@JNRP-re0-fpc7:pfe> show interfaces ae0
Name: ae0 Index: 128 IflCount: 1 Type: 29 Weight: 1
CfgState: Up OverallState: Up GlobalSlot: 255 PfeInst: 255
Local: Yes IsAggregate: Yes MTU: 9192 PfeId: 0
ChassisId: 0 ChassisLocal: Yes LocalSlot: 255
LinkState: Up Macsec: Disabled Pic: 255 PicPort: 0
VlanEnabled: False
ChannelCount: 0 IfdSpeed: 200000000000
Flags: 0x0000000000008000
SpecificFlags: 0x0000000000000000
MacAddress: b6:f9:5d:c4:bb:02
InitTime: Tue Jun 4 23:55:18 2024
StateChangeTime: Mon Jun 17 03:59:05 2024
GE Flags:
lpbk: False flowCtrl: False lacp: False
lldp: False padFrame: False restrictQ: False
No Tpids
OutputStream: 65535
StreamNumber: 65535
StreamToken: None
StreamFlags: 0x00
IIFTable: Null
Redundancy States:
[0] selectorToken:14390 listToken:14391 unilistToken:14392
selMode:2 type:17 aggType:4 selType:2 selSubtype:0 unilistMode:5
targeting:No adaptive:No randomMode:No rotateHash:No local-bias:No
Steering (chassisId:0 pfeId:10 nhToken:2351 fwdToken:14409)
Use Count:3 Operational Weight:1
Default links:
[0] et-2/2/1 token:14409 hits:0
[1] et-2/2/1 token:14409 hits:0
[2] et-2/2/1 token:14409 hits:0
[3] et-2/2/1 token:14409 hits:0
[4] et-2/2/1 token:14409 hits:0
[5] et-2/2/1 token:14409 hits:0
[6] et-2/2/1 token:14409 hits:0
[7] et-2/2/1 token:14409 hits:0
IfdName IfdIndex LinkIndex Weight OperState SelectorToken
et-2/2/1 190 64 1 Up 2941
et-7/2/1 218 64 1 Down 14388
et-0/2/1 263 64 1 Up 8950
et-0/5/1 272 64 1 Down 14336
802.1BR Extended Port: EC-ID: 0 Satellite-Id: 0
Aggregate member list
IfdIndex Name Weight Activated State ActiveToken
190 et-2/2/1 1 Yes Up 2942
218 et-7/2/1 1 No Down 14389
263 et-0/2/1 1 Yes Up 8951
272 et-0/5/1 1 No Down 14337
oot@JNRP-re0-fpc7:pfe> show interfaces et-7/2/1
Name: et-7/2/1 Index: 218 IflCount: 1 Type: 172 Weight: 1
CfgState: Up OverallState: Up GlobalSlot: 7 PfeInst: 2
Local: Yes IsAggregate: No MTU: 9192 PfeId: 30
ChassisId: 0 ChassisLocal: Yes LocalSlot: 7
LinkState: Up Macsec: Enabled Pic: 2 PicPort: 11
ChannelCount: 1 IfdSpeed: 100000000000
SpecificFlags: 0x0000000000100000
InitTime: Tue Jun 4 23:55:41 2024
StateChangeTime: Tue Jun 18 11:10:38 2024
lpbk: False flowCtrl: False lacp: True
lldp: True padFrame: False restrictQ: False
tpid[0] = 0x8100
OutputStream: 1100
StreamNumber: 696
StreamToken: 3067
StreamFlags: 0xa5 (LACP | ESMC | LLDP | DOT1X)
FeatureList:
Container token: 3065
#6 IIFLkup tokens:
Mask : 0x1
[ IIFLkup:3068 ]
IIFTableToken: 3068
VlanTblSize: 0
802.1BR Extended Port: EC-ID: 1 Satellite-Id: 0
oot@JNRP-re0-fpc7:pfe> show macsec_drv list
ASIC_INDEX | PORT_GROUP | KEY_INDEX | KEY_NAME
0 | 0 | 210 | et-7/0/0
0 | 4 | 211 | et-7/0/1
1 | 0 | 216 | et-7/1/4
1 | 1 | 215 | et-7/1/3
1 | 2 | 213 | et-7/1/1
1 | 3 | 212 | et-7/1/0
1 | 4 | 214 | et-7/1/2
2 | 0 | 221 | et-7/2/4
2 | 1 | 220 | et-7/2/3
2 | 2 | 218 | et-7/2/1
root@JNRP-re0-fpc7:pfe> show macsec_drv 218 dump_ds
MACSEC_DRV_DUMP_PORT [0x7fd23d7dc800]:
MD port lookup key 218 lookup name[ et-7/2/1 ]: index 0, AN 0, SA idx 0 SC idx 0, parent_ptr[0x7fd23d75beb0]
TX SC
mem0: protframe 1, xpnmode 1 ptextoff 0 ptextsz 0 sci -5406186778055475199 stagoff 0 saindex1Valid 1 index1 128
mem1: saindex0Valid 1 index0 0 npn 1
mem2: keytype 1 ssci 1
RX SC
mem0: ptextoff 0 ptextsz 0 sci -5406187029235892223 sa index0 0 index1 128 index2 256 index3 384
mem1: keytype 1 xpnmode 1 ssci 0
TX SA
RX SA
MACSEC_DRV_DUMP_PG [0x7fd23d75beb0]:
debug_flag 0 index 2 port_count 10 vport_count 32 mdpg_fake_kats_err 0 fcs_bytes_enable 0
rx_reg_addr 0x60210000 tx_reg_addr 0x60248000 parent_ptr 0x7fd1097de2c0 child_ptr 0x7fd23d7dc800 vchild_ptr 0x7fd1099c6540
MACSEC_DRV_DUMP_DRV [0x7fd1097de2c0]:
debug flag 1 lookup key 2 pg count 5 child_ptr 0x7fd23d75be00
root@JNRP-re0-fpc7:pfe> show macsec_drv 218 rx_stats
in_pkts_notusing_sa : in_pkts_unused_sa : in_pkts_ok : in_pkts_invalid : in_pkts_not_valid
=======================================================================================================
0 : 0 : 0 : 0 : 0 - sa[ 0]
0 : 0 : 0 : 0 : 0 - sa[ 1]
0 : 0 : 0 : 0 : 0 - sa[ 2]
0 : 0 : 0 : 0 : 0 - sa[ 3]
in_pkts_unchk : 0
in_pkts_delayed : 0
in_pkts_late : 0
stat_untagged : 0
stat_notag : 0
stat_badtag : 0
stat_unknown_sci : 0
stat_no_sci : 0
InPktsOverrun : 0
octets_validated : 0
octets_decrypted : 0
link_drop_pkt_count : 0
ingress_pkt_count : 0
egress_pkt_count : 0
runt_pkt_drop_count : 0
root@JNRP-re0-fpc7:pfe> show macsec_drv 218 tx_stats
out_pkts_protected : out_pkts_encrypted
====================================================
0 : 0 - sa[ 0]
0 : 0 - sa[ 1]
stat_unctrl : 0
untagged Packets : 0
OutPktsTooLong : 0
stat_err : 0
octets_protected : 0
octets_encrypted : 0
egress_pkt_drop_count : 0
ruvora@JNRP-re0# run show lacp statistics interfaces ae0
Jun 18 12:40:21
LACP Statistics: LACP Rx LACP Tx Unknown Rx Illegal Rx
et-2/2/1 2475 2475 0 0
et-0/2/1 2475 2475 0 0
et-7/2/1 3 66 0 0
et-0/5/1 0 2475 0 0
{master}[edit]
Jun 18 12:40:24
et-2/2/1 2478 2478 0 0
et-0/2/1 2478 2479 0 0
et-7/2/1 3 69 0 0
et-0/5/1 0 2479 0 0
Jun 18 12:40:25
et-2/2/1 2480 2480 0 0
et-0/2/1 2480 2480 0 0
et-7/2/1 3 71 0 0
et-0/5/1 0 2480 0 0
Jun 18 12:40:27
et-2/2/1 2481 2481 0 0
et-0/2/1 2481 2482 0 0
et-7/2/1 3 72 0 0
et-0/5/1 0 2482 0 0
https://gnats.juniper.net/web/default/1811300#external_tab
On Junos EVO MX2008/MX2010/MX2020 platforms with MX2K-MPC11E line cards, and MACSec (IEEE 802.1AE standard) configured on line card ports. When line card comes online for first time, it is seen that ports are not being mapped correctly (port group value mismatch between picd and security) resulting in MACSec not working on some ports.