This article provides a solution and method to recover the MACsec session.
On EX4600 running Junos OS 17.3R3-S3.3, MACsec will not come up after physically flapping one link of an AE.
MACsec in a working state:
root@EX4600-Lab> show security macsec connections interface xe-0/0/18 Interface name: xe-0/0/18 CA name: ca1 Cipher suite: GCM-AES-128 Encryption: on Key server offset: 0 Include SCI: no Replay protect: off Replay window: 0 Outbound secure channels SC Id: 64:64:9B:4E:AB:F1/1 Outgoing packet number: 1 Secure associations AN: 0 Status: inuse Create time: 00:52:13 Inbound secure channels SC Id: F0:1C:2D:4A:23:F0/1 Secure associations AN: 0 Status: inuse Create time: 00:52:13
After a physical interface flap, MACsec does not recover on its own:
{MASTER} root@EX4600-Lab> show security macsec connections interface xe-0/0/18 CA name: ca1 Cipher suite: GCM-AES-128 Encryption: on Key server offset: 0 Include SCI: no Replay protect: off Replay window: 0
Logs indicate the interface is going down:
SNMP_TRAP_LINK_DOWN: ifIndex 513, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/0/18 /kernel: if_msg_ifd_add: l2_count in add (0), for ifd:xe-0/0/18 fpc0 BRCM_COS_HALP(brcm_cos_halp_ifd_setup:1356): cos_halp_ifd 659(xe-0/0/18) already allocated mib2d[2150]: SNMP_TRAP_LINK_UP: ifIndex 513, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-0/0/18
SNMP_TRAP_LINK_DOWN: ifIndex 513, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/0/18
/kernel: if_msg_ifd_add: l2_count in add (0), for ifd:xe-0/0/18 fpc0 BRCM_COS_HALP(brcm_cos_halp_ifd_setup:1356): cos_halp_ifd 659(xe-0/0/18) already allocated mib2d[2150]: SNMP_TRAP_LINK_UP: ifIndex 513, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-0/0/18
The interface and the ae are up, but MACsec did not recover:
root@EX4600-Lab> show interfaces xe-0/0/18 Physical interface: xe-0/0/18, Enabled, Physical link is Up Interface index: 659, SNMP ifIndex: 513 Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, Source filtering: Disabled Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Flow control: Disabled, Media type: Fiber Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Link flags : None CoS queues : 12 supported, 12 maximum usable queues Current address: 64:64:9b:4e:ab:f1, Hardware address: 64:64:9b:4e:ab:15 Last flapped : 2019-03-26 07:40:24 PDT (00:01:36 ago) root@PR-1423597-EX4600-1> show interfaces ae0 terse Interface Admin Link Proto Local Remote ae0 up up ae0.0 up up inet 10.10.10.2/30
root@EX4600-Lab> show interfaces xe-0/0/18 Physical interface: xe-0/0/18, Enabled, Physical link is Up Interface index: 659, SNMP ifIndex: 513 Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, Source filtering: Disabled Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Flow control: Disabled, Media type: Fiber Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Link flags : None CoS queues : 12 supported, 12 maximum usable queues Current address: 64:64:9b:4e:ab:f1, Hardware address: 64:64:9b:4e:ab:15 Last flapped : 2019-03-26 07:40:24 PDT (00:01:36 ago)
root@PR-1423597-EX4600-1> show interfaces ae0 terse Interface Admin Link Proto Local Remote ae0 up up ae0.0 up up inet 10.10.10.2/30
Related Configuration:
root@EX4600-Lab> show configuration security macsec connectivity-association ca1 { security-mode static-cak; pre-shared-key { ckn 1234567890; cak "XXXXXXXXXXXXXXXXX"; ## SECRET-DATA } } interfaces { xe-0/0/18 { connectivity-association ca1; } } {master:0} root@EX4600-Lab> show configuration interfaces xe-0/0/18 ether-options { 802.3ad ae0; }
root@EX4600-Lab> show configuration security macsec connectivity-association ca1 { security-mode static-cak; pre-shared-key { ckn 1234567890; cak "XXXXXXXXXXXXXXXXX"; ## SECRET-DATA } } interfaces { xe-0/0/18 { connectivity-association ca1; } } {master:0}
root@EX4600-Lab> show configuration interfaces xe-0/0/18 ether-options { 802.3ad ae0; }
Due to an error in Broadcom MACsec software a MACsec session may not re-establish after a physical link flap. This issue is resolved in Junos OS 15.1R7 17.2R3 17.3R3 17.4R2 18.1R1. See PR1283314 .
To recover the MACsec session for the stale state, bounce the ae interface:
set interfaces ae0 disable delete interfaces ae0 disable {MASTER} root@EX4600-Lab# set interfaces ae0 disable {master:0}[edit] root@EX4600-Lab# commit configuration check succeeds commit complete {MASTER} root@EX4600-Lab# rollback 1 load complete {MASTER} root@EX4600-Lab# commit and-quit configuration check succeeds commit complete Exiting configuration mode root@EX4600-Lab> show security macsec connections interface xe-0/0/18 Interface name: xe-0/0/18 CA name: ca1 Cipher suite: GCM-AES-128 Encryption: on Key server offset: 0 Include SCI: no Replay protect: off Replay window: 0 Outbound secure channels SC Id: 64:64:9B:4E:AB:F1/1 Outgoing packet number: 1 Secure associations AN: 0 Status: inuse Create time: 00:1:09 Inbound secure channels SC Id: F0:1C:2D:4A:23:F0/1 Secure associations AN: 0 Status: inuse Create time: 00:1:09