This document covers data collection steps in case macsec is not working.
Macsec does not establish.
CLI Commands to tshoot MACSEC: ********************************************************************************************************************* Step 1 : Identify the PFE/BRCM Port Number and Diag Port Number for the problematic macsec interface on both side ********************************************************************************************************************** In this example, Problematic MacSec Interface = ge-0/0/4 Corresponding Pfe/BRCM Port = 18 Corresponding Diag Port = ge17 % cprod -A fpc0 -c "show ex-bcm-pic 0" | grep -i ge-0/0/4, ifd - ge-0/0/4, pfe port num - 18 % cprod -A fpc0 -c "show shim bridge interface" | egrep -i "Index|ge-0/0/4" Index Name Type Local ModID/BCM_Port/Diag_Port PVID/Is_IRB 556 ge-0/0/4.0 L2 T 0/18/ge17 1/ ****************************************************************************************** Step 2 : Enable MacSec Traceoptions during problem state on both EX4300 Macsec Endpoints ***************************************************************************************** set security macsec traceoptions file macsec-trace.log set security macsec traceoptions file size 10m set security macsec traceoptions file files 4 set security macsec traceoptions flag all set security macsec interfaces ge-0/0/4 traceoptions file mka.log set security macsec interfaces ge-0/0/4 traceoptions file size 10m set security macsec interfaces ge-0/0/4 traceoptions file files 4 set security macsec interfaces ge-0/0/4 traceoptions flag all ***************************************************************************************************************************** Step 3 : Perform Packet capturing for Macsec Interface for 10 mins on both sides during problem State and collect rtsockmon ***************************************************************************************************************************** tcpdump -ni ge-0/0/4 -s 9000 -w /var/tmp/PS-Site-A-ge-0-0-4.pcap rtsockmon -rmt >> /var/tmp/PS-Site-A-rtsockmon-txt ********************************************************************************************************************************************************* Step 4 : Append the following line to enable "syslog any any" configuration during the course of TS & data collection and revert the changes once done. ********************************************************************************************************************************************************* set system syslog file messages any any ***************************************************************************************************************************************** Step 5 : Perform the CLI & PFE Data Collection - Need the below set of commands to be run every 30 seconds interval for ~5 minutes. ***************************************************************************************************************************************** date cli -c "set cli timestamp" cli -c "show security macsec connections | no-more" cli -c "show security macsec statistics | no-more " cli -c "show security macsec statistics detail | no-more" cli -c "show security macsec statistics interface ge-0/0/4 | no-more" cli -c "show security macsec statistics interface ge-0/0/4 detail | no-more" cli -c "show security mka sessions | no-more" cli -c "show security mka sessions detail | no-more" cli -c "show security mka statistics | no-more" date cli -c "show interfaces ge-0/0/4 media | no-more" cli -c "show interfaces ge-0/0/4 extensive | no-more" <<<<<<<< run twice in quick succession during every iteration cli -c "show interfaces ge-0/0/4 extensive | no-more" <<<<<<<< run twice in quick succession during every iteration cli -c "show interfaces diagnostics optics ge-0/0/4 | no-more" cli -c "show ethernet-switching interface ge-0/0/4 | no-more" cli -c "show ethernet-switching interface ge-0/0/4 detail | no-more" cli -c "show spanning-tree interface" cli -c "show chassis hardware | no-more" cli -c "show chassis pic fpc-slot 0 pic-slot 0 | no-more" <<<<<<<<<<<<<<<<< use the correct pic if using uplinks cli -c "show chassis fpc | no-more" cli -c "show chassis routing-engine | no-more" cli -c "show system processes extensive | no-more" cli -c "show system memory | no-more" cli -c "show system statistics bridge | no-more" cli -c "show ethernet-switching table interface ge-0/0/4" cli -c "show ethernet-switching table interface ge-0/0/4 extensive" cli -c "show ethernet-switching mac-learning-log | no-more" cli -c "show lldp neighbors" cli -c "show lldp statistics" cli -c "show lldp statistics" date cprod -A fpc0 -c "show shim bridge interface" cprod -A fpc0 -c "show ifl brief" cprod -A fpc0 -c "show ifd brief" date cprod -A fpc0 -c "show macsec port list" cprod -A fpc0 -c "show macsec port 18 capability" cprod -A fpc0 -c "show macsec port 18 config" cprod -A fpc0 -c "show macsec port 18 flow 0" cprod -A fpc0 -c "show macsec port 18 flow 1" cprod -A fpc0 -c "show macsec port 18 flow 2" cprod -A fpc0 -c "show macsec port 18 flow 3" cprod -A fpc0 -c "show macsec port 18 flow 4" cprod -A fpc0 -c "show macsec port 18 flow 5" cprod -A fpc0 -c "show macsec port 18 flow 6" cprod -A fpc0 -c "show macsec port 18 flow 7" cprod -A fpc0 -c "show macsec port 18 flow 8" cprod -A fpc0 -c "show macsec port 18 flow 9" cprod -A fpc0 -c "show shim bridge interface" cprod -A fpc0 -c "show halp-pkt pkt-stats" cprod -A fpc0 -c "show ttp statistics" cprod -A fpc0 -c "show threads" cprod -A fpc0 -c "show heap" cprod -A fpc0 -c "show filter hw all non_zero_only" cprod -A fpc0 -c "show filter hw all drop non_zero_only" cprod -A fpc0 -c "show filter hw all drop" date cprod -A fpc0 -c 'set exbcm bcmshell "show c cpu"' <<<<<<<< run twice in quick succession during every iteration cprod -A fpc0 -c 'set exbcm bcmshell "show c cpu"' <<<<<<<< run twice in quick succession during every iteration cprod -A fpc0 -c 'set exbcm bcmshell "show c ge17"' <<<<<<<< run twice in quick succession during every iteration cprod -A fpc0 -c 'set exbcm bcmshell "show c ge17"' <<<<<<<< run twice in quick succession during every iteration cprod -A fpc0 -c 'set exbcm bcmshell "show c "'. <<<<<<<< run twice in quick succession during every iteration cprod -A fpc0 -c 'set exbcm bcmshell "show c "' <<<<<<<< run twice in quick succession during every iteration cprod -A fpc0 -c "show syslog messages" date cprod -A fpc0 -c "show ex-bcm-pic 0 ifd" cprod -A fpc0 -c "show ex-bcm-pic 0 link 18" cprod -A fpc0 -c "show ex-bcm-pic 0 link 18 stat". <<<<<<<< run twice in quick succession during every iteration cprod -A fpc0 -c "show ex-bcm-pic 0 link 18 stat". <<<<<<<< run twice in quick succession during every iteration date cprod -A fpc0 -c 'set exbcm bcmshell "ps"' cprod -A fpc0 -c 'set exbcm bcmshell "phy info"' cprod -A fpc0 -c 'set exbcm bcmshell "ps ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "phy ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "soc"' cprod -A fpc0 -c "show sfp all_data" date cprod -A fpc0 -c 'set exbcm bcmshell "ms sync"' cprod -A fpc0 -c 'set exbcm bcmshell "ms flowlist ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "ms Version"' cprod -A fpc0 -c 'set exbcm bcmshell "ms show"' cprod -A fpc0 -c 'set exbcm bcmshell "ms pcg ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "ms scl ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "ms sal ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "ms scl ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "g XMAC_RX_CTRL.ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "g XMAC_TX_CTRL.ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "g XMAC_RX_LSS_STATUS.ge17"' *************************************************************************************************************************************** Step 6 : Manually Collect the following output in the given order. Rerun the complete sequence of commands atleast 3-4 times. **************************************************************************************************************************************** Step 1 : ----------- cprod -A fpc0 -c "show macsec port 18 sc list" <<<<<<<<<<<<< (Device the ingress and egress channel Id's using the PFE port number) cprod -A fpc0 -c "show macsec port 18 sc <secure-channel-id>" <<<<<<<<<<<< (Dump this output for all the secure channel ids for ingress and egress) Step 2 : ----------- First run the following two command to determine the Ingress and Egress Channel ID's and Association ID (mapped to each channel ID) : cprod -A fpc0 -c 'set exbcm bcmshell "ms scl ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "ms sal ge17"' cprod -A fpc0 -c 'set exbcm bcmshell "ms scl ge17"' ______________________________________________________________________________ Device/Port ChannelId SCI Flags Secure Offset ______________________________________________________________________________ 2.5 1 0x100e7ea109e70001 Ingress:Confidential,AutoActivate, <<<<<<<<<<<<<<---------- Channel one is the RX/Ingress Channel 2.5 0 0x88e0f31a24470001 Egress:Confidential,TX_SCI,AutoActivate, <<<<<<<<<<<<<<---------- Channel zero is the TX/Egress Channel cprod -A fpc0 -c 'set exbcm bcmshell "ms sal ge17"' __________________________________________________________________________________________________________________________________ Unit/Port ChannelId AssocId AssocNum KEY Flags Secure Offset Next PN ___________________________________________________________________________________________________________________________________ 2.5 1 1 AssocNum=0 0x80f0cf4b61616f6e1a9ec4d7e719cff5 ACTIVE ReplayProtected 0 0x00123d1e. <<<<<<<<<------ For the RX Channel 1, Association ID is 1. 2.5 0 0 AssocNum=0 0x80f0cf4b61616f6e1a9ec4d7e719cff5 ACTIVE 0 0x0011d559 <<<<<<<<<------ For the Tx Channel 0, Association ID is 0 Here RX channel id 1 and Association Id is 1 for Ingress path. Hence dump the following RX packets : . cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxErrorPkts ChanID=1 AssocID=1"' Acprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxUntagPkts ChanID=1 AssocID=1"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxNotagPkts ChanID=1 AssocID=1"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxBadtagPkts ChanID=1 AssocID=1"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxUnknownsciPkts ChanID=1 AssocID=1"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxNosciPkts ChanID=1 AssocID=1"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxSAOkPkts ChanID=1 AssocID=1"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxOvErrOverrunPkts ChanID=1 AssocID=1"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 rxDiscardPkts ChanID=1 AssocID=1"' Here TX channel id 0 and Association Id is 0 for Egress path. Hence dump the following Tx packets : cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 txControlledErrorPkts ChanID=0 AssocID=0"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 txUncontrolledErrorPkts ChanID=0 AssocID=0"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 txControlledBytes ChanID=0 AssocID=0"' cprod -A fpc0 -c 'set exbcm bcmshell "ms statsget ge17 txUncontrolledBytes ChanID=0 AssocID=0"' 1. tcpdump -i <> on both switches. 2. On vty - "show halp-pkt pkt-stats" on both sides - This shows if EAPOL packets are received in the host path. Other than this, we should also capture the tracelogs in MKA and MACsec to understand if any EAPOL's are being dropped "set security macsec traceoptions file <> size <>" "set security macsec traceoptions flag all" "set security macsec interface <> traceoptions file <> size <>" "set security macsec interface <> traceoptions flag all" Other than this - usual MACsec show commands might help, along with /var/log/messages "show security mka session" "show security macsec connections" "show security macsec statistics detail"