The syslog message "%KERN-3: i802_3_slow_recv_input:oam/esmc PDU dropped" is continuously being generated on EX4300 devices. Consequently Periodic Packet Manager Daemon (PPMD) usage has spiked up, resulting in high CPU usage as well on the device.
OAM/ESMC fall under the slow protocols category and the protocol PDUs are handled by the Periodic Packet Manager Daemon (PPMD). When such a packet ingresses the device, it is processed by PPMD, and then punted to the kernel where it is dropped because the protocol is not running on the device.
To resolve the issue, you need to configure a firewall filter to block these packets from being processed.
You may see the following log messages being generated on the device:
Mar 29 18:54:27 sw1 /kernel: %KERN-3: i802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU droppedi802_3_slow_recv_input:oam/esmc PDU dropped
You may also observe high CPU usage on the device, caused by the ppmd and lfmd processes.
user@sw1> show system processes extensive | except 0.0 | refresh 5 ---(refreshed at 2022-04-20 05:55:58 UTC)--- last pid: 28177; load averages: 1.04, 0.95, 0.59 up 21+10:43:16 05:55:58 176 processes: 6 running, 147 sleeping, 1 zombie, 22 waiting Mem: 1201M Active, 99M Inact, 139M Wired, 373M Cache, 112M Buf, 47M Free Swap: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 1385 root 76 0 27812K 16116K RUN 86.6H 63.92% ppmd 1396 root 76 0 25448K 13696K select 34.0H 14.79% lfmd 1348 root 57 0 642M 226M RUN 156.7H 6.98% pfex_junos 10 root 155 52 0K 16K RUN 194.5H 4.79% idle 1348 root -52 -52 642M 226M select 156.7H 1.03% pfex_junos 1220 root 51 0 72488K 35668K select 537:05 0.98% chassisd 899 root 43 0 34304K 16668K select 72:50 0.10% eventd
The above log messages are generated when the OAM (Operation, Administration, and Maintenance) / ESMC (Ethernet Synchronization Message Channel) protocol related protocol data units (PDUs) are dropped by the kernel. Note that the log can be generated even when these protocols are not running on the device.
The cause in such a case is that the packet is ingressing the device with an ether type that matches with slow protocols.
Below is a sample packet capture taken on the device:
Juniper PCAP Flags [Ext, In], PCAP Extension(s) total length 16 Device Media Type Extension TLV #3, length 1, value: Ethernet (1) Logical Interface Encapsulation Extension TLV #6, length 1, value: VPLS (49) Device Interface Index Extension TLV #1, length 2, value: 45826 Logical Interface Index Extension TLV #4, length 4, value: 600 -----original packet----- 1a:81:d3:33:ff:db > 3a:b7:44:e9:8e:2a, ethertype 802.1Q (0x8100), length 1492: vlan 803, p 0, ethertype Slow Protocols, unknown (53)v115, length 1474 0x0000: 3573 accc d005 29c0 53fc 28db d867 d052 0x000f: a553 6b21 27a5 1df9 d425 ae9c 2cb4 4539 0x001f: 1992 fa8d f8aa 43df 8477 5540 8302 0993 0x002f: 27bd 4347 8f0e d855 bb02 2190 7158 6a8b 0x003f: e210 af8c b0bc 2231 07ad d107 bd40 05b5 0x004f: 273b 8a2b 2b48 3e12 f9ac ca85 c0ce aab9 0x005f: 60e4 5a47 0451 7924 e44d 6979 bfd2 4034 0x006f: d52d 874a bbd9 7a96 3d1b e267 2010 7add 0x007f: 5f7e 6a2d 9dcb e81d ec5f 0625 ba93 598f 0x008f: 4c74 4abd 43f0 26d0 8404 61aa cc79 54a5 0x009f: 1a82 cff2 2804 ebed a521 d6d5 41cb 5d59 0x00af: e796 5c73 b6b5 5f21 55e7 8b21 ca68 c83f 0x00bf: 9ace c3ad c64d c691 cae9 b4f5 6ecf aa63 0x00cf: 52f4 1708 61b1 352b 750c 1dcf 7bd5 564d 0x00df: e1b7 bda8 eaec f00d b6ae 42e7 ce0b 87e5 0x00ef: 2a69 a290 2545 255d 724d cfdb 0107 bd1a 0x00ff: bd79 ca9f 4f48 a0e2 c826 9c1c 7f26 ce92 0x010f: fcff 7604 b88a 3184 c8fb a04c 5bd7 9499 0x011f: 74d7 60cf 5204 a34a 93b9 9f4f 04a4 a636 0x012f: 0157 dcac 565a 9f96 ebd5 34cf 0c97 dd62 0x013f: 97c4 c4e9 efc8 25de 14b2 eb07 f2ad 9a57 0x014f: fc68 cfd5 59fe 3f86 262f 10b3 e05e 91a0 0x015f: ce47 5e45 fb81 ba20 b560 83c6 d3dd 7c71 0x016f: a461 ed62 2794 eddf c20c df22 10fb b324 0x017f: d8ef 0c21 a389 dd99 0000 8dcb a21c 064f 0x018f: 0fb2 3dda c605 7c61 5f09 dec4 3f42 9863 0x019f: 69d3 f4d4 65cd f77b e927 aee4 f1b2 5f7a 0x01af: f0a8 7c73 5b41 300e d595 6cf4 da5f a8fe 0x01bf: d3a9 c2a0 f35d d244 1cf8 d92e 0c16 d0ad 0x01cf: 4dfb 42ee 54a7 9d94 99e5 d18b 99ff c2f0 0x01df: 0d01 4da6 0341 6a11 b388 ad9d d8e8 c037 0x01ef: f9a9 c50a 976c 1e24 f722 fa2c e38d 8b54 0x01ff: 839f e122 9cbf 646a 62f0 121b d42c 6bb4 0x020f: b060 3ea0 5d9b 59d0 ae58 90d3 342a af1e 0x021f: da66 de83 ec13 b3ef 7b3c 1c77 52ff 5ae1 0x022f: dc80 413b 55e3 30a4 b202 1d68 558d ba1d 0x023f: 1877 d47f 4b13 c528 2e4c 8deb 5e08 47e9 0x024f: 68d6 a24d c4b3 75d3 f02e 7b99 6b2c e7e1 0x025f: b234 74db 4bdd f74e baff e79f 890e de08 0x026f: 226a 42db 624b 8c3a f63e 2550 da2d c7f0 0x027f: ae45 24c3 270d 23db 7be8 d6bb cac4 2760 0x028f: 6098 7c6c 4ce5 1d77 b199 101b f44a dcc9 0x029f: 60c8 8cbb df1e 92f7 f7ee d3b5 05aa 766e 0x02af: 7b23 b557 76f7 e365 4e79 bd30 47f7 e649 0x02bf: 7afe b576 7729 a5bb 02df 6bd6 dc7f 021f 0x02cf: 85c5 0e39 eaad 1c84 98f9 a83f d67c 9870 0x02df: 0bf9 8c96 18f4 e0f6 05c0 916e 4f25 a05e 0x02ef: a593 c53e 498f ed74 ae98 8eb0 ea5a 4bc8 0x02ff: 08f3 4153 1ec6 3fca e216 9db8 89aa fcb7 0x030f: eaaa 37e7 1459 b32e 4f95 c707 34dd ce05 0x031f: 48f6 fe8f 9c4b 7490 ce46 4534 f237 9d70 0x032f: 49c5 1ae9 ae72 a2a3 2c37 fc7e 490f 427b 0x033f: ac6b 6d7f 74be c416 1597 56f5 1ebf b8db 0x034f: b04d dd2c e46c d2b4 3cfd cc68 92f9 e633 0x035f: 1d27 f05e 7de0 f047 d052 0902 2b6c f346 0x036f: 6664 5cf6 aaa1 e0cc c3db d580 31b5 0225 0x037f: 9da5 ec46 f27d e4b3 0446 5e48 4402 1804 0x038f: 9a16 18a6 7126 3f29 ba7c 1672 764f 5e99 0x039f: 2964 76f6 be5b d63b d6b0 45c9 afbd b640 0x03af: df37 2da4 6ad6 ba36 fc5f 5daf 4e58 739b 0x03bf: 3df6 3963 06ef 36e5 1075 bd30 a50d 6bad 0x03cf: c0e4 f70c 8704 b42d c292 6d71 5e75 ee16 0x03df: 7fed 8ee3 2ee3 f834 4b90 78f4 b6b4 a702 0x03ef: d2f5 af5c ffea 867f e745 f3ef 479e 119b 0x03ff: afe9 a24c 0fc2 697a 482a 536b b6da 307b 0x040f: e5ca 7b59 cac3 9e37 ed02 6fcd 96ea cc6a 0x041f: de0d b0c3 b1ce 9b45 430a 5143 70fc 2bc9 0x042f: 085f c046 04b4 faf9 a107 7984 6aed 2ad2 0x043f: e650 1ccc acf7 90d7 4539 4344 66d9 b336 0x044f: 5884 49ca 6e3f c4ea 32be 3b12 bd31 ecf6 0x045f: 611c 18d3 7c64 62f4 9afb 21bd 4917 95dd 0x046f: 1097 9d84 508a 708e d900 a994 6217 5ae9 0x047f: 4684 e11e 6919 8fe7 4f57 732b 35a6 c386 0x048f: e13a 8513 ee74 9e66 a350 7fb9 9b90 c843 0x049f: cb03 3538 1bf1 2b6b 67ec 6b56 fb5e ed00 0x04af: 87cb 1b7d 3013 86fc b958 b828 8a8d 3a24 0x04bf: 19eb 0386 75bc c9ce 1804 8102 93c2 907e 0x04cf: ba46 87d2 2652 1bf6 b7bf 5b23 fa2b da5e 0x04df: 2956 f311 d98b 4f4c cdbc 7cb2 4fee 6075 0x04ef: 8710 aca5 fb51 273e d10e 909a 6f1d c93f 0x04ff: 66eb a9e6 a643 19e5 8194 8633 420f 56ac 0x050f: dcf9 2644 99e8 1b12 b314 eb8f e589 1e56 0x051f: e989 3755 a17e 3d04 da06 3d8b 4a44 5ac2 0x052f: 5efb 798b 9d94 8a40 41c4 2b63 2eb8 8c07 0x053f: 92bb c805 721a 8637 0333 d4dc 7577 133f 0x054f: 74cf deda 91a7 d616 26e0 333f 14bc 3dc2 0x055f: fc57 113b d0c5 d7c8 6f66 cff2 2400 0578 0x056f: 8060 028d 9764 6275 f76d bc7c 7c05 5ba0 0x057f: 3fdb 9e1b 7e1b 2973 0687 3797 6b59 8ff2 0x058f: 8df3 98fb ce90 9beb 7714 bfbd 3b63 e6fa 0x059f: 2463 2c4d e64c 3294 f125 4783 4ed5 53b9 0x05af: e154 598c 9a01 adc2 a8d0 4357 4c4e cd6b 0x05bf: 5b54
Below is how the packet looks in Wireshark:
Use the command show system processes extensive to check CPU usage:
{master:0} user@sw1> show system processes extensive last pid: 28179; load averages: 1.09, 1.00, 0.65 up 21+10:44:53 05:57:35 176 processes: 7 running, 145 sleeping, 1 zombie, 23 waiting Mem: 1201M Active, 99M Inact, 139M Wired, 373M Cache, 112M Buf, 47M Free Swap: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 1385 root 76 0 27812K 16008K RUN 86.4H 54.44% ppmd<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 1396 root 76 0 25448K 13692K select 33.9H 20.46% lfmd<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 10 root 155 52 0K 16K RUN 193.2H 8.54% idle 1348 root 65 0 642M 226M RUN 156.3H 7.81% pfex_junos
Resolution
To resolve the issue, perform the following steps:
Execute monitor traffic interface <interface name> on the interface on which the suspected packet is ingressing.
Example
user@sw1> monitor traffic interface ge-0/0/45 extensive count 10 Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay. Address resolution timeout is 4s. Listening on ge-0/0/45, capture size 1514 bytes 06:04:48.279455 In Juniper PCAP Flags [Ext, In], PCAP Extension(s) total length 16 Device Media Type Extension TLV #3, length 1, value: Ethernet (1) Logical Interface Encapsulation Extension TLV #6, length 1, value: VPLS (49) Device Interface Index Extension TLV #1, length 2, value: 45826 Logical Interface Index Extension TLV #4, length 4, value: 600 -----original packet----- 1a:81:d3:33:ff:db > 3a:b7:44:e9:8e:2a, ethertype 802.1Q (0x8100), length 1492: vlan 803, p 0, ethertype Slow Protocols, unknown (53)v115, length 1474 0x0000: 3573 accc d005 29c0 53fc 28db d867 d052 0x000f: a553 6b21 27a5 1df9 d425 ae9c 2cb4 4539 0x001f: 1992 fa8d f8aa 43df 8477 5540 8302 0993 0x002f: 27bd 4347 8f0e d855 bb02 2190 7158 6a8b 0x003f: e210 af8c b0bc 2231 07ad d107 bd40 05b5 0x004f: 273b 8a2b 2b48 3e12 f9ac ca85 c0ce aab9 0x005f: 60e4 5a47 0451 7924 e44d 6979 bfd2 4034 0x006f: d52d 874a bbd9 7a96 3d1b e267 2010 7add 0x007f: 5f7e 6a2d 9dcb e81d ec5f 0625 ba93 598f 0x008f: 4c74 4abd 43f0 26d0 8404 61aa cc79 54a5 0x009f: 1a82 cff2 2804 ebed a521 d6d5 41cb 5d59 0x00af: e796 5c73 b6b5 5f21 55e7 8b21 ca68 c83f 0x00bf: 9ace c3ad c64d c691 cae9 b4f5 6ecf aa63 0x00cf: 52f4 1708 61b1 352b 750c 1dcf 7bd5 564d 0x00df: e1b7 bda8 eaec f00d b6ae 42e7 ce0b 87e5 0x00ef: 2a69 a290 2545 255d 724d cfdb 0107 bd1a 0x00ff: bd79 ca9f 4f48 a0e2 c826 9c1c 7f26 ce92 0x010f: fcff 7604 b88a 3184 c8fb a04c 5bd7 9499 0x011f: 74d7 60cf 5204 a34a 93b9 9f4f 04a4 a636 0x012f: 0157 dcac 565a 9f96 ebd5 34cf 0c97 dd62 0x013f: 97c4 c4e9 efc8 25de 14b2 eb07 f2ad 9a57 0x014f: fc68 cfd5 59fe 3f86 262f 10b3 e05e 91a0 0x015f: ce47 5e45 fb81 ba20 b560 83c6 d3dd 7c71 0x016f: a461 ed62 2794 eddf c20c df22 10fb b324 0x017f: d8ef 0c21 a389 dd99 0000 8dcb a21c 064f 0x018f: 0fb2 3dda c605 7c61 5f09 dec4 3f42 9863 0x019f: 69d3 f4d4 65cd f77b e927 aee4 f1b2 5f7a 0x01af: f0a8 7c73 5b41 300e d595 6cf4 da5f a8fe 0x01bf: d3a9 c2a0 f35d d244 1cf8 d92e 0c16 d0ad 0x01cf: 4dfb 42ee 54a7 9d94 99e5 d18b 99ff c2f0 0x01df: 0d01 4da6 0341 6a11 b388 ad9d d8e8 c037 0x01ef: f9a9 c50a 976c 1e24 f722 fa2c e38d 8b54 0x01ff: 839f e122 9cbf 646a 62f0 121b d42c 6bb4 0x020f: b060 3ea0 5d9b 59d0 ae58 90d3 342a af1e 0x021f: da66 de83 ec13 b3ef 7b3c 1c77 52ff 5ae1 0x022f: dc80 413b 55e3 30a4 b202 1d68 558d ba1d 0x023f: 1877 d47f 4b13 c528 2e4c 8deb 5e08 47e9 0x024f: 68d6 a24d c4b3 75d3 f02e 7b99 6b2c e7e1 0x025f: b234 74db 4bdd f74e baff e79f 890e de08 0x026f: 226a 42db 624b 8c3a f63e 2550 da2d c7f0 0x027f: ae45 24c3 270d 23db 7be8 d6bb cac4 2760 0x028f: 6098 7c6c 4ce5 1d77 b199 101b f44a dcc9 0x029f: 60c8 8cbb df1e 92f7 f7ee d3b5 05aa 766e 0x02af: 7b23 b557 76f7 e365 4e79 bd30 47f7 e649 0x02bf: 7afe b576 7729 a5bb 02df 6bd6 dc7f 021f 0x02cf: 85c5 0e39 eaad 1c84 98f9 a83f d67c 9870 0x02df: 0bf9 8c96 18f4 e0f6 05c0 916e 4f25 a05e 0x02ef: a593 c53e 498f ed74 ae98 8eb0 ea5a 4bc8 0x02ff: 08f3 4153 1ec6 3fca e216 9db8 89aa fcb7 0x030f: eaaa 37e7 1459 b32e 4f95 c707 34dd ce05 0x031f: 48f6 fe8f 9c4b 7490 ce46 4534 f237 9d70 0x032f: 49c5 1ae9 ae72 a2a3 2c37 fc7e 490f 427b 0x033f: ac6b 6d7f 74be c416 1597 56f5 1ebf b8db 0x034f: b04d dd2c e46c d2b4 3cfd cc68 92f9 e633 0x035f: 1d27 f05e 7de0 f047 d052 0902 2b6c f346 0x036f: 6664 5cf6 aaa1 e0cc c3db d580 31b5 0225 0x037f: 9da5 ec46 f27d e4b3 0446 5e48 4402 1804 0x038f: 9a16 18a6 7126 3f29 ba7c 1672 764f 5e99 0x039f: 2964 76f6 be5b d63b d6b0 45c9 afbd b640 0x03af: df37 2da4 6ad6 ba36 fc5f 5daf 4e58 739b 0x03bf: 3df6 3963 06ef 36e5 1075 bd30 a50d 6bad 0x03cf: c0e4 f70c 8704 b42d c292 6d71 5e75 ee16 0x03df: 7fed 8ee3 2ee3 f834 4b90 78f4 b6b4 a702 0x03ef: d2f5 af5c ffea 867f e745 f3ef 479e 119b 0x03ff: afe9 a24c 0fc2 697a 482a 536b b6da 307b 0x040f: e5ca 7b59 cac3 9e37 ed02 6fcd 96ea cc6a 0x041f: de0d b0c3 b1ce 9b45 430a 5143 70fc 2bc9 0x042f: 085f c046 04b4 faf9 a107 7984 6aed 2ad2 0x043f: e650 1ccc acf7 90d7 4539 4344 66d9 b336 0x044f: 5884 49ca 6e3f c4ea 32be 3b12 bd31 ecf6 0x045f: 611c 18d3 7c64 62f4 9afb 21bd 4917 95dd 0x046f: 1097 9d84 508a 708e d900 a994 6217 5ae9 0x047f: 4684 e11e 6919 8fe7 4f57 732b 35a6 c386 0x048f: e13a 8513 ee74 9e66 a350 7fb9 9b90 c843 0x049f: cb03 3538 1bf1 2b6b 67ec 6b56 fb5e ed00 0x04af: 87cb 1b7d 3013 86fc b958 b828 8a8d 3a24 0x04bf: 19eb 0386 75bc c9ce 1804 8102 93c2 907e 0x04cf: ba46 87d2 2652 1bf6 b7bf 5b23 fa2b da5e 0x04df: 2956 f311 d98b 4f4c cdbc 7cb2 4fee 6075 0x04ef: 8710 aca5 fb51 273e d10e 909a 6f1d c93f 0x04ff: 66eb a9e6 a643 19e5 8194 8633 420f 56ac 0x050f: dcf9 2644 99e8 1b12 b314 eb8f e589 1e56 0x051f: e989 3755 a17e 3d04 da06 3d8b 4a44 5ac2 0x052f: 5efb 798b 9d94 8a40 41c4 2b63 2eb8 8c07 0x053f: 92bb c805 721a 8637 0333 d4dc 7577 133f 0x054f: 74cf deda 91a7 d616 26e0 333f 14bc 3dc2 0x055f: fc57 113b d0c5 d7c8 6f66 cff2 2400 0578 0x056f: 8060 028d 9764 6275 f76d bc7c 7c05 5ba0 0x057f: 3fdb 9e1b 7e1b 2973 0687 3797 6b59 8ff2 0x058f: 8df3 98fb ce90 9beb 7714 bfbd 3b63 e6fa 0x059f: 2463 2c4d e64c 3294 f125 4783 4ed5 53b9 0x05af: e154 598c 9a01 adc2 a8d0 4357 4c4e cd6b 0x05bf: 5b54
Based on the capture, configure a firewall filter to block this packet from getting processed on the device:
firewall { family ethernet-switching { filter test term one { from { ether-type 0x8809; destination-mac-address { 3a:b7:44:e9:8e:2a/48; } } then { discard; count oam_esmc; } } term two { then accept; }
Note: ether-type "0x8809" is also used for LACP so care must be taken while blocking the packet with ether-type when LACP is configured on the device.
Verify that CPU usage has decreased on the device by executing show system processes extensive again.
2022-09-19: Initial publication