Description

Why should we configure the 'link-speed' statement within an Gigabit Ethernet based Aggregated Ethernet bundle configuration?

Symptoms

While the configuration requirements for Aggregated Ethernet do not require the 'link-speed' statement in the configuration, the scalability of a Gigabit Ethernet based Aggregated Ethernet interface is directly affected by this configuration.

Symptoms:
  • The 20th VRRP group is not working after configuring Link aggregation; the 19 VRRP group is working properly
  • Some VRRP VIP unable to forward traffic.
  • DCD_CONFIG_WRITE_FAILED: configuration write failed

Solution

In production network environments, when the link-speed configuration statement is not provided under the [ edit interfaces aeX aggregated-ether-options ] hierarchy, you will likely observe the following messages:

LabRouter_re0 /kernel: arp_add_mac: ae0 ifd_mfilter failed for mac 00:00:5e:00:01:0f error (12)
LabRouter_re0 dcd[4318]: DCD_CONFIG_WRITE_FAILED: Interface ae0, configuration write failed for an IFA ADD: Cannot allocate memory
LabRouter_re0 /kernel: ae_mfilter: ae0 limit of 20 filters exceeded for 00:00:5e:00:01:10
LabRouter_re0 /kernel: arp_add_mac: ae0 ifd_mfilter failed for mac 00:00:5e:00:01:10 error (12)
"In addition, you will observe that there may be issues with VRRP where logical units will either not be added or not be able to forward traffic."

The reason for the discrepancy is, in Junos, when an AE interface is used, the number of CAM (Content Addressable Memory) entries will be decided by its 'link-speed' configuration. For instance, when the link-speed is configured to 1g (gigabit), the AE interface will support up to 1024 CAM entries. Otherwise, on Gigabit (GE) interfaces and Fast Ethernet (FE) interfaces, if the link speed is not configured, and/or no child links are configured for the bundle, the 20 CAM entry limit will be set.

In regard to impact, the DCD_CONFIG_WRITE_FAILED message indicates that the configured AE interface does not have the memory available to allocate (in this case CAM entries) and thus, specific MAC addresses will not be added to the MAC database/ARP entries, and connectivity would be interrupted until the relevant MAC address is added to the table.

Thus, configuring the link-speed statement (see below) will address the issue.

lab> show configuration interfaces ae0 aggregated-ether-options
link-speed 1g;
...

Example - test and findings:
  1. Without the link-speed config, the following error messages occur:

    LabRouter_re0 /kernel: KERN_ARP_DUPLICATE_ADDR: duplicate IP address 10.84.136.34! sent from address: 00:00:5e:00:01:11 (error count = 1)
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 last message repeated 19 times
    LabRouter_re0 /kernel: ae_mfilter: ae0 limit of 20 filters exceeded for 00:00:5e:00:01:12 <<<<<<<<<

  • When VRRP is configured in the lab router the same error messages are reported.

  • When new logical units with VRRP were added, the issue could be reproduced:.

    LabRouter_re0 mgd[3880]: UI_COMMIT: User 'lab' performed commit: no comment
    LabRouter_re0 dcd[4839]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 init: ntp (PID 4704) exit on SIGHUP, will be restarted to get the new config
    LabRouter_re0 init: ntp (PID 5028) started
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 last message repeated 2 times
    LabRouter_re0 xntpd[5028]: ntpd 4.1.0-a Fri Nov 3 04:10:02 GMT 2006 (1)
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 last message repeated 15 times
    LabRouter_re0 /kernel: kernel overwrite ae0 link-speed with child speed 1000000000
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.401 (lidx=0) attached to bundle ae0.401
    LabRouter_re0 /kernel: bundle ae0.401: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.401 joins ae0.401
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.402 (lidx=0) attached to bundle ae0.402
    LabRouter_re0 /kernel: bundle ae0.402: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.402 joins ae0.402
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.403 (lidx=0) attached to bundle ae0.403
    LabRouter_re0 /kernel: bundle ae0.403: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.403 joins ae0.403
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.404 (lidx=0) attached to bundle ae0.404
    LabRouter_re0 /kernel: bundle ae0.404: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.404 joins ae0.404
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.405 (lidx=0) attached to bundle ae0.405
    LabRouter_re0 /kernel: bundle ae0.405: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.405 joins ae0.405
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.401 (lidx=1) attached to bundle ae0.401
    LabRouter_re0 /kernel: ge-1/2/1.401 joins ae0.401
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.402 (lidx=1) attached to bundle ae0.402
    LabRouter_re0 /kernel: ge-1/2/1.402 joins ae0.402
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.403 (lidx=1) attached to bundle ae0.403
    LabRouter_re0 /kernel: ge-1/2/1.403 joins ae0.403
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.404 (lidx=1) attached to bundle ae0.404
    LabRouter_re0 /kernel: ge-1/2/1.404 joins ae0.404
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.405 (lidx=1) attached to bundle ae0.405
    LabRouter_re0 /kernel: ge-1/2/1.405 joins ae0.405
    LabRouter_re0 vrrpd[3086]: VRRPD_NEWMASTER_TRAP: Interface ae0.401 (local addr: 10.84.148.18) is now master for vrrp-group 19
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 vrrpd[3086]: VRRPD_NEWMASTER_TRAP: Interface ae0.402 (local addr: 10.84.158.18) is now master for vrrp-group 20
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 last message repeated 3 times
    LabRouter_re0 vrrpd[3086]: VRRPD_NEWMASTER_TRAP: Interface ae0.403 (local addr: 10.84.168.18) is now master for vrrp-group 21
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 vrrpd[3086]: VRRPD_NEWMASTER_TRAP: Interface ae0.404 (local addr: 10.84.178.18) is now master for vrrp-group 22
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 vrrpd[3086]: VRRPD_NEWMASTER_TRAP: Interface ae0.405 (local addr: 10.84.188.18) is now master for vrrp-group 23
    LabRouter_re0 dcd[2979]: ae0 : aggregated-ether-options link-speed not set! default to 0
    LabRouter_re0 last message repeated 34 times
    LabRouter_re0 /kernel: ae_mfilter: ae0 limit of 20 filters exceeded for 00:00:5e:00:01:14 <<<<<<<<<<
    LabRouter_re0 /kernel: ae_mfilter: ae0 limit of 20 filters exceeded for 00:00:5e:00:01:15 <<<<<<<<<<
    LabRouter_re0 /kernel: ae_mfilter: ae0 limit of 20 filters exceeded for 00:00:5e:00:01:16 <<<<<<<<<<
    LabRouter_re0 /kernel: ae_mfilter: ae0 limit of 20 filters exceeded for 00:00:5e:00:01:17 <<<<<<<<<<

  • So, the faulty VRRP groups were removed.

  • Then the link-speed config was added and committed. All links bounced and came back up.

  • The configuration is shown below along with logs. There are no errors seen any more.

    lab@Batty_re0# show | compare rollback 2
    [edit interfaces ae0 aggregated-ether-options]
    + link-speed 1g; <<<<<<<<<<<<<<

    [edit interfaces ae0]

    ==== logs...========

    LabRouter_re0 mgd[3880]: UI_COMMIT: User 'lab' performed commit: no comment
    LabRouter_re0 rpd[3083]: task_connect: taskBGP_64512.10.64.0.5+179 addr 10.64.0.5+179: No route to host
    LabRouter_re0 rpd[3083]: bgp_connect_start: connect 10.64.0.5 (Internal AS 64512): No route to host
    LabRouter_re0 rpd[3083]: task_connect: task BGP_64512.10.64.0.11+179 addr 10.64.0.11+179: No route to host
    LabRouter_re0 rpd[3083]: bgp_connect_start: connect 10.64.0.11 (Internal AS 64512): No route to host
    LabRouter_re0 init: ntp (PID 5227) exit on SIGHUP, will be restarted to get the new config
    LabRouter_re0 init: ntp (PID 5620) started
    LabRouter_re0 xntpd[5620]: ntpd 4.1.0-a Fri Nov 3 04:10:02 GMT 2006 (1)
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.401 (lidx=0) attached to bundle ae0.401
    LabRouter_re0 /kernel: bundle ae0.401: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.401 joins ae0.401
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.402 (lidx=0) attached to bundle ae0.402
    LabRouter_re0 /kernel: bundle ae0.402: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.402 joins ae0.402
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.403 (lidx=0) attached to bundle ae0.403
    LabRouter_re0 /kernel: bundle ae0.403: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.403 joins ae0.403
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.404 (lidx=0) attached to bundle ae0.404
    LabRouter_re0 /kernel: bundle ae0.404: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.404 joins ae0.404
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/0.405 (lidx=0) attached to bundle ae0.405
    LabRouter_re0 /kernel: bundle ae0.405: bundle IFL state changed to UP
    LabRouter_re0 /kernel: ge-1/2/0.405 joins ae0.405
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.401 (lidx=1) attached to bundle ae0.401
    LabRouter_re0 /kernel: ge-1/2/1.401 joins ae0.401
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.402 (lidx=1) attached to bundle ae0.402
    LabRouter_re0 /kernel: ge-1/2/1.402 joins ae0.402
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.403 (lidx=1) attached to bundle ae0.403
    LabRouter_re0 /kernel: ge-1/2/1.403 joins ae0.403
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.404 (lidx=1) attached to bundle ae0.404
    LabRouter_re0 /kernel: ge-1/2/1.404 joins ae0.404
    LabRouter_re0 /kernel: ae_link_op: link ge-1/2/1.405 (lidx=1) attached to bundle ae0.405
    LabRouter_re0 /kernel: ge-1/2/1.405 joins ae0.405
    LabRouter_re0 rpd[3083]: task_connect: task BGP_64512.10.64.0.6+179 addr 10.64.0.6+179: No
Thus, the 'link-speed' is an integral part of the scalability of Gigabit Ethernet based Aggregated Ethernet bundles, to ensure that the maximum number of CAM entries are available for the interface and it's members.




In later versions of Junos, such as Junos 10.4R4.5, the "link-speed 1g" statement does not need to be added for the AE interface whose member links are Gigabit Ethernets; the default maximum number of CAM entries is 1K by default.
Refer to the following test: 9.4S3.1 VS 10.4R4.5 

***NOTE: The same test has been repeated in Junos versions 10.4R4.5, 11.1R3.5, 11.2R1.10, 11.4R1.14, 12.1R1.9, 12.2R1.8, with the same result as 10.4R4.5. All more recent versions of the mentioned tested versions will include the same behavior.***


On 9.4S3.1 (without link-speed knob) 

Created 250 VRRP groups for ae0.0 (ge-0/0/2 is member link)
lab@m10i-1-re0# show interfaces ae0
Jun 10 11:24:59
unit 0 {
    family inet {
        address 192.168.1.1/24 {
            vrrp-group 1 {
                virtual-address 192.168.1.2;
            }
        }
    ........

lab@m10i-1-re0# show interfaces ae0 |match vrrp | count
Count: 250 lines

Result:
Jun 10 12:30:03 m10i-1-re0 dcd[5301]: ae0 : aggregated-ether-options link-speed no kernel value! default to 0
Jun 10 12:30:06 m10i-1-re0 dcd[4550]: ae0 : aggregated-ether-options link-speed mismatch detected! reset to 0
Jun 10 12:30:06 m10i-1-re0 /kernel: kernel overwrite ae0 link-speed with child speed 1000000000  <---  set link-speed 1g automatically
Jun 10 12:30:06 m10i-1-re0 /kernel: ae_bundlestate_ifd_change: bundle ae0: bundle IFD state changed to UP
Jun 10 12:30:06 m10i-1-re0 /kernel: ae_link_op: link ge-0/0/2.0 (lidx=0) attached to bundle ae0.0
Jun 10 12:30:06 m10i-1-re0 /kernel: bundle ae0.0: bundle IFL state changed to UP
Jun 10 12:30:06 m10i-1-re0 /kernel: ge-0/0/2.0 joins ae0.0
Jun 10 12:30:06 m10i-1-re0 /kernel: ae_bundlestate_ifd_change: bundle ae0: bundle IFD minimum links not met 0 < 1
Jun 10 12:30:06 m10i-1-re0 /kernel: ae_bundlestate_ifd_change: bundle ae0: bundle IFD state changed to DOWN
Jun 10 12:30:06 m10i-1-re0 /kernel: bundle ae0.0: bundle IFL state changed to DOWN
Jun 10 12:30:07 m10i-1-re0 /kernel: ae_bundlestate_ifd_change: bundle ae0: bundle IFD state changed to UP
Jun 10 12:30:07 m10i-1-re0 /kernel: bundle ae0.0: bundle IFL state changed to UP
Jun 10 12:30:59 m10i-1-re0 /kernel: arp_add_mac: ae0 ifd_mfilter failed for mac 00:00:5e:00:01:f3 error (12)
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory <--- due to 20 ae_mfilter limit
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory
Jun 10 12:30:59 m10i-1-re0 dcd[4550]: DCD_CONFIG_WRITE_FAILED: configuration write failed for an RT ADD: Cannot allocate memory

lab@m10i-1-re0> show arp no-resolve | match ae0 | count
Count: 17 lines <---- only created 17 arp items, less than vrrp group numbers. Proved ae only support 20 ae_mfilter limit without link-speed knob.

lab@m10i-1-re0> show arp no-resolve | match ae0
00:00:5e:00:01:01 192.168.1.2 ae0.0 permanent published
00:00:5e:00:01:02 192.168.2.2 ae0.0 permanent published
00:00:5e:00:01:03 192.168.3.2 ae0.0 permanent published
00:00:5e:00:01:04 192.168.4.2 ae0.0 permanent published
00:00:5e:00:01:05 192.168.5.2 ae0.0 permanent published

On 10.4R4.5 (without link-speed knob)

Created 250 VRRP groups for ae0.0 (ge-0/0/2 is member link)

lab@m10i-1-re0# show interfaces ae0
Jun 10 11:24:59
unit 0 {
    family inet {
        address 192.168.1.1/24 {
            vrrp-group 1 {
                virtual-address 192.168.1.2;
            }
        }
    ......


lab@m10i-1-re0# show interfaces ae0 |match vrrp | count
Count: 250 lines

Result:
Jun 10 11:12:05 m10i-1-re0 /kernel: ge-0/0/2.0 joins ae0.0
Jun 10 11:12:06 m10i-1-re0 /kernel: ae_bundlestate_ifd_change: bundle ae0: bundle IFD state changed to UP
Jun 10 11:12:06 m10i-1-re0 /kernel: bundle ae0.0: bundle IFL state changed to UP
Jun 10 11:12:36 m10i-1-re0 dcd[14640]: ae0 : Warning: aggregated-ether-options link-speed no kernel value! default to 0 <--- no link-speed configured
Jun 10 11:12:47 m10i-1-re0 dcd[14610]: ae0 : aggregated-ether-options link-speed set to kernel value of 1000000000 <---- set link-speed 1g automatically
Jun 10 11:12:48 m10i-1-re0 dcd[14610]: ae0 : aggregated-ether-options link-speed set to kernel value of 1000000000 <--- no “Cannot allocate memory” error

lab@m10i-1-re0> show arp no-resolve | match ae0 | count   <---  successfully created 250 ARP items, match VRRP group numbers. Proved ae support more than 20 ae_mfilter limit without link-speed knob.
Count: 250 lines

lab@m10i-1-re0> show arp no-resolve | match ae0
00:00:5e:00:01:01 192.168.1.2 ae0.0 permanent published
00:00:5e:00:01:02 192.168.2.2 ae0.0 permanent published
00:00:5e:00:01:03 192.168.3.2 ae0.0 permanent published
00:00:5e:00:01:04 192.168.4.2 ae0.0 permanent published
00:00:5e:00:01:05 192.168.5.2 ae0.0 permanent published
00:00:5e:00:01:06 192.168.6.2 ae0.0 permanent published
00:00:5e:00:01:07 192.168.7.2 ae0.0 permanent published
00:00:5e:00:01:08 192.168.8.2 ae0.0 permanent published
00:00:5e:00:01:09 192.168.9.2 ae0.0 permanent published
00:00:5e:00:01:0a 192.168.10.2 ae0.0 permanent published
00:00:5e:00:01:0b 192.168.11.2 ae0.0 permanent published