Description

In a Junos Fusion deployment with dual Aggregation Devices (ADs) and Satellite Devices (SDs) configured as dual-homed mode, the satellites might be not able to form Link Aggregation Control Protocol (LACP) bundles with attached servers via extended ports.

Symptoms

Customers have multiple LACP bundles between VM servers and Satellite Devices extended ports, and they noticed all the LACP bundles were stuck in the 'Attached' state after they rebooted the SD devices. 

  • All the physical ports are showing in UP state and no Layer 2 errors.
  • The LACP statistics counters indicate packets from both RX and TX direction are increasing.
  • Bounced extended ports between SD and servers, and cascade ports between SD and AD did not work.
  • Rebooted SDs, and it did not help to resolve the issue neither.
  • SDs are configured in dual-home mode but only one AD (AD2) is available. The other AD (AD1) is offline because of an administration issue.    
{MASTER}
root@QFX> show lacp interfaces ae688 extensive
Apr 10 07:45:23
Aggregated interface: ae688
    LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-98/0/28     Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-98/0/28   Partner    No    No   Yes  Yes   No   Yes     Fast    Active
      xe-99/0/28     Actor    No    No    No   No  Yes   Yes     Fast    Active
      xe-99/0/28   Partner    No    No    No   No   No   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State
      xe-98/0/28                Current   Fast periodic           Attached
      xe-99/0/28                Current   Fast periodic           Attached
    LACP info:        Role     System             System       Port     Port    Port
                             priority         identifier   priority   number     key
      xe-98/0/28     Actor        127  00:00:00:00:00:01        127     8478     689
      xe-98/0/28   Partner      65535  ec:f4:bb:e7:de:c8        255        1      15
      xe-99/0/28     Actor        127  00:00:00:00:00:01        127     8734     689
      xe-99/0/28   Partner      65535  ec:f4:bb:e7:de:c8        255        2      15
 
root@QFX> show interfaces xe-98/0/28
Apr 10 07:45:51
Physical interface: xe-98/0/28, Enabled, Physical link is Up, Extended Port, multi-homed
  Interface index: 3703, SNMP ifIndex: 5426
  Description: c1-compute045.eby.lc3.com:eth0
  Link-level type: Ethernet, MTU: 1518, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None,
  Ethernet-Switching Error: None, Source filtering: Disabled
  Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Flow control: Disabled,
  Media type: Fiber
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Current address: 54:4b:8c:5b:81:1e, Hardware address: 54:1e:56:48:7f:21
  Last flapped   : 2019-04-10 07:30:35 UTC (00:15:16 ago)
  Input rate     : 2704 bps (3 pps)
  Output rate    : 0 bps (0 pps)
  Active alarms  : None
  Active defects : None
  Interface transmit statistics: Disabled
 
  Logical interface xe-98/0/28.0 (Index 678) (SNMP ifIndex 5468)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Input packets : 10136
    Output packets: 0
    Protocol aenet, AE bundle: ae688.0
 
root@QFX> show interfaces xe-99/0/28
Physical interface: xe-99/0/28, Enabled, Physical link is Up, Extended Port, multi-homed
  Interface index: 3660, SNMP ifIndex: 5510
  Description: c1-compute045.eby.lc3.com:eth1
  Link-level type: Ethernet, MTU: 1518, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, Loop Detect PDU Error: None,
  Ethernet-Switching Error: None, Source filtering: Disabled
  Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Flow control: Disabled,
  Media type: Fiber
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Current address: 54:4b:8c:5b:81:1e, Hardware address: 54:1e:56:47:ee:21
  Last flapped   : 2019-04-10 07:30:35 UTC (00:15:22 ago)
  Input rate     : 904 bps (1 pps)
  Output rate    : 0 bps (0 pps)
  Active alarms  : None
  Active defects : None
  Interface transmit statistics: Disabled
 
  Logical interface xe-99/0/28.0 (Index 992) (SNMP ifIndex 5553)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Input packets : 5083
    Output packets: 0
    Protocol aenet, AE bundle: ae688.0

root@QFX> show lacp statistics interfaces ae688
Apr 10 07:46:10
Aggregated interface: ae688
    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx
      xe-99/0/28                76          41            0            0
      xe-98/0/28               151          78            0            0
 

root@QFX> show lacp statistics interfaces ae688
Apr 10 07:46:14
Aggregated interface: ae688
    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx
      xe-99/0/28                79          42            0            0
      xe-98/0/28               156          80            0            0
 

root@QFX> show lacp statistics interfaces ae688
Apr 10 07:46:15
Aggregated interface: ae688
    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx
      xe-98/0/28               160          82            0            0
      xe-99/0/28                80          43            0            0

Solution

When a Satellite Device (SD) boots up, it will receive a configuration file from its Aggregation Devices (ADs). If the Satellite Device is configured in dual-homed mode, the configuration file instructs the SD to communicate to both ADs. If one of the ADs is offline at the time the SD trying to connect with it, due to a broken cascade port connection or entire chassis down, the SD will stay in the malfunction state and not be able to properly transmit LACP PDUs until it communicates and synchronizes with both ADs as specified in the received configuration file.

In order to recover the SDs LACP bundles with the attached VM servers in the single-homed AD scenario, the following work-around can be applied to convert the SDs from multi-home to single-home mode. 

delete chassis satellite-management redundancy-groups rg1 satellite 98
delete chassis satellite-management redundancy-groups rg1 satellite 99
set chassis satellite-management single-home satellite 98
set chassis satellite-management single-home satellite 99
commit


Example: After applying the configuration change above, the LACP bundle ae688 comes up online. Both member links are in the distributing state. The work-around successfully resolved the LACP issue. 

{MASTER}
root@qfx> show lacp interfaces ae688
Apr 11 21:25:40
Aggregated interface: ae688
LACP state: Role    Exp Def Dist Col Syn Aggr Timeout Activity
xe-99/0/28  Actor   No  No  Yes  Yes Yes Yes  Fast    Active
xe-99/0/28  Partner No  No  Yes  Yes Yes Yes  Fast    Active
xe-98/0/28  Actor   No  No  Yes  Yes Yes Yes  Fast    Active
xe-98/0/28  Partner No  No  Yes  Yes Yes Yes  Fast    Active

LACP protocol: Receive State   Transmit State   Mux State
xe-99/0/28     Current         Fast periodic    Collecting distributing
xe-98/0/28     Current         Fast periodic    Collecting distributing