Description

This article explains the behavior of a Virtual Chassis (VC) when the em0 interface on one member is connected to a network port on the other member and vice versa (reference diagram in the next section). It also explains the steps to overcome this behavior and considerations to make. This behavior is most likely to be seen on EX4600 devices and QFX5k series devices involved in VC.

Symptoms

Consider a 2 member ( FPC 0 and FPC 1) Virtual Chassis scenario with FPC 0 as the primary. Let us consider the network ports and em0/em1 connected as below:

 ---------                                   ---------
 | FPC 0 |---(em0)--------------(ge-1/0/4)---| FPC 1 |
 ---------                                   ---------
     |                                           |
     |                                           |
 (ge-0/0/4)------------------------------------(em1)

In this scenario, it is possible to observe that the IRB interface will not be able to communicate with the VME interface. 

The MAC addresses assigned for multiple cases is as shown below:​

Base Address: 

root# run show chassis mac-addresses   
FPC 0
    Base address     12:12:12:12:a5:00
    Count            1280

IRB MAC Address: 

[master:0][edit]
root# run show interfaces irb extensive | match hard
  Current address: 12:12:12:12:a5:00, Hardware address: 12:12:12:12:a5:00

VME MAC Address:

[master:0][edit]
root# run show interfaces vme extensive | match hard   
  Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a5:01


Management Interface MAC:

  [master:0][edit]
root# run show interfaces em0 extensive | match hard   
  Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a9:f8

As we can see in the above outputs, IRB MAC address is the same as the Base address. This is not a favorable case and leads PFE to drop the packets.

Solution

This issue is seen when the IRB MAC address takes the same MAC address as the Base address for the FPC.

In order to overcome the situation, there has been a hidden-knob introduced for setups similar to the one depicted above. The irb-mac-shift   knob under the packet-forwarding-options  hierarchy directs the IRB MAC to change to "Base MAC + 2"

Example: Considering the outputs provided in the Symptoms section, ​the base address being 12:12:12:12:a5:00, the IRB MAC address changes to "base_address +2". The resulting IRB MAC becomes 12:12:12:12:a5:02.

Enable IRB MAC Shift: 

[master:0][edit]
root# set system packet-forwarding-options irb-mac-shift

Commit the configuration change: 

As you try to commit the configuration, the system indicates that a reboot is mandatory in order for the change to be reflected. Please make sure you are prepared for a reboot while you make this configuration change. 

root# show system packet-forwarding-options               
irb-mac-shift;
 
[master:0][edit]
root# commit
[edit system]
  'packet-forwarding-options'
    warning: Reboot the system for the irb-mac-shift to take effect
configuration check succeeds
commit complete

Perform Reboot: 

[master:0][edit]
root# run request system reboot
Reboot the system ? [yes,no] (no) yes
*** FINAL System shutdown message from root@ ***                            
System going down IMMEDIATELY
Once the device is up and running after a reboot, there should be no issues seen w.r.t the communication between IRB and VME interfaces. 

Verify the IRB MAC: 
root# run show interfaces irb extensive | match hard
  Current address: 12:12:12:12:a5:02, Hardware address: 12:12:12:12:a5:02
    Flags: Hardware-Down Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 
[master:0][edit]
root# run show interfaces vme extensive | match hard
  Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a5:01
 
[master:0][edit]
root# run show interfaces em0 extensive | match hard
  Current address: 12:12:12:12:a5:01, Hardware address: 12:12:12:12:a9:f8

***IMPORTANT: This knob is only necessary for this specific network setting where management interface and network ports are connected in the form of a ring in VC formation. In other scenarios, configuring this is NOT recommended. Consult JTAC if you want to make sure about the need of this configuration as per your network setup***

Notes: 
  1. The management interface of the switch could be em/me ( Here, the document contains em0/em1 for demonstration purpose)
  2. The hidden knob can be used in the following releases: ​junos:14.1X53-D47, junos:17.1R3, junos:17.2R3, junos:17.3R4, junos:17.4R2, junos:18.1R3, junos:18.2R2, junos:18.3R1.
  3. The " irb-mac-shift " knob does not modify VME interface MAC and network port addresses.
  4. This knob makes sense only for regular system upgrades (not for NSSU/ISSU) as a reboot is required for the change to take effect. 
  5. The knob is of hidden nature. Using a tab key for completion will not auto-populate the knob. Please type the entire knob manually to avoid errors.
============================================================

Similar scenario added on 2023-06-28 in Junos 21.4R3.X

>>> Topology:

EX4600_r2062 (Master_RE) === (em0 and ge-0/0/10) === (ge-0/0/12 and ge-0/0/10) EX4600_r2060 (Layer-2 switch)

|              |

EX4600_r2063 (Backup_RE)


** Interface em0 on EX4600_r2062 is connecting to access port ge-0/0/12 which is a member of VLAN100 on L2 switch.

** Port ge-0/0/12 on both EX4600_r2062 and L2 switch are accessing VLAN100 as well.

** Interface irb.100 is enabled with address 10.0.100.254/24 residing in virtual router “Server” on EX4600-VC.

** Interface vme.0 is configured with address 10.0.100.1/24 on EX4600-VC


>>> Test output after adding “system packet-forwarding-options irb-mac-shift”:

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run show interfaces terse | grep "em0|vme|irb"

Jun 26 07:19:17

em0                     up    up

em0.0                   up    up   eth-switch

irb                     up    up

irb.100                 up    up   inet     10.0.100.254/24

vme                     up    up

vme.0                   up    up   inet     10.0.100.1/24 
 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run show version | grep Junos:

Jun 26 07:17:28

Junos: 21.4R3.15

Junos: 21.4R3.15

 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run show interfaces irb extensive | grep hardware   

Jun 26 07:15:32

  Current address: 68:f3:8e:c2:35:00, Hardware address: 68:f3:8e:c2:35:00

{master:0}[edit]

root@jtac-ex4600-40f-r2062# set system packet-forwarding-options irb-mac-shift

Jun 26 07:15:49

{master:0}[edit]

root@jtac-ex4600-40f-r2062# show | compare

Jun 26 07:15:53

[edit]

+  system {

+      packet-forwarding-options {

+          irb-mac-shift;

+      }

+  }
 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# commit        <<<<< No warning messages on commit requesting to reboot the VC.

Jun 26 07:15:55

configuration check succeeds

fpc1:

warning: [groups default] is protected, 'groups default system host-name' cannot be changed

warning: [groups default] is protected, 'groups default interfaces em0 unit 0 family inet address 10.219.40.234/26' cannot be created

warning: [groups default] is protected, 'groups default interfaces em0 unit 0 family inet address 10.219.40.235/26' cannot be deleted

warning: [groups default] is protected, 'groups default system host-name' cannot be deleted

commit complete

commit complete
 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run show interfaces irb extensive | grep hardware    

Jun 26 07:16:14

  Current address: 68:f3:8e:c2:35:00, Hardware address: 68:f3:8e:c2:35:00        <<<<< IRB MAC address remains the same.

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run show arp no-resolve

Jun 26 07:16:52

MAC Address       Address         Interface                Flags

68:f3:8e:c2:35:00 10.0.100.254    vme.0                    none                <<<<< No change on MAC address of irb interface.

02:00:00:00:00:0b 128.0.0.5       bme0.0                   permanent

02:00:00:00:00:0b 128.0.0.6       bme0.0                   permanent

02:00:00:00:00:0b 128.0.0.17      bme0.0                   permanent

68:f3:8e:c2:35:03 192.168.1.1     em2.32768                none

Total entries: 5

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run ping 10.0.100.1 routing-instance Server source 10.0.100.254        <<<<< Ping is yet not working.

Jun 26 07:17:11

PING 10.0.100.1 (10.0.100.1): 56 data bytes

^C

--- 10.0.100.1 ping statistics ---

4 packets transmitted, 0 packets received, 100% packet loss 


>>> Output after changing IRB MAC address via “set interfaces irb.100 mac 68:f3:8e:c2:35:02”: 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# set interfaces irb.100 mac 68:f3:8e:c2:35:02

Jun 26 07:21:14


{master:0}[edit]

root@jtac-ex4600-40f-r2062# show | compare

Jun 26 07:21:17

[edit interfaces irb unit 100]

+    mac 68:f3:8e:c2:35:02;

 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# commit

Jun 26 07:21:19

configuration check succeeds

fpc1:

warning: [groups default] is protected, 'groups default system host-name' cannot be changed

warning: [groups default] is protected, 'groups default interfaces em0 unit 0 family inet address 10.219.40.234/26' cannot be created

warning: [groups default] is protected, 'groups default interfaces em0 unit 0 family inet address 10.219.40.235/26' cannot be deleted

warning: [groups default] is protected, 'groups default system host-name' cannot be deleted

commit complete

commit complete

 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run show arp no-resolve

Jun 26 07:21:35

MAC Address       Address         Interface                Flags

68:f3:8e:c2:35:02 10.0.100.254    vme.0                    none                <<<<<<<<< IRB MAC address was changed accordingly.

02:00:00:00:00:0b 128.0.0.5       bme0.0                   permanent

02:00:00:00:00:0b 128.0.0.6       bme0.0                   permanent

02:00:00:00:00:0b 128.0.0.17      bme0.0                   permanent

68:f3:8e:c2:35:03 192.168.1.1     em2.32768                none

Total entries: 5
 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run ping 10.0.100.1 routing-instance Server source 10.0.100.254    <<<<<<< Ping works

Jun 26 07:21:44

PING 10.0.100.1 (10.0.100.1): 56 data bytes

64 bytes from 10.0.100.1: icmp_seq=0 ttl=64 time=12.652 ms

64 bytes from 10.0.100.1: icmp_seq=1 ttl=64 time=12.058 ms

64 bytes from 10.0.100.1: icmp_seq=2 ttl=64 time=11.139 ms

^C

--- 10.0.100.1 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max/stddev = 11.139/11.950/12.652/0.622 ms
 

{master:0}[edit]

root@jtac-ex4600-40f-r2062# run show arp no-resolve                                           

Jun 26 07:21:49

MAC Address       Address         Interface                Flags

68:f3:8e:c2:35:01 10.0.100.1      irb.100 [ge-0/0/10.0]    none

68:f3:8e:c2:35:02 10.0.100.254    vme.0                    none

02:00:00:00:00:0b 128.0.0.5       bme0.0                   permanent

02:00:00:00:00:0b 128.0.0.6       bme0.0                   permanent

02:00:00:00:00:0b 128.0.0.17      bme0.0                   permanent

68:f3:8e:c2:35:03 192.168.1.1     em2.32768                none

Total entries: 6
 

>>> Summary based on test results above:

1. Configuration knob “set system packet-forwarding-options irb-mac-shift” is not applicable for Junos 21.4R3.15. That’s why it’s not working.

2. PFE will drop packets destined to IRB MAC address same as chassis MAC so we need to change IRB MAC address to a different one via command “set interfaces irb.X mac XXX” manually.

3. IRB MAC address change also aligns with the rule "base MAC + 2" as long as no other interfaces on the VC using the same MAC.

Modification History

2021-03-24: Updated the article terminology to align with Juniper's Inclusion & Diversity initiatives

2023-06-28: Added similar scenario in Junos 21.4R3.X

Related Information