This article describes a possible scenario with a design issue in a nested VM (Virtual Machine).
NorthStar Controller is supported to work in a single VM by nested virtualization using VMware ESXi. Under a particular network setting between ESXi host server and L3 node device with bridging ports, an unexpected issue may occur. Any broadcast/unknown-unicast/multicast packets from Junos VM in NorthStar Controller are never able to communicate out of ESXi nodes.
In the example setup below, the Linux OS which accommodates NorthStar Controller (NorthStar APP and Junos VM) is located in ESXi host server. The ESXi host server has 2 NIC ports; and they are connected to L3 switch for the sake of redundancy.
The circled numbers in the figure correspond to the following lines to explain the problematic scenario:
[1]
Junos VM transmits ARP Request by broadcast to find the gateway IP address.
[2]
At the port external0 (Linux bridge), ARP request is received from "port no 2" side. MAC Address of Junos VM 52:54:00:11:11:00 is learnt in external0. [root@linux ~]# brctl showmacs external0 port no mac addr is local? ageing timer 2 52:54:00:11:11:00 no 0.00 Linux port eth0 is bound to esternal0 bridge. Since this ARP packet destines broadcast, it goes through vSwitch and being sent out from physical NIC port2 of ESXi host server. Here is the frame captured at external0 at this time. Size is 42 Bytes. 16:08:19.935832 52:54:00:11:11:00 -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.0.0.254? Tell 10.0.0.1 0000 ff ff ff ff ff ff 52 54 00 11 11 00 08 06 00 01 ......RT. Q..... 0010 08 00 06 04 00 01 52 54 00 11 11 00 0a 63 01 e6 ......RT. Q..c.. 0020 00 00 00 00 00 00 0a 63 01 e4 .......c..
At the port external0 (Linux bridge), ARP request is received from "port no 2" side.
MAC Address of Junos VM 52:54:00:11:11:00 is learnt in external0.
[root@linux ~]# brctl showmacs external0 port no mac addr is local? ageing timer 2 52:54:00:11:11:00 no 0.00 Linux port eth0 is bound to esternal0 bridge. Since this ARP packet destines broadcast, it goes through vSwitch and being sent out from physical NIC port2 of ESXi host server. Here is the frame captured at external0 at this time. Size is 42 Bytes. 16:08:19.935832 52:54:00:11:11:00 -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.0.0.254? Tell 10.0.0.1 0000 ff ff ff ff ff ff 52 54 00 11 11 00 08 06 00 01 ......RT. Q..... 0010 08 00 06 04 00 01 52 54 00 11 11 00 0a 63 01 e6 ......RT. Q..c.. 0020 00 00 00 00 00 00 0a 63 01 e4 .......c..
[root@linux ~]# brctl showmacs external0 port no mac addr is local? ageing timer 2 52:54:00:11:11:00 no 0.00
16:08:19.935832 52:54:00:11:11:00 -> ff:ff:ff:ff:ff:ff ARP 42 Who has 10.0.0.254? Tell 10.0.0.1 0000 ff ff ff ff ff ff 52 54 00 11 11 00 08 06 00 01 ......RT. Q..... 0010 08 00 06 04 00 01 52 54 00 11 11 00 0a 63 01 e6 ......RT. Q..c.. 0020 00 00 00 00 00 00 0a 63 01 e4 .......c..
When ARP request is sent from Physical NIC Port2, padding data 0000... is added because of filling out to meet the minimum length of Ethernet frame, 64 bytes. Note: Captured packet shows 60 bytes because the 4 last bytes of FCS(Frame Check Sequence) is omitted.
ARP request is received by VLAN L3 interface(vlan.x, irb.x, etc.) in Switch. Now, the packet itself is broadcast. Hence, it is flooded to the same L2 segment.
[5]
As a result of [4], ARP request packet is transmitted towards another physical NIC port 1 of ESXi host server. By this process, ARP request packet arrives at ESXi host and goes back through vSwitch.
[6]
The flooded ARP request is received at "port no 1" of external0, hence MAC 52:54:00:11:11:00 moves to "port no 1" as below. [root@linux ~]# brctl showmacs external0 port no mac addr is local? ageing timer 1 52:54:00:11:11:00 no 0.50 Also, the ARP request packet was added padding data in [3], then it is seen as 60 bytes in capture hex data at external0. 16:08:19.935895 52:54:00:11:11:00 -> ff:ff:ff:ff:ff:ff ARP 60 Who has 10.0.0.254? Tell 10.0.0.1 0000 ff ff ff ff ff ff 52 54 00 11 11 00 08 06 00 01 ......RT. Q..... 0010 08 00 06 04 00 01 52 54 00 11 11 00 0a 63 01 e6 ......RT. Q..c.. 0020 00 00 00 00 00 00 0a 63 01 e4 00 00 00 00 00 00 .......c........ <-- 18 bytes of "0" are padded. 0030 00 00 00 00 00 00 00 00 00 00 00 00 ............ This broadcast packet is forwarded to Junos VM. In/Out both packets can be collected in Junos VM. 03:17:47.589523 Out 52:54:00:11:11:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42 : arp who-has 10.0.0.254 tell 10.0.0.1 03:17:47.589745 In 52:54:00:11:11:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60 : arp who-has 10.0.0.254 tell 10.0.0.1
The flooded ARP request is received at "port no 1" of external0, hence MAC 52:54:00:11:11:00 moves to "port no 1" as below.
[root@linux ~]# brctl showmacs external0 port no mac addr is local? ageing timer 1 52:54:00:11:11:00 no 0.50
Also, the ARP request packet was added padding data in [3], then it is seen as 60 bytes in capture hex data at external0.
16:08:19.935895 52:54:00:11:11:00 -> ff:ff:ff:ff:ff:ff ARP 60 Who has 10.0.0.254? Tell 10.0.0.1 0000 ff ff ff ff ff ff 52 54 00 11 11 00 08 06 00 01 ......RT. Q..... 0010 08 00 06 04 00 01 52 54 00 11 11 00 0a 63 01 e6 ......RT. Q..c.. 0020 00 00 00 00 00 00 0a 63 01 e4 00 00 00 00 00 00 .......c........ <-- 18 bytes of "0" are padded. 0030 00 00 00 00 00 00 00 00 00 00 00 00 ............
This broadcast packet is forwarded to Junos VM. In/Out both packets can be collected in Junos VM.
03:17:47.589523 Out 52:54:00:11:11:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42 : arp who-has 10.0.0.254 tell 10.0.0.1 03:17:47.589745 In 52:54:00:11:11:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60 : arp who-has 10.0.0.254 tell 10.0.0.1
During [5] is processed in the switch, ARP reply is sent by unicast from L3 switch vlan interface. Generally, this happens later than [5] because generating ARP reply packet takes more time than broadcast packet forwarding.
ARP reply by [7] reaches to external0. 16:08:19.937041 00:33:33:33:33:33 -> 52:54:00:11:11:00 ARP 60 10.0.0.254 is at 00:33:33:33:33:33 0000 52 54 00 11 11 00 00 33 33 33 33 33 08 06 00 01 RT. Q.$......... 0010 08 00 06 04 00 02 00 33 33 33 33 33 0a 63 01 e4 ......$......c.. 0020 52 54 00 11 11 00 0a 63 01 e6 00 00 00 00 00 00 RT. Q..c........ 0030 00 00 00 00 00 00 00 00 00 00 00 00 ............ However, the second ARP request by [6] has already reached to external0 and Junos VM's MAC 52:54:00:11:11:00 was learnt as "port no 1" side. Therefore, the ARP reply is returned back to vSwitch direction. As a result, ARP resolution in Junos VM never completes since ARP reply is unable to reach to Junos VM. In fact, external0 keeps learning "port no 1" for Junos VM's MAC most of its working. Momentarily, index 2 is seen 1 out of 10000 collecting 'brctl showmacs <>'. [root@linux ~]# cat showmac.log | grep -C3 -E "^ 2" 1 52:54:00:11:11:00 no 0.19 1 52:54:00:11:11:00 no 0.19 1 52:54:00:11:11:00 no 0.19 2 52:54:00:11:11:00 no 0.00 <-- This moment 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 -- -- 1 52:54:00:11:11:00 no 0.49 1 52:54:00:11:11:00 no 0.50 1 52:54:00:11:11:00 no 0.50 2 52:54:00:11:11:00 no 0.00 <-- This moment 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 --
ARP reply by [7] reaches to external0.
16:08:19.937041 00:33:33:33:33:33 -> 52:54:00:11:11:00 ARP 60 10.0.0.254 is at 00:33:33:33:33:33 0000 52 54 00 11 11 00 00 33 33 33 33 33 08 06 00 01 RT. Q.$......... 0010 08 00 06 04 00 02 00 33 33 33 33 33 0a 63 01 e4 ......$......c.. 0020 52 54 00 11 11 00 0a 63 01 e6 00 00 00 00 00 00 RT. Q..c........ 0030 00 00 00 00 00 00 00 00 00 00 00 00 ............
However, the second ARP request by [6] has already reached to external0 and Junos VM's MAC 52:54:00:11:11:00 was learnt as "port no 1" side. Therefore, the ARP reply is returned back to vSwitch direction. As a result, ARP resolution in Junos VM never completes since ARP reply is unable to reach to Junos VM. In fact, external0 keeps learning "port no 1" for Junos VM's MAC most of its working. Momentarily, index 2 is seen 1 out of 10000 collecting 'brctl showmacs <>'.
[root@linux ~]# cat showmac.log | grep -C3 -E "^ 2" 1 52:54:00:11:11:00 no 0.19 1 52:54:00:11:11:00 no 0.19 1 52:54:00:11:11:00 no 0.19 2 52:54:00:11:11:00 no 0.00 <-- This moment 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 -- -- 1 52:54:00:11:11:00 no 0.49 1 52:54:00:11:11:00 no 0.50 1 52:54:00:11:11:00 no 0.50 2 52:54:00:11:11:00 no 0.00 <-- This moment 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 1 52:54:00:11:11:00 no 0.00 --
If Junos VM does not need to send BUM packet in the customer's design, configuring static ARP entry can be a solution or workaround.
In NorthStar Controller, Junos VM uses PCEP and BGP. However, it depends on the customer's requirement. (OSPF/ISIS are options)
set interfaces em1 unit 0 family inet address 10.0.0.1 arp 10.0.0.254 mac 00:33:33:33:33:33
As another workaround, setting aging timer to zero in Linux can solve the situation. However, you must confirm the Linux OS usage with the customer.
brctl setaging external0 0
The root cause is that ESXi host server has two available interfaces with remote switch. It creates a type of L2 loop condition among the physical switch, vSwitch, and bridge ports. Network design may be needed to consider changing that Aggregation/Teaming, or other interface control feature is to be used.