• Remote SPAN configuration required on EVPN/VXLAN ERB scenario.
• Needs to capture traffic at the spine and send it to a collector connected to one of the remote leaves.

• Need to enable remote port mirroring on a lean spine switch.
• Traffic should be sent through a GRE tunnel to a remote monitoring station.
• Utilizing the port mirroring instance method.
• Operating within an EVPN-VXLAN ERB architecture.
• Lean spine devices don't terminate VXLAN tunnels but provide IP forwarding.
• Mirroring is required for both intra-VLAN (bridged) and inter-VLAN (routed) traffic.
• Tenant-specific information isn't provisioned at lean spines.
• Analyzer port capacity is a concern; mirroring all overlay traffic may overload the analyzer port.
• Frame drops may occur if mirroring exceeds monitoring station capacity.

1. Remote Port Mirroring in EVPN-VXLAN IP Fabrics:

• Utilize GRE tunneling to send mirrored traffic to a monitoring station.
• GRE is essential for non-routable Layer 2 or overlay Layer 3 traffic.
• GRE tunnels are established once IP reachability to the monitoring subnet is established via the underlay.

1. Port Mirroring Methods: Analyzer Instance and Port Mirroring Instance:

• Choose between Analyzer Instance (simplified) and Port Mirroring Instance (configurable).
• Specific scenarios may require a Port Mirroring Instance for fine-grained control.

1. Supported Data Center Architectures:

• Port mirroring configurations differ based on your data center architecture.
• In EVPN-VXLAN ERB, lean spines are not VXLAN-aware; they use local VTEP addresses.
• Configuration should accommodate mirroring all overlay traffic sent by the leaf device.

Topology:

Configuration Steps:

1. Deactivate BGP on Spine 2 (Optional):

• Deactivating BGP on Spine 2 simplifies testing for mirroring between Leaf 1 and Leaf 4.

jcluser@spine2# deactivate protocols bgp

jcluser@spine2# commit

• Ensure all traffic goes through Spine 1 for mirroring purposes.

{master:0}

jcluser@leaf1> show route 10.1.255.14

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

10.1.255.14/32    *[BGP/170] 1d 01:58:39, localpref 100

                     AS path: 65001 65014 I, validation-state: unverified

                   > to 10.1.11.1 via xe-0/0/6.0

:vxlan.inet.0: 14 destinations, 14 routes (14 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

10.1.255.14/32    *[Static/1] 1d 01:58:35, metric2 0

                   > to 10.1.11.1 via xe-0/0/6.0

1. Create a Mirroring Instance at Spine 1:

• Define a mirroring instance to send mirrored traffic to the remote monitoring station.

jcluser@spine1# set forwarding-options port-mirroring instance mirror-leaf1-and-leaf4 family inet output ip-address 172.16.1.2

1. Set Up Firewall Filters at Spine 1: Filters are based on source addresses (10.1.255.11 and 10.1.255.14).

• Filters match the source and destination IP addresses of Leaf 1 and Leaf 4.

{master:0}[edit firewall family inet filter port-mirror-from-leaf1]

jcluser@spine1# show | display set | no-more

set firewall family inet filter port-mirror-from-leaf1 term 1 from source-address 10.1.255.11/32

set firewall family inet filter port-mirror-from-leaf1 term term1 from destination-address 10.1.255.14/32

set firewall family inet filter port-mirror-from-leaf1 term term1 then count from-leaf1-vtep

set firewall family inet filter port-mirror-from-leaf1 term term1 then port-mirror-instance mirror-leaf1-and-leaf4

set firewall family inet filter port-mirror-from-leaf1 term term1 then accept

set firewall family inet filter port-mirror-from-leaf1 term term2 then accept

• Set Firewall Filters for Traffic Sent from Leaf 1 and Leaf 4:

{master:0}[edit firewall family inet filter port-mirror-from-leaf1]

jcluser@spine1# top edit firewall family inet filter port-mirror-from-leaf4

{master:0}[edit firewall family inet filter port-mirror-from-leaf4]

jcluser@spine1# show | display set | no-more

set firewall family inet filter port-mirror-from-leaf4 term term1 from source-address 10.1.255.14/32

set firewall family inet filter port-mirror-from-leaf4 term term1 from destination-address 10.1.255.11/32

set firewall family inet filter port-mirror-from-leaf4 term term1 then count from-leaf4-vtep

set firewall family inet filter port-mirror-from-leaf4 term term1 then port-mirror-instance mirror-leaf1-and-leaf4

set firewall family inet filter port-mirror-from-leaf4 term term1 then accept

set firewall family inet filter port-mirror-from-leaf4 term term2 then accept

1. Apply Filters to Fabric Interfaces at Spine 1.

Filters are applied to Leaf 1 and Leaf 4-facing fabric interfaces.

jcluser@spine1# set interfaces xe-0/0/6 unit 0 family inet filter input port-mirror-from-leaf4

jcluser@spine1# set interfaces xe-0/0/7 unit 0 family inet filter input port-mirror-from-leaf1

• By using input filters that match on the local leaf's VTEP as a source address and the remote leaf's VTEP as a destination address, we ensure that only overlay traffic between these two leaves is mirrored. If you wish to mirror all VXLAN traffic sent by a leaf, regardless of the destination leaf, simply match on the local leaf's source address in your filter.

5. Add Additional Filters (Optional):

• In this example so far, we have assumed that Server 1 is single-homed only to Leaf 1. If the server is multihomed to Leaf 1 and Leaf 2, you need to adapt the filters shown to also catch traffic sent from Leaf 2 to Leaf 4, and vice versa.
• Modify filters to accommodate multihomed servers connected to multiple leaf switches.

Imput from step 3.

{master:0}[edit firewall family inet filter port-mirror-from-leaf4]

jcluser@spine1# show | display set

set firewall family inet filter port-mirror-from-leaf4 term term1 from source-address 10.1.255.14/32

set firewall family inet filter port-mirror-from-leaf4 term term1 from destination-address 10.1.255.11/32

set firewall family inet filter port-mirror-from-leaf4 term term1 then count from-leaf4-vtep

set firewall family inet filter port-mirror-from-leaf4 term term1 then port-mirror-instance mirror-leaf1-and-leaf4

set firewall family inet filter port-mirror-from-leaf4 term term1 then accept

set firewall family inet filter port-mirror-from-leaf4 term term2 then accept

A quick modification to the existing port-mirror-from-leaf4 filter is needed to also match on the

destination address of Leaf's 2 VTEP and create an input filter for Spine 1's Leaf 2-facing interface. (No applied on this example)

set firewall family inet filter port-mirror-from-leaf4 term term1 from source-address 10.1.255.12/32

6. Commit Changes at Spine 1:

• Review and commit the configuration changes at Spine 1.

jcluser@spine1# show | compare rollback 2

[edit interfaces]

[edit interfaces xe-0/0/6 unit 0 family inet]

+      filter {

+          input port-mirror-from-leaf4;

+      }

[edit interfaces xe-0/0/7 unit 0 family inet]

+      filter {

+          input port-mirror-from-leaf1;

+      }

[edit forwarding-options]

+  port-mirroring {

+      instance {

+          mirror-leaf1-and-leaf4 {

+              family inet {

+                  output {

+                      ip-address 172.16.1.2;

+                  }

+              }

+          }

+      }

+  }

[edit]

+ firewall {

+     family inet {

+         filter port-mirror-from-leaf4 {

+             term term1 {

+                 from {

+                     source-address {

+                         10.1.255.14/32;

+                     }

+                     destination-address {

+                         10.1.255.11/32;

+                     }

+                 }

+                 then {

+                     count from-leaf4-vtep;

+                     port-mirror-instance mirror-leaf1-and-leaf4;

+                     accept;

+                 }

+             }

+             term term2 {

+                 then accept;

+             }

+         }

+         filter port-mirror-from-leaf1 {

+             term 1 {

+                 from {

+                     source-address {

+                         10.1.255.11/32;

+                     }

+                 }

+             }

+             term term1 {

+                 from {

+                     destination-address {

+                         10.1.255.14/32;

+                     }

+                 }

+                 then {

+                     count from-leaf1-vtep;

+                     port-mirror-instance mirror-leaf1-and-leaf4;

+                     accept;

+                 }

+             }

+             term term2 {

+                 then accept;

+             }

+         }

+     }

+ }

7. Modify Configuration at Leaf 3:

• Configure the xe-0/0/0 interface connected to the monitor station.

jcluser@bl-leaf3# show interfaces xe-0/0/0

unit 0 {

   family inet {

       address 172.16.1.1/24;

   }

}

• Ensure the monitoring station's IP address is reachable in the fabric underlay.
• The leaf device connected to the monitoring station does not need explicit GRE configuration. This is because the GRE tunnel terminates at the monitoring station. You must ensure that the monitoring station's IP address is reachable in the fabric underlay. Stated differently, the spine cannot send the GRE encapsulated traffic to the monitor station if it does not have underlay reachability to the monitor station.

8. Modify Underlay Export Policy at Border Leaf 3:

• Add a new term to the existing underlay export policy to advertise the monitoring station's prefix.

{master:0}[edit policy-options policy-statement send-direct]

jcluser@bl-leaf3# show | display set

set policy-options policy-statement send-direct term 1 from protocol direct

set policy-options policy-statement send-direct term 1 from route-filter 10.1.255.13/32 exact

set policy-options policy-statement send-direct term 1 then accept

set policy-options policy-statement send-direct term 2 from protocol direct

set policy-options policy-statement send-direct term 2 from route-filter 172.16.1.0/24 exact

set policy-options policy-statement send-direct term 2 then accept

• Commit the changes on border Leaf 3.

jcluser@bl-leaf3# top show | compare rollback 4

[edit interfaces]

+  xe-0/0/3 {

+      unit 0 {

+          family inet {

+              address 10.1.13.2/30;

+          }

+      }

+  }

[edit policy-options policy-statement send-direct]

    term 1 { ... }

+   term 2 {

+       from {

+           protocol direct;

+           route-filter 172.16.1.0/24 exact;

+       }

+       then accept;

+   }

Verification:

1. Verify Underlay Connectivity: Confirm connectivity from Spine 1 to the monitoring station by checking routes and pinging the monitoring station.

{master:0}

jcluser@spine1> show route 172.16.1.0/24

inet.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

172.16.1.0/24     *[BGP/170] 10:18:46, localpref 100

                     AS path: 65013 I, validation-state: unverified

                   > to 10.1.13.2 via xe-0/0/5.0

{master:0}

jcluser@spine1> ping 172.16.1.2 count 2

PING 172.16.1.2 (172.16.1.2): 56 data bytes

64 bytes from 172.16.1.2: icmp_seq=0 ttl=63 time=115.706 ms

64 bytes from 172.16.1.2: icmp_seq=1 ttl=63 time=116.592 ms

--- 172.16.1.2 ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max/stddev = 115.706/116.149/116.592/0.443 ms

1. Confirm Port Mirroring Instances: Ensure the remote port mirroring state is up on Spine 1.

{master:0}[edit]

jcluser@spine1# run show forwarding-options port-mirroring detail

Instance Name: mirror-leaf1-and-leaf4

 Instance Id: 2

 Input parameters:

   Rate                 : 1

   Run-length           : 0

   Maximum-packet-length : 0

 Output parameters:

   Family             State    Destination         Next-hop

   inet               up       172.16.1.2          .local..0

1. Verify Filters: Verify that filters applied to Spine 1 correctly reflect test traffic by checking firewall counters.

[labuser@linuxpc ~]$ ping -I eth1 -c 10 -W 1 10.1.101.20

PING 10.1.101.20 (10.1.101.20) from 10.1.101.101 eth1: 56(84) bytes of data.

64 bytes from 10.1.101.20: icmp_seq=1 ttl=64 time=131 ms

64 bytes from 10.1.101.20: icmp_seq=2 ttl=64 time=133 ms

64 bytes from 10.1.101.20: icmp_seq=3 ttl=64 time=130 ms

64 bytes from 10.1.101.20: icmp_seq=4 ttl=64 time=227 ms

64 bytes from 10.1.101.20: icmp_seq=5 ttl=64 time=130 ms

64 bytes from 10.1.101.20: icmp_seq=6 ttl=64 time=311 ms

64 bytes from 10.1.101.20: icmp_seq=7 ttl=64 time=132 ms

64 bytes from 10.1.101.20: icmp_seq=8 ttl=64 time=126 ms

64 bytes from 10.1.101.20: icmp_seq=9 ttl=64 time=128 ms

64 bytes from 10.1.101.20: icmp_seq=10 ttl=64 time=133 ms

{master:0}[edit]

jcluser@spine1# run show firewall

Filter: port-mirror-from-leaf1

Counters:

Name                                               Bytes             Packets

from-leaf1-vtep                                        0                   0

Filter: port-mirror-from-leaf4

Counters:

Name                                               Bytes             Packets

from-leaf4-vtep                                        0                   0

{master:0}[edit]

jcluser@spine1# run show firewall

Sep 02 05:09:29

Filter: port-mirror-from-leaf1

Counters:

Name                                               Bytes             Packets

from-leaf1-vtep                                        0                   0

Filter: port-mirror-from-leaf4

Counters:

Name                                               Bytes             Packets

from-leaf4-vtep                                     1532                  12

1. Capture and Decode Traffic: Use Wireshark or a similar tool to capture and decode the mirrored traffic for analysis.

• This detailed configuration guide outlines the steps to enable remote port mirroring on a lean spine within an EVPN-VXLAN ERB architecture, allowing for comprehensive traffic monitoring and analysis.

Remote Port Mirroring for EVPN-VXLAN Fabrics