Description

This article provides a detailed step-by-step procedure to troubleshoot generic Virtual Private LAN Service (VPLS) Forwarding Plane issues.

Symptoms

  • Cannot ping from CE to CE.   (If the CE is a L2 device, then also try pinging from an end system at the site.)
  • Cannot forward traffic from CE-CE
  • VPLS connection is UP, but traffic is not passing.  

    Example of ' show vpls connections ' output:

    <snip>
    Instance: vpls
      Local site: CE2 (20)
        connection-site              Type   St   Time last up         # Up trans
        10                           rmt     Up   Apr 10 14:05:52 2012 1 <-------
          Remote PE: 1.1.1.1, Negotiated control-word: No
          Incoming label: 800025, Outgoing label: 800035
          Local interface: vt-1/2/10.1051905, Status: Up, Encapsulation: VPLS
             Description: Intf - vpls vpls local site 20 remote site 10


Note: If the VPLS connection is not UP, then go to KB25025 - Resolution Guide - MX - VPLS Troubleshooting (Control Plane) - Connection not listed or down [juniper.net] .

Solution

Topology used for this guide:

alt
 

Perform the following steps:

Note: While these troubleshooting steps can be applied for all the code versions and all the platforms, this troubleshooting guide is written primarily keep ing MX platform in mind.

For the flowchart version of these steps, click the flowchart icon: alt
 

  1. This is a new setup or new configuration (which includes a new VPLS routing instance being configured on the affected PE or a new CE bought up)?

    • No - Continue to Step 2  (existing setup and was working properly before)
    • Yes - Jump to Step 4

  2. Before the problem started, were there any config or hardware/software changes (such as the following)?:

      • New sites added to the existing VPLS network
      • Upgrade or downgrade of the Junos code on the affected PE
      • Replacing a Routing-Engine or FPC on the router
    • Yes - Continue to Step 3
    • No - Jump to Step 4

  3. Perform the following steps to revert to the previous known working configuration.

    Note: Perform this during a maintenance window. Ifit cannot be done, continue to Step 4 to continue troubleshooting.

    a.  Save the current non-working configuration.

    b.  Collect the logs/information in the KB22637 - [M, MX, T Routers] Data Collection Checklist [juniper.net] under the 'VPLS' section.  
    Note:  This is an important step to complete before proceeding to Step c.

    c. Revert to the previous working state and see if you can resolve the issue, i.e.
    • Reverting to the previous Junos version on which you did not see any issue
    • Rollback to a previous working configuration.          

    Did reverting to the previous working condition work?

    • Yes - Analyze the differences in the logs collected
    • No - Continue to Step 4

  4. Verify the status of the pseudowire on the PEs in the path of the CEs that are not able to communicate. To do this, run the command ‘ show vpls connections extensive ’.

    For example, in the PE2 Router output below (taken from the sample topology), the PE2 VPLS connection is UP with PE1.   

    user@PE2# run show vpls connections    
Layer-2 VPN connections:
---SNIPPET----
Instance: vpls  
Local site: CE2 (20)  
connection-site           Type  St     Time last up          # Up trans  
10                        rmt   Up     Apr 10 14:05:52 2012           1  <-------  
Remote PE: 1.1.1.1, Negotiated control-word: No  
Incoming label: 800025, Outgoing label: 800035  
Local interface: vt-1/2/10.1051905, Status: Up, Encapsulation: VPLS  
Description: Intf - vpls vpls local site 20 remote site 10

    Are the VPLS pseudowires UP on the PE’s?

  5. Run VPLS ping over the pseudowire to see if forwarding is working between the PE’s.

    The ‘ping vpls instance’ command relies on the LSP ping and trace infrastructure defined in RFC 4379. When you issue a ping vpls instance command, a chassis MAC address is drawn from the ingress PE router's pool of MAC addresses and used to create the VPLS ping packet. The ping packet is then forwarded to the egress PE router. When the egress PE router receives the ping packet, it learns the MAC address from the VPLS ping packet. The MAC address is added to the egress PE router's MAC table. The same thing happens when the egress router replies to the ingress router's ping request and the ingress router learns the MAC addresses of the ingress router
    Example output:
    user@PE1> ping vpls instance red destination-mac 00:89:67:1a:23:6f source-ip 10.255.17.138
    ! -> PE1:VPLS_INSTANCE:ge-4/1/1.0
    ! -> PE1:VPLS_INSTANCE:ge-4/1/1.0
    ! -> PE1:VPLS_INSTANCE:ge-4/1/1.0
    ! -> PE1:VPLS_INSTANCE:ge-4/1/1.0

    --- vpls ping statistics ---
    4 packets transmitted, 4 packets received, 0% packet loss
    In the above example output, as CE1 is not able to ping CE2, a vpls ping is initiated from the ingress (PE1) PE routers ‘VPLS_INSTANCE’ routing-instance and with destination MAC address of the egress router (PE2) and IP address of the outgoing interface of the ingress PE router (PE1).

    Since the VPLs instance does not have any interface with family inet, so we need to configure an IRB interface within the same address range that CPEs are configure
    set interface irb unit <x> family inet address <x>
    set routing instance red routing-interface irb. <x>

    so when generating the ping we could have an address to assign to the probe.

    Also please make sure that under the remote PE you are trying to ping, the ping mpls is open otherwise we will have drops packets.

    Does VPLS ping work between the PE’s?

  6.  Verify if all the interested PE routers have learned the MAC addresses of the CE devices involved. To do this, run the command ‘ show route forwarding-table family vpls ’. 

    Working example output: 
    user@PE1# run show route forwarding-table family vpls       
Routing table: VPLS_INSTANCE.vpls             
VPLS:
Destination        Type RtRef Next hop          Type  Index NhRef Netif
default            dynm     0                   flood   293     1
default            perm     0                   dscd    280     1
ge-0/2/0.4         dynm     0                   flood   295     1   00:90:69:6c:5c:5d/48
                   dynm     0                   ucst   294      2 ge-0/2/0.4  
00:90:69:d1:ec:bc/48
                   dynm     0                   indr   305     4
                                                Push 800000, Push 100016(top) so-1/1/2.0

    In the above example output, we see that there is a flood entry for interface ge-0/2/0.4 which is for the CE facing interface and two MAC addresses one for the local CE and another for the remote CE device.

    Alternatively, the command ‘ show vpls mac-table ’ can also be used to verify that the interested PE routers have learned the MAC addresses of the CE devices.

    Example output:

    user@PE1> show vpls mac-table
    MAC flags (S -static MAC, D -dynamic MAC,
    SE -Statistics enabled, NM -Non configured MAC)

    Routing instance : vpls_red
    MAC MAC Logical
    address flags interface
    00:90:69:6c:5c:5d D ge-0/2/0.4
    00:90:69:d1:ec:bc D lsi.1051138

    Are MAC addresses correctly programmed on the interested PE routers?

  7. Investigate if there is any Firewall Filter or Policer applied on the path between the two interested CE devices.

    Tip:  One method to do this is to use the ping utility and ICMP filters and verify where the drops are happening on the path between the CEs. A VPLS Firewall Filter can be created on each CE to verify if the packets are reaching the PEs. On PE1, the source-mac-address for a host at CE1 can be specified, and on PE2, the source-mac-address for a host at CE2 can be specified.

    The Filter or Policer may be on the CE facing or core facing side of the ingress PE or egress PE router.

    Verify the configuration of the Firewall Filter or policer and see if it has any issues.

    If possible, deactivate the Firewall Filter or Policer and verify if you can ping from CE to CE.

    Can ping or forward traffic after deactivating the Firewall Filter or Policer?

    • Yes - Identify/fix what is causing the Firewall or Policer to block the forwarding of packets. 
    • No - If you are not able to ping CE-CE, even after deactivating the Firewall Filter or Policer, continue to Step 8 . 

  8. If possible, during a maintenance window, perform the following steps to deactivate/activate the CE-facing ifl and VPLS instance:

    a.  Save the current configuration.

    b.  Collect the logs/information in the KB22637 - [M, MX, T Routers] Data Collection Checklist [juniper.net] under the 'VPLS' section.  

    Note:  This is an important step to complete before proceeding to Step c.

    c. Deactivate/activate the CE-facing ifl (logical interface going towards the affected CE) on the interested PE router.

    Try ping from CE to CE again.    If forwarding is still not working, then continue to Step d .
    d. Deactivate/activate the VPLS instance on the interested PE in which you have defined the affected site.

    Try ping from CE to CE again.  If forwarding is still not working, then continue to Step 9 .

  9. Check if no-tunnel-services is used on the interested PE router.  If so, verify the restrictions below.

    Keep in mind the following restrictions that may apply for normal operation of VPLS.  (Note: These restrictions apply primarily to M/T.)

    • An enhanced FPC is required.
    • ATM1 interfaces are not supported.
    • Aggregated SONET/SDH interfaces are not supported as core-facing interfaces.
    • Channelized interfaces are not supported as core-facing interfaces.
    • GRE-encapsulated interfaces are not supported as core-facing interfaces.


    If the VPLS issue is still not resolved, collect the logs/information in the KB22637 - [M, MX, T Routers] Data Collection Checklist [juniper.net] under the 'VPLS' section, and open a case with your technical support representative.

Related Information