Multihop BGP session negotiates MSS 500 for its TCP connection. Observed this behaviour while working with a request reported that the BGP session for a set of bgp neighbors is showing 500 MSS. 

The reported device has MPLSoUDP between these end systems, and the user reported this problem, concerned about the low MSS between these BGP peers. 

When you observe a BGP connection with an MSS of 500, despite an interface with a default MTU of 1514 or more, it indicates a very significant reduction in the TCP segment size. This drastic decrease points to specific network conditions or configurations that are severely limiting the effective MTU. Here's a breakdown of the potential causes:

Key Factors Leading to a Low MSS:

• • Tunnels (GRE, IPSec): This is the most probable cause. Tunnels introduce substantial overhead due to encapsulation. GRE, IPSec, MPLSoUDP, or combinations of these can add numerous bytes to each packet. If your BGP sessions are traversing tunnels, the available payload space is greatly diminished. The device must reduce the MSS to accommodate the tunnel headers.
  • VPNs: Similar to tunnels, VPNs also add significant overhead.
  • MPLS with Extensive Label Stacking: While MPLS overhead is typically less than tunnel overhead, extensive label stacking (multiple MPLS labels) can accumulate, significantly reducing the effective MTU.
  • Extremely Restrictive QoS Policies: In rare cases, highly restrictive Quality of Service (QoS) policies might be configured to limit packet sizes, indirectly affecting the MSS.
  • Misconfiguration: There may be a misconfiguration somewhere along the path. This could involve manually configured, overly restrictive MSS settings. Incorrect MTU settings on intermediate devices.
  • Fragmentation Issues: Although TCP is designed to handle fragmentation, severe or repeated fragmentation can lead devices to aggressively reduce the MSS. This could be due to Path MTU Discovery (PMTUD) issues or inconsistent MTU settings.
  • Legacy Systems: Older network devices or systems might have limitations or default configurations that result in smaller packet sizes.
  • Link Layer Overhead: Some link layer protocols, especially those that are not standard Ethernet, can have very high overhead.

• In this request configuration on the reported device was not using MTU discovery for the BGP peers reported.
• At the same time, there was one PEER on the same devices RSI shows MTU DISCOVERY enabled. For this PEER MTU, in requester cli output ("show system connections extensive") shows 1448mss. 
• Checked the behaviour in the lab for multihop BGP neighbour shows 500mss in "show system connections extensive" output when the mtu-discovery configuration is not used. As soon as adding the config mtu-discovery configuration on both end devices, BGP config and refreshed the neighbour, then the mss moved to 1448 mss for the BGP session. 
• There is MPLSoUDP between these 2 BGP neighbours to match the customer-mentioned setup, and IGP is also running for loopback reachability along with MPLS. 
• Moved the test to confirm if any flow overlay on top of these devices shows less MTU for any application data flow. 
• Verified with file copy and flow shows MTU 1448mss end-to-end CEs (CE1 to CE4) which are using above mentioned BGP endpoints. 
• Below is the setup used to verify this in the lab. 

BGP session with mss 1448 after mtu discovery is added on PE. 
root@CE4_RE> show system connections extensive    
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address                                 Foreign Address                               (state)
tcp4    5792      0  10.54.14.245.22                               10.54.16.221.56735                            ESTABLISHED
   sndsbcc:          0 sndsbmbcnt:          0  sndsbmbmax:     266432
sndsblowat:       2048 sndsbhiwat:      33304
   rcvsbcc:       5792 rcvsbmbcnt:       9216  rcvsbmbmax:     532864
rcvsblowat:          1 rcvsbhiwat:      66608
jnxinpflag:       4096 inprtblidx:          0    inpdefif:          0
       iss:  172938861      sndup:  174305435
    snduna:  174305435     sndnxt:  174305435      sndwnd:      66608
    sndmax:  174305435    sndcwnd:     131070 sndssthresh: 1073725440
       irs: 1687453678      rcvup:  687848160
    rcvnxt:  687848160     rcvadv:  687908976      rcvwnd:      66608
       rtt:          0       srtt:         20        rttv:         11
    rxtcur:       1200   rxtshift:          0       rtseq:  174305399
    rttmin:       1000  mss:       1448        jlocksmode:          1
     flags: NODELAY REQ_SCALE RCVD_SCALE REQ_TSTMP RCVD_TSTMP SACK_PERMIT [0x20003e4]

Troubleshooting Steps:

• • Trace the Path: Use traceroute or similar tools to identify the path your iBGP traffic is taking. Look for tunnels, VPNs, or MPLS networks.
  • Packet Capture: Capture packets on the BGP interfaces using Wireshark or similar tools. Analyze the packet headers to determine the overhead introduced by tunnels, MPLS, or other protocols.
  • Check Device Configurations: Review the configurations of all devices along the path, paying close attention to MTU, MSS, tunnel, VPN, and QoS settings.
  • Verify PMTUD: Ensure that PMTUD is enabled and functioning correctly.
  • Look for Tunnels and VPNs: This is the most likely cause, so check all routers and firewalls along the path for configured tunnels or VPNs.
  • MPLS verification: If the network uses MPLS, verify the number of labels that are being pushed onto the packets.

An MSS of 500 is very low, and it will drastically reduce the throughput of your BGP sessions. Therefore, it is important to find the cause of the problem.

In this request configuration on the reported device was not using MTU discovery for the BGP peers reported.