Product Affected

This is a zero day issue which affects all JUNOS routers configured to generate MPLS TTL Expired packets.
Critical
8.5 (AV:N/AC:L/Au:N/C:N/I:P/A:C)

Problem

An obscure flaw in mbuf (memory buffer) management and recovery can cause a slow mbuf memory leak, which in turn will cause a JUNOS router to run out of mbufs. The mbuf leak occurs over the span of weeks and can be observed using the ' show system buffers ' command as shown below.

The condition has been correlated to the reception of MPLS TTL Expired packets on the affected router. mbuf packet headers were observed to increment when the TTL expired MPLS packet counter increments: 

user@junos> show system buffers            
Nov 03 12:06:44
8773/8800/120000 mbufs in use (current/peak/max):
        8507 mbufs allocated to packet headers
        9 mbufs allocated to socket options
        1 mbufs allocated to pfe data
        256 mbufs allocated to fxp data (rx)
4510/13530/30000 mbuf clusters in use (current/peak/max)
29260 Kbytes allocated to network (32% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

user@junos> show system statistics mpls    
Nov 03 12:06:47
mpls:
        10823 total mpls packets received
        675052 packets forwarded
        0 packets dropped
        0 with header too small
        0 after tagging, can't fit link MTU
        0 with IPv4 explicit NULL tag
        0 with IPv4 explicit NULL cksum errors
        0 with router alert tag
        0 lsp ping packets (ttl-expired/router alert)
        10823 with ttl expired
        0 with tag encoding error
        0 packets discarded, no route
        1084 packets used first nexthop in ecmp unilist

user@junos> show system buffers            
Nov 03 12:09:42
8780/8800/120000 mbufs in use (current/peak/max):
        8513 mbufs allocated to packet headers
        9 mbufs allocated to socket options
        1 mbufs allocated to socket send data
        1 mbufs allocated to pfe data
        256 mbufs allocated to fxp data (rx)
4513/13530/30000 mbuf clusters in use (current/peak/max)
29260 Kbytes allocated to network (32% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

user@junos> show system statistics mpls    
Nov 03 12:09:45
mpls:
        10830 total mpls packets received
        675596 packets forwarded
        0 packets dropped
        0 with header too small
        0 after tagging, can't fit link MTU
        0 with IPv4 explicit NULL tag
        0 with IPv4 explicit NULL cksum errors
        0 with router alert tag
        0 lsp ping packets (ttl-expired/router alert)
        10830 with ttl expired
        0 with tag encoding error
        0 packets discarded, no route
        1085 packets used first nexthop in ecmp unilist

Only one mbuf is lost per TTL Expired packet generated.

Note: In later releases of JUNOS, the format of the ' show system buffers ' command has changed. Monitoring "bytes allocated to network" will provide an indication that mbufs are being leaked due to MPLS TTL Expired packets.

Solution


All JUNOS software releases built on or after 2010-03-18 have fixed this specific issue. Releases containing the fix specifically include: 8.5S5, 9.1S8, 9.3S8, 9.6S4, 10.0S3.1, 10.1R1 and all subsequent releases (i.e. all releases built after 10.1R1).

This issue is being tracked as PR 485635. While this PR is not viewable by customers, it can be used as a reference when discussing the issue with JTAC.

KB16765 [juniper.net] - "In which releases are vulnerabilities fixed?" describes which release vulnerabilities are fixed as per our End of Engineering and End of Life support policies.


How to obtain Service Releases:

Security vulnerabilities are fixed in the next available Maintenance Release of each supported JUNOS version. In some cases, a Maintenance Release is not planned to be available in an appropriate time-frame. For these cases, Service Releases are made available in order to be more timely. Security Advisory notices will indicate which Maintenance and/or Service Releases contain fixes for the issues described. Upon request to JTAC, customers will be provided download instructions for a Service Release. Although Juniper does not provide formal Release Note documentation for a Service Release, a list of "PRs fixed" can be provided on request.

Workaround


Enabling ' no-propagate-ttl ' under [edit protocols mpls] will reduce the impact of this issue by avoiding TTL Exceeded replies as a result of traceroute.

Severity Assessment

Information for how Juniper Networks uses CVSS can be found at KB 16446 [juniper.net] "Common Vulnerability Scoring System (CVSS) and Juniper's Security Advisories."

Related Information