Description

This article explains why AP-P Out of Port Errors are seen in a lightly used CGNAT system, and how to determine if a single user is exceeding their allotted number of ports?

Symptoms

An MX configured for CGNAT with a low number of unique pool users is seeing AP-P Out of Port Errors incrementing consistently over time. The AP-P error applies to cases where the pool has paired pooling causing the same external address to be used for all sessions originating from the same internal host. The error indicates that there are no free ports or blocks available. However, there should be plenty of ports available given the number of active users. So, why would the counter still be incrementing? The system in this case is configured for Port Block Allocation using a block size of 200 ports and up to 8 blocks allocated to each user: 

pool nat44-pool1 {
    address 13.13.0.0/21;
    port {
        automatic;
        secured-port-block-allocation block-size 200 max-blocks-per-address 8 active-block-timeout 0;
    }
    address-allocation round-robin;
    mapping-timeout 120;
}

The address allocation is round-robin, ensuring that each user who initiates a connection gets a unique address so that a single address in the pool is not over utilized. NAT rules are configured for address-pooling paired: 

rule r1 {
    match-direction input;
    term t1 {
        from {
            source-address {
                3.0.0.0/8;
            }
        }
        then {
            translated {
                source-pool nat44-pool1;
                translation-type {
                    napt-44;
                }
                address-pooling paired;
            }
            syslog;
        }
    }
}

There are only 100 users active on the system, with 1699 ports and 107 port blocks in use. But, AP-P out of port errors are incrementing: 
lab> show services nat pool detail 
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 1699, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 192
    Max number of port blocks used: 107, Current number of port blocks in use: 107, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 1
    Unique pool users: 100

lab> show services nat pool detail    
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 1699, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 240
    Max number of port blocks used: 107, Current number of port blocks in use: 107, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 1
    Unique pool users: 100

Solution


The counter is incrementing in this instance because a single user has exceeded the number of ports that it is allowed. The configuration is set for 8 blocks of 200 addresses. One of the 100 unique pool users is going over that limit: 

lab> show services nat mappings detail | match 1600 
Ports In Use     :  1600 

lab> show services nat mappings detail    
Interface: sp-0/0/0, Service set: nat44

NAT pool: nat44-pool1

Mapping          : 3.0.0.16        --> 13.13.0.104       
Ports In Use     :     1 
Session Count    :     1 
Mapping State    : Active 
...
Mapping          : 3.0.0.3         --> 13.13.0.201       
Ports In Use     :  1600 
Session Count    :     1 
Mapping State    : Active

To see that this is happening without viewing the full mapping table, the counter for 'Port blocks limit exceeded errors' should be referenced. In this case, it is set to 1. That counter shows the number of users who have exceeded the number of ports they are allocated. Every time that user requests a port beyond its allocated number, the AP-P port error will increment: 
lab> show services nat pool detail    
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 1699, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 336
    Max number of port blocks used: 107, Current number of port blocks in use: 107, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 1
    Unique pool users: 100

lab> show services nat pool detail    
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 1699, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 408
    Max number of port blocks used: 107, Current number of port blocks in use: 107, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 1
    Unique pool users: 100

The block limit remains the same. If the user falls back under the 1600 port limit, the out of port error will stop rising. But, the block limit exceeded error will stay active as long as that user is still using any ports. Shown below is output with each of the 100 users only using a single port: 
lab> show services nat pool detail    
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 100, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 408
    Max number of port blocks used: 107, Current number of port blocks in use: 100, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 1
    Unique pool users: 100

If the unique user tied to the port block limit exceeded error goes away, i.e. has all of its flows removed due to the inactivity timer (or TCP connection being closed), the counter for limit exceeded remains at 1: 
lab> show services nat pool detail    
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 99, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 408
    Max number of port blocks used: 107, Current number of port blocks in use: 99, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 1
    Unique pool users: 99

If the user comes back again later and again consumes all 1600 ports available, the number of limit exceeded errors will increment to 2 and AP-P out of port will start to rise again: 
lab> show services nat pool detail    
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 1699, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 458
    Max number of port blocks used: 107, Current number of port blocks in use: 107, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 2
    Unique pool users: 100

lab> show services nat pool detail    
Interface: sp-0/0/0, Service set: nat44
  NAT pool: nat44-pool1, Translation type: dynamic
    Address range: 13.13.0.1-13.13.7.254
    Port range: 512-65535, Ports in use: 1699, Out of port errors: 0, Max ports used: 1699
    AP-P out of port errors: 483
    Max number of port blocks used: 107, Current number of port blocks in use: 107, Port block allocation errors: 0, Port block memory allocation errors: 0
    Port blocks limit exceeded errors: 2
    Unique pool users: 100

Since the block limit exceeded counter is historical and not real-time, there is no easy way to determine how many unique users are currently going beyond their limit for number of available ports. Thus, determining if the AP-P out of port errors are tied to that problem can be difficult as it requires parsing the entire NAT mapping table. Ideally, there would be a counter for the number of active users exceeding port block limits, or some specific syslog message that indicated when a user was consuming all the ports available to them.