Applications that open and close a large number of client TCP/IP sockets run the risk of running out of available socket ports. This can happen in a load and performance testing scenario using a tool like LISA Test from iTKO, or it could happen in a production environment if an active application simply needs to rapidly open and close a large number of outbound connections.
On the .NET platform, the exception raised reads “
System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted <host>:<port>“.
In Java, the exception is “
java.net.BindException: Address already in use: connect“.
Both exceptions are misleading because they are generally associated with server socket conflicts – not outbound client socket connections. However, a better understanding of the TCP state machine sheds some light on this behavior – and a solution.
Common Port Conflict Exceptions
Whenever TCP/IP encounters a port conflict, you can expect one of the two following exceptions to be thrown depending upon your environment:
In a C# environment, you’ll see this exception:
System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted <host>:<port>
In Java, you’ll see this:
java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
If you see these exceptions thrown when a server socket attempts to listen for incoming connections, the cause is obvious: the port you’re attempting to listen on is already in use. For example, if you bring up a Web server, but another Web server is already running, an exception will be thrown because port 80 (or 8080) is already listening on behalf of another thread or application.
These exceptions are often confusing, however, when thrown setting up a client connection. Client TCP connections are always assigned an OS-selected port on the local side, so why is the operating system selecting an active port? The truth is the exception indicates no local port numbers are available to the client. This misreporting of the error by the OS is half the confusion.
Tuning Local Client Port Range
The problem is two-fold. First, Linux and Windows make only a certain number of ports available to client sockets – the default is in the range of 1024 to 5000. Hence, you may have only 3,976 active client connections at a time. For most systems, this is plenty. However, in specific circumstances on systems requiring a large number of outbound connections, this limit can be exhausted.
This range, however, can be tuned. On Windows, the upper bound for client port assignments can be adjusted using the
DWORD value on this registry key:
Of course, rather than using the cumbersome
regedit, you can download IPTuner for free to quickly optimize your Windows IP stack
On Linux, both the lower and upper bounds can be set using the following parameter:
net.ipv4.ip_local_port_range = 32768 65536
How this parameter is set in Linux varies depending upon the flavor and version of Linux you’re using – and you’ll need to restart networking after you change it.
Tuning TCP TIME_WAIT Timeout Value
The second cause of these exceptions has to do with the TCP state model and the way sockets are closed. Even after a socket has officially been “closed”, it hangs around in a
TIME_WAIT state as a safety mechanism to deal with stray packets. The default wait time on all operating systems, generally, is ridiculously long (240 seconds on Windows). So, even if an application doesn’t require a lot of concurrent connections, it can still run out of available ports in a high load situation. If even one connection is repeatedly opened and closed fast enough, you’ll soon have all available local sockets hanging around in a
TIME_WAIT state and none available for new clients.
TIME_WAIT state duration, however, is also tunable.
On Windows, using the same registry key, the
TCPTimedWaitDelay value can be used to adjust the
TIME_WAIT duration from 30 to 300 seconds. Of course, rather than using the cumbersome
regedit, you can download IPTuner for free to quickly optimize your Windows IP stack.
On Linux, the wait delay is configured using the following parameter:
net.ipv4.tcp_fin_timeout = 30
By decreasing the TCP wait delay, closed sockets spend less time in the
TIME_WAIT state and get returned to the pool of available client ports faster. However, to avoid communication problems, do not lower this value below 30 seconds.
Administrator Privileges Required
On both Linux and Windows (even if using IP tuner or
regedit), you’ll require administrator privileges to change these parameters. However, anyone can view these settings to at least verify if they make sense.