UsingRealBandwith

How many of you have notice that no matter you have a 100 Mb line you only get near 2 Mbps. Well that is because the TCP window size is only of 32 kB or 64 kB on Linux (depends on distribution) and 8 kB on M$ Windos.

The TCP window size is the amount of data that will be send on a connection before a host stops and waits for an acknowledgment. This is used by TCP to prevent congestion. Ideally it should be:

@@WARNING@@ If your window size is too small, you won't use the network to it's full capacity If your window size is too big, you risk overloading the network and creating congestion and packet loss * On a WAN, setting the TCP window size correctly plays a big part in getting good performance (it can easily double performance or more)

The peak bandwidth of the link is typically expressed in Mbps. The round-trip delay for a link can be measured with traceroute or ping, and for high-speed WAN links is typically between 10 msec and 100 msec. For a 60 msec, 120 Mbps path, the bandwidth*delay product would be 7200 kbit, or 900 kByte (kB).

TCP Window size on Linux in order to achieve higher bandwidth.

#cd /proc/sys/net/core 
#ls 
message_burst    netdev_max_backlog      rmem_default    wmem_default 
message_cost     optmem_max      rmem_max        wmem_max 

The important file holding the secret are: /proc/sys/net/core/rmem_default - default receive window /proc/sys/net/core/rmem_max - maximum receive window /proc/sys/net/core/wmem_default - default send window /proc/sys/net/core/wmem_max - maximum send window

# cat    wmem_default    wmem_max        rmem_default    rmem_max 
65535 
65535 
65535 
65535 
# 

(If you change these numbers you are changing the TCP window)

The theorical values are 65535 on all of them because asume bandwidth of 100 Mbits/s and the round trip time was 5 msec, the TCP window should be:

(100x10^6) bytes/sec * (5x10^-3) sec = 65000 bytes or 65 kilobytes

or

500x10^3 bits (65 kilobytes)

But imagine right now we (the UPR) have a DS3 (45Mbit/sec) with Sprint and the average round trip is 115 ms (do ping to anywhere outside and probably you will get higher numbers).

So the computation will be:

45 Mbit/sec  115 ms 
         = 45e6  115e-3 
         = 5,175,000 bits / 8 / 1024 
         = 631 KBytes 

That means that our ideal TCP Window is 631 KBytes?.

# echo 646875 > /proc/sys/net/core/wmem_max 
# echo 646875 > /proc/sys/net/core/wmem_default 
# echo 646875 > /proc/sys/net/core/rmem_max 
# echo 646875 > /proc/sys/net/core/rmem_default 

Well, hope this works for you. Note that this is not using the Internet2 link yet. Probably will require a much smaller TCP Window. Please let me know if you find any difference on performance.

DISCLAIMER: These are teorical numbers and are not guaranty to work for everyone in the same way. For us they just work great!

For M$ Windows 9x users please refer to: * http://moat.nlanr.net/Software/TCPtune/

Download the software on this link and setup with the number you got from the previous calculation.

REFERENCES: http://dast.nlanr.net/Articles/GettingStarted/TCP_window_size.html http://ncne.nlanr.net/research/tcp/testrig/ * A very useful presentation: http://ncne.nlanr.net/training/techs/1998/980128/talks/welch

General Info:

• http://www.ncsa.uiuc.edu/People/vwelch/net_perf/tcp_windows.html http://www.psc.edu/networking/perf_tune.html (outdated 1999 but useful) http://www.ncsa.uiuc.edu/People/vwelch/net_perf_tools.htm

-- Main.WilliamCaban? - 20 Aug 2001