【转】Linux调整TCP/IP相关参数,优化服务器网络吞吐量和性能

作者: wiflish 分类: 技术 发布时间: 2014-05-12 14:30 ė 6没有评论

现象:

多台机器通过同一个网关(比如同一个公司内的不同机器访问服务器)访问服务器时,会出现某一台机器可以访问,另外一台机器连不上服务器的情况。

原因是调整了Linux的TCP/IP相关参数导致:

net.ipv4.tcp_tw_recycle = 1; #表示开启TCP连接中TIME-WAIT sockets的快速回收,0表示关闭。

net.ipv4.tcp_timestamps =1;    #表示开启时间戳,服务端就会缓存请求的时间戳。

 

解决办法:

net.ipv4.tcp_tw_recycle = 0

或者

net.ipv4.tcp_timestamps = 0

 

结论:

当tcp_tw_recycle和tcp_timestamps同时打开时会激活TCP的一种隐藏属性:缓存连接的时间戳。60秒内,同一源IP的后续请求的时间戳小于缓存中的时间戳,内核就会丢弃该请求。

转自:http://www.speedguide.net/articles/linux-tweaking-121

 

Linux Tweaking

Raising netowrk limits for broadband under Linux
2003.04.01 16:03 by Philip
Tags: TCP WindowWANTCP/IPtweakLinux

The TCP/IP parameters for tweaking a Linux-based machine for fast internet connections are located in /proc/sys/net/… (assuming 2.1+ kernel). This location is volatile, and changes are reset at reboot. There are a couple of methods for reapplying the changes at boot time, ilustrated below.

Locating the TCP/IP related parameters

All TCP/IP tunning parameters are located under /proc/sys/net/…  For example, here is a list of the most important tunning parameters, along with short description of their meaning:

/proc/sys/net/core/rmem_max - Maximum TCP Receive Window
/proc/sys/net/core/wmem_max – Maximum TCP Send Window
/proc/sys/net/ipv4/tcp_rmem – memory reserved for TCP receive buffers
/proc/sys/net/ipv4/tcp_wmem – memory reserved for TCP send buffers
/proc/sys/net/ipv4/tcp_timestamps - timestamps (RFC 1323) add 12 bytes to the TCP header
/proc/sys/net/ipv4/tcp_sack – tcp selective acknowledgements.
/proc/sys/net/ipv4/tcp_window_scaling – support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535.

Keep in mind everything under /proc is volatile, so any changes you make are lost after reboot.

There are some additional internal memory buffers for the TCP Window, allocated for each connection:
/proc/sys/net/ipv4/tcp_rmem – memory reserved for TCP rcv buffers (reserved memory per connection default)
/proc/sys/net/ipv4/tcp_wmem  – memory reserved for TCP snd buffers (reserved memory per connection default)

The tcp_rmem and tcp_wmem contain arrays of three parameter values: the 3 numbers represent minimum, default and maximum memory values. Those 3 values are used to bound autotunning and balance memory usage while under global memory stress.

Applying TCP/IP Parameters at System Boot

You can edit /etc/rc.local, or /etc/boot.local depending on your distribution so the parameters get automatically reapplied at boot time. The TCP/IP parameters should be self-explanatory: we’re basically setting the TCP Window to 256960, disabling timestamps (to avoid 12 byte header overhead), enabling tcp window scaling, and selective acknowledgements:

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max




echo 0 > /proc/sys/net/ipv4/tcp_timestamps 
echo 1 > /proc/sys/net/ipv4/tcp_sack 
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

Change the values above as desired, depending on your internet connection and maximum bandwidth/latency. There are other parameters you can change from the default if you’re confident in what you’re doing – just find the correct syntax of the values in /proc/sys/net/… and add a line in the above code analogous to the others. To revert to the default parameters, you can just comment or delete the above code from /etc/rc.local and restart.

Another method to reapply the values upon boot is to include the following in your /etc/sysctl.conf (adjust RWIN values as needed):

net.core.rmem_default = 256960
net.core.rmem_max = 256960 
net.core.wmem_default = 256960
net.core.wmem_max = 256960 

net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_window_scaling = 1

Notes:
Execute sysctl -p to make these new settings take effect.
To manually set the MTU value under Linux, use the command: 
ifconfig eth0 mtu 1500   (where 1500 is the desired MTU size)

 

Changing Current Values without rebooting 

The current TCP/IP parameters can be edited without the need for reboot in the following locations:

/proc/sys/net/core/
rmem_default = Default Receive Window
rmem_max = Maximum Receive Window
wmem_default = Default Send Window
wmem_max = Maximum Send Window

/proc/sys/net/ipv4/
You’ll find timestamps, window scalling, selective acknowledgements, etc.

Keep in mind the values in /proc will be reset upon reboot. You still need to add the code in /etc/rc.local or /etc/boot.local in order to have the changes applied at boot time as described above.

 

Other TCP Parameters to consider

TCP_FIN_TIMEOUT
This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During this TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, making more resources available for new connections. Addjust this in the presense of many connections sitting in the TIME_WAIT state:

(default: 60 seconds, recommended 15-30 seconds)

Notes:
You can use any of the earlier described methods to reapply these settings at boot time.
Here is a quick way to view the number of connections and their states:

netstat -tan | grep ‘:80 ‘ | awk ‘{print $6}’ | sort | uniq -c

 

TCP_KEEPALIVE_INTERVAL
This determines the wait time between isAlive interval probes. To set:

echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
(default: 75 seconds, recommended: 15-30 seconds)

 

TCP_KEEPALIVE_PROBES
This determines the number of probes before timing out. To set:

echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
(default: 9, recommended 5)

 

TCP_TW_RECYCLE
It enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). The sysctl documentation incorrectly states the default as enabled. It can be changed to 1 (enabled) in many cases. Known to cause some issues with hoststated (load balancing and fail over) if enabled, should be used with caution.

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
(boolean, default: 0)
  

TCP_TW_REUSE
This allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
(boolean, default: 0)

Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.

Another option for applying the above kernel parameters is to enter them in /etc/sysctl.conf in the format:
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1

 

Linux Netfilter Tweaks

Try this for a list netfilter parameters:  sysctl -a | grep netfilter

We can add the following commands to the /etc/sysctl.conf file to tune individual parameters, as follows.
To reduce the number of connections in TIME_WAIT state, we can decrease the number of seconds connections are kept in this state before being dropped:

# reduce TIME_WAIT from the 120s default
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

You can commit the sysctl.conf changes without rebooting (and test for possible syntax errors) by executing: sysctl -p
To check sysctl parameters, use: sysctl -a

Misc Notes: You may want to reduce net.netfilter.nf_conntrack_tcp_timeout_established to 900 or some manageable number as well.
To check the actual number of current connections in the TIME_WAIT state, for example, try: netstat -n | grep TIME_WAIT | wc -l

 

Kernel Recompile Option

There is another method one can use to set TCP/IP parameters, involving kernel recompile… If you’re brave enough. Look for the parameters in the following files:
/LINUX-SOURCE-DIR/include/linux/skbuff.h  (Look for SK_WMEM_MAX & SK_RMEM_MAX)
/LINUX-SOURCE-DIR/include/net/tcp.h (Look for MAX_WINDOW & MIN_WINDOW)

References

See also the complete ip-sysctl parameters reference -here-
TCP Variables

本文出自 Wiflish's Blog,转载时请注明出处及相应链接。

本文永久链接: http://wiflish.fyard.com/archives/111

发表评论

电子邮件地址不会被公开。 必填项已用*标注

您可以使用这些HTML标签和属性: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Ɣ回顶部