Documentation/networking/ixgb.txt - linux - Git at Google

 Linux* Base Driver for the Intel(R) PRO/10GbE Family of Adapters
 ================================================================

 November 17, 2004


 Contents
 ========

 - In This Release
 - Identifying Your Adapter
 - Command Line Parameters
 - Improving Performance
 - Support


 In This Release
 ===============

 This file describes the Linux* Base Driver for the Intel(R) PRO/10GbE Family
 of Adapters, version 1.0.x.

 For questions related to hardware requirements, refer to the documentation
 supplied with your Intel PRO/10GbE adapter. All hardware requirements listed
 apply to use with Linux.

 Identifying Your Adapter
 ========================

 To verify your Intel adapter is supported, find the board ID number on the
 adapter. Look for a label that has a barcode and a number in the format
 A12345-001.

 Use the above information and the Adapter & Driver ID Guide at:

   http://support.intel.com/support/network/adapter/pro100/21397.htm

 For the latest Intel network drivers for Linux, go to:

     http://downloadfinder.intel.com/scripts-df/support_intel.asp

 Command Line Parameters
 =======================

 If the driver is built as a module, the  following optional parameters are
 used by entering them on the command line with the modprobe or insmod command
 using this syntax:

      modprobe ixgb [<option>=<VAL1>,<VAL2>,...]

      insmod ixgb [<option>=<VAL1>,<VAL2>,...]

 For example, with two PRO/10GbE PCI adapters, entering:

     insmod ixgb TxDescriptors=80,128

 loads the ixgb driver with 80 TX resources for the first adapter and 128 TX
 resources for the second adapter.

 The default value for each parameter is generally the recommended setting,
 unless otherwise noted. Also, if the driver is statically built into the
 kernel, the driver is loaded with the default values for all the parameters.
 Ethtool can be used to change some of the parameters at runtime.

 FlowControl
 Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx)
 Default: Read from the EEPROM
          If EEPROM is not detected, default is 3
     This parameter controls the automatic generation(Tx) and response(Rx) to
     Ethernet PAUSE frames.

 RxDescriptors
 Valid Range: 64-512
 Default Value: 512
     This value is the number of receive descriptors allocated by the driver.
     Increasing this value allows the driver to buffer more incoming packets.
     Each descriptor is 16 bytes.  A receive buffer is also allocated for
     each descriptor and can be either 2048, 4056, 8192, or 16384 bytes,
     depending on the MTU setting. When the MTU size is 1500 or less, the
     receive buffer size is 2048 bytes. When the MTU is greater than 1500 the
     receive buffer size will be either 4056, 8192, or 16384 bytes. The
     maximum MTU size is 16114.

 RxIntDelay
 Valid Range: 0-65535 (0=off)
 Default Value: 6
     This value delays the generation of receive interrupts in units of
     0.8192 microseconds.  Receive interrupt reduction can improve CPU
     efficiency if properly tuned for specific network traffic. Increasing
     this value adds extra latency to frame reception and can end up
     decreasing the throughput of TCP traffic. If the system is reporting
     dropped receives, this value may be set too high, causing the driver to
     run out of available receive descriptors.

 TxDescriptors
 Valid Range: 64-4096
 Default Value: 256
     This value is the number of transmit descriptors allocated by the driver.
     Increasing this value allows the driver to queue more transmits. Each
     descriptor is 16 bytes.

 XsumRX
 Valid Range: 0-1
 Default Value: 1
     A value of '1' indicates that the driver should enable IP checksum
     offload for received packets (both UDP and TCP) to the adapter hardware.

 XsumTX
 Valid Range: 0-1
 Default Value: 1
     A value of '1' indicates that the driver should enable IP checksum
     offload for transmitted packets (both UDP and TCP) to the adapter
     hardware.

 Improving Performance
 =====================

 With the Intel PRO/10 GbE adapter, the default Linux configuration will very
 likely limit the total available throughput artificially.  There is a set of
 things that when applied together increase the ability of Linux to transmit
 and receive data.  The following enhancements were originally acquired from
 settings published at http://www.spec.org/web99 for various submitted results
 using Linux.

 NOTE: These changes are only suggestions, and serve as a starting point for
 tuning your network performance.

 The changes are made in three major ways, listed in order of greatest effect:
 - Use ifconfig to modify the mtu (maximum transmission unit) and the txqueuelen
   parameter.
 - Use sysctl to modify /proc parameters (essentially kernel tuning)
 - Use setpci to modify the MMRBC field in PCI-X configuration space to increase
   transmit burst lengths on the bus.

 NOTE: setpci modifies the adapter's configuration registers to allow it to read
 up to 4k bytes at a time (for transmits).  However, for some systems the
 behavior after modifying this register may be undefined (possibly errors of some
 kind). A power-cycle, hard reset or explicitly setting the e6 register back to
 22 (setpci -d 8086:1048 e6.b=22) may be required to get back to a stable
 configuration.

 - COPY these lines and paste them into ixgb_perf.sh:
 #!/bin/bash
 echo "configuring network performance , edit this file to change the interface"
 # set mmrbc to 4k reads, modify only Intel 10GbE device IDs
 setpci -d 8086:1048 e6.b=2e
 # set the MTU (max transmission unit) - it requires your switch and clients to change too!
 # set the txqueuelen
 # your ixgb adapter should be loaded as eth1 for this to work, change if needed
 ifconfig eth1 mtu 9000 txqueuelen 1000 up
 # call the sysctl utility to modify /proc/sys entries
 sysctl -p ./sysctl_ixgb.conf
 - END ixgb_perf.sh

 - COPY these lines and paste them into sysctl_ixgb.conf:
 # some of the defaults may be different for your kernel
 # call this file with sysctl -p <this file>
 # these are just suggested values that worked well to increase throughput in
 # several network benchmark tests, your mileage may vary

 ### IPV4 specific settings
 net.ipv4.tcp_timestamps = 0 # turns TCP timestamp support off, default 1, reduces CPU use
 net.ipv4.tcp_sack = 0 # turn SACK support off, default on
 # on systems with a VERY fast bus -> memory interface this is the big gainer
 net.ipv4.tcp_rmem = 10000000 10000000 10000000 # sets min/default/max TCP read buffer, default 4096 87380 174760
 net.ipv4.tcp_wmem = 10000000 10000000 10000000 # sets min/pressure/max TCP write buffer, default 4096 16384 131072
 net.ipv4.tcp_mem = 10000000 10000000 10000000 # sets min/pressure/max TCP buffer space, default 31744 32256 32768

 ### CORE settings (mostly for socket and UDP effect)
 net.core.rmem_max = 524287 # maximum receive socket buffer size, default 131071
 net.core.wmem_max = 524287 # maximum send socket buffer size, default 131071
 net.core.rmem_default = 524287 # default receive socket buffer size, default 65535
 net.core.wmem_default = 524287 # default send socket buffer size, default 65535
 net.core.optmem_max = 524287 # maximum amount of option memory buffers, default 10240
 net.core.netdev_max_backlog = 300000 # number of unprocessed input packets before kernel starts dropping them, default 300
 - END sysctl_ixgb.conf

 Edit the ixgb_perf.sh script if necessary to change eth1 to whatever interface
 your ixgb driver is using.

 NOTE: Unless these scripts are added to the boot process, these changes will
 only last only until the next system reboot.


 Resolving Slow UDP Traffic
 --------------------------

 If your server does not seem to be able to receive UDP traffic as fast as it
 can receive TCP traffic, it could be because Linux, by default, does not set
 the network stack buffers as large as they need to be to support high UDP
 transfer rates. One way to alleviate this problem is to allow more memory to
 be used by the IP stack to store incoming data.

 For instance, use the commands:
     sysctl -w net.core.rmem_max=262143
 and
     sysctl -w net.core.rmem_default=262143
 to increase the read buffer memory max and default to 262143 (256k - 1) from
 defaults of max=131071 (128k - 1) and default=65535 (64k - 1). These variables
 will increase the amount of memory used by the network stack for receives, and
 can be increased significantly more if necessary for your application.

 Support
 =======

 For general information and support, go to the Intel support website at:

     http://support.intel.com

 If an issue is identified with the released source code on the supported
 kernel with a supported adapter, email the specific information related to
 the issue to linux.nics@intel.com.
	Linux* Base Driver for the Intel(R) PRO/10GbE Family of Adapters
	================================================================

	November 17, 2004


	Contents
	========

	- In This Release
	- Identifying Your Adapter
	- Command Line Parameters
	- Improving Performance
	- Support


	In This Release
	===============

	This file describes the Linux* Base Driver for the Intel(R) PRO/10GbE Family
	of Adapters, version 1.0.x.

	For questions related to hardware requirements, refer to the documentation
	supplied with your Intel PRO/10GbE adapter. All hardware requirements listed
	apply to use with Linux.

	Identifying Your Adapter
	========================

	To verify your Intel adapter is supported, find the board ID number on the
	adapter. Look for a label that has a barcode and a number in the format
	A12345-001.

	Use the above information and the Adapter & Driver ID Guide at:

	http://support.intel.com/support/network/adapter/pro100/21397.htm

	For the latest Intel network drivers for Linux, go to:

	http://downloadfinder.intel.com/scripts-df/support_intel.asp

	Command Line Parameters
	=======================

	If the driver is built as a module, the following optional parameters are
	used by entering them on the command line with the modprobe or insmod command
	using this syntax:

	modprobe ixgb [<option>=<VAL1>,<VAL2>,...]

	insmod ixgb [<option>=<VAL1>,<VAL2>,...]

	For example, with two PRO/10GbE PCI adapters, entering:

	insmod ixgb TxDescriptors=80,128

	loads the ixgb driver with 80 TX resources for the first adapter and 128 TX
	resources for the second adapter.

	The default value for each parameter is generally the recommended setting,
	unless otherwise noted. Also, if the driver is statically built into the
	kernel, the driver is loaded with the default values for all the parameters.
	Ethtool can be used to change some of the parameters at runtime.

	FlowControl
	Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx)
	Default: Read from the EEPROM
	If EEPROM is not detected, default is 3
	This parameter controls the automatic generation(Tx) and response(Rx) to
	Ethernet PAUSE frames.

	RxDescriptors
	Valid Range: 64-512
	Default Value: 512
	This value is the number of receive descriptors allocated by the driver.
	Increasing this value allows the driver to buffer more incoming packets.
	Each descriptor is 16 bytes. A receive buffer is also allocated for
	each descriptor and can be either 2048, 4056, 8192, or 16384 bytes,
	depending on the MTU setting. When the MTU size is 1500 or less, the
	receive buffer size is 2048 bytes. When the MTU is greater than 1500 the
	receive buffer size will be either 4056, 8192, or 16384 bytes. The
	maximum MTU size is 16114.

	RxIntDelay
	Valid Range: 0-65535 (0=off)
	Default Value: 6
	This value delays the generation of receive interrupts in units of
	0.8192 microseconds. Receive interrupt reduction can improve CPU
	efficiency if properly tuned for specific network traffic. Increasing
	this value adds extra latency to frame reception and can end up
	decreasing the throughput of TCP traffic. If the system is reporting
	dropped receives, this value may be set too high, causing the driver to
	run out of available receive descriptors.

	TxDescriptors
	Valid Range: 64-4096
	Default Value: 256
	This value is the number of transmit descriptors allocated by the driver.
	Increasing this value allows the driver to queue more transmits. Each
	descriptor is 16 bytes.

	XsumRX
	Valid Range: 0-1
	Default Value: 1
	A value of '1' indicates that the driver should enable IP checksum
	offload for received packets (both UDP and TCP) to the adapter hardware.

	XsumTX
	Valid Range: 0-1
	Default Value: 1
	A value of '1' indicates that the driver should enable IP checksum
	offload for transmitted packets (both UDP and TCP) to the adapter
	hardware.

	Improving Performance
	=====================

	With the Intel PRO/10 GbE adapter, the default Linux configuration will very
	likely limit the total available throughput artificially. There is a set of
	things that when applied together increase the ability of Linux to transmit
	and receive data. The following enhancements were originally acquired from
	settings published at http://www.spec.org/web99 for various submitted results
	using Linux.

	NOTE: These changes are only suggestions, and serve as a starting point for
	tuning your network performance.

	The changes are made in three major ways, listed in order of greatest effect:
	- Use ifconfig to modify the mtu (maximum transmission unit) and the txqueuelen
	parameter.
	- Use sysctl to modify /proc parameters (essentially kernel tuning)
	- Use setpci to modify the MMRBC field in PCI-X configuration space to increase
	transmit burst lengths on the bus.

	NOTE: setpci modifies the adapter's configuration registers to allow it to read
	up to 4k bytes at a time (for transmits). However, for some systems the
	behavior after modifying this register may be undefined (possibly errors of some
	kind). A power-cycle, hard reset or explicitly setting the e6 register back to
	22 (setpci -d 8086:1048 e6.b=22) may be required to get back to a stable
	configuration.

	- COPY these lines and paste them into ixgb_perf.sh:
	#!/bin/bash
	echo "configuring network performance , edit this file to change the interface"
	# set mmrbc to 4k reads, modify only Intel 10GbE device IDs
	setpci -d 8086:1048 e6.b=2e
	# set the MTU (max transmission unit) - it requires your switch and clients to change too!
	# set the txqueuelen
	# your ixgb adapter should be loaded as eth1 for this to work, change if needed
	ifconfig eth1 mtu 9000 txqueuelen 1000 up
	# call the sysctl utility to modify /proc/sys entries
	sysctl -p ./sysctl_ixgb.conf
	- END ixgb_perf.sh

	- COPY these lines and paste them into sysctl_ixgb.conf:
	# some of the defaults may be different for your kernel
	# call this file with sysctl -p <this file>
	# these are just suggested values that worked well to increase throughput in
	# several network benchmark tests, your mileage may vary

	### IPV4 specific settings
	net.ipv4.tcp_timestamps = 0 # turns TCP timestamp support off, default 1, reduces CPU use
	net.ipv4.tcp_sack = 0 # turn SACK support off, default on
	# on systems with a VERY fast bus -> memory interface this is the big gainer
	net.ipv4.tcp_rmem = 10000000 10000000 10000000 # sets min/default/max TCP read buffer, default 4096 87380 174760
	net.ipv4.tcp_wmem = 10000000 10000000 10000000 # sets min/pressure/max TCP write buffer, default 4096 16384 131072
	net.ipv4.tcp_mem = 10000000 10000000 10000000 # sets min/pressure/max TCP buffer space, default 31744 32256 32768

	### CORE settings (mostly for socket and UDP effect)
	net.core.rmem_max = 524287 # maximum receive socket buffer size, default 131071
	net.core.wmem_max = 524287 # maximum send socket buffer size, default 131071
	net.core.rmem_default = 524287 # default receive socket buffer size, default 65535
	net.core.wmem_default = 524287 # default send socket buffer size, default 65535
	net.core.optmem_max = 524287 # maximum amount of option memory buffers, default 10240
	net.core.netdev_max_backlog = 300000 # number of unprocessed input packets before kernel starts dropping them, default 300
	- END sysctl_ixgb.conf

	Edit the ixgb_perf.sh script if necessary to change eth1 to whatever interface
	your ixgb driver is using.

	NOTE: Unless these scripts are added to the boot process, these changes will
	only last only until the next system reboot.


	Resolving Slow UDP Traffic
	--------------------------

	If your server does not seem to be able to receive UDP traffic as fast as it
	can receive TCP traffic, it could be because Linux, by default, does not set
	the network stack buffers as large as they need to be to support high UDP
	transfer rates. One way to alleviate this problem is to allow more memory to
	be used by the IP stack to store incoming data.

	For instance, use the commands:
	sysctl -w net.core.rmem_max=262143
	and
	sysctl -w net.core.rmem_default=262143
	to increase the read buffer memory max and default to 262143 (256k - 1) from
	defaults of max=131071 (128k - 1) and default=65535 (64k - 1). These variables
	will increase the amount of memory used by the network stack for receives, and
	can be increased significantly more if necessary for your application.

	Support
	=======

	For general information and support, go to the Intel support website at:

	http://support.intel.com

	If an issue is identified with the released source code on the supported
	kernel with a supported adapter, email the specific information related to
	the issue to linux.nics@intel.com.