Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 1 | Introduction |
| 2 | ============ |
| 3 | |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 4 | ATA over Ethernet is a network protocol that provides simple access to |
| 5 | block storage on the LAN. |
| 6 | |
| 7 | http://support.coraid.com/documents/AoEr11.txt |
| 8 | |
Ed Cashin | eecdf22 | 2012-10-04 17:16:39 -0700 | [diff] [blame] | 9 | The EtherDrive (R) HOWTO for 2.6 and 3.x kernels is found at ... |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 10 | |
Ed Cashin | eecdf22 | 2012-10-04 17:16:39 -0700 | [diff] [blame] | 11 | http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO.html |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 12 | |
Ed Cashin | eecdf22 | 2012-10-04 17:16:39 -0700 | [diff] [blame] | 13 | It has many tips and hints! Please see, especially, recommended |
| 14 | tunings for virtual memory: |
| 15 | |
| 16 | http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO-5.html#ss5.19 |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 17 | |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 18 | The aoetools are userland programs that are designed to work with this |
| 19 | driver. The aoetools are on sourceforge. |
| 20 | |
| 21 | http://aoetools.sourceforge.net/ |
| 22 | |
Mauro Carvalho Chehab | 83bbf6e | 2019-06-27 12:59:40 -0300 | [diff] [blame] | 23 | The scripts in this Documentation/admin-guide/aoe directory are intended to |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 24 | document the use of the driver and are not necessary if you install |
| 25 | the aoetools. |
| 26 | |
| 27 | |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 28 | Creating Device Nodes |
| 29 | ===================== |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 30 | |
| 31 | Users of udev should find the block device nodes created |
| 32 | automatically, but to create all the necessary device nodes, use the |
| 33 | udev configuration rules provided in udev.txt (in this directory). |
| 34 | |
| 35 | There is a udev-install.sh script that shows how to install these |
| 36 | rules on your system. |
| 37 | |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 38 | There is also an autoload script that shows how to edit |
Lucas De Marchi | 970e248 | 2012-03-30 13:37:16 -0700 | [diff] [blame] | 39 | /etc/modprobe.d/aoe.conf to ensure that the aoe module is loaded when |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 40 | necessary. Preloading the aoe module is preferable to autoloading, |
| 41 | however, because AoE discovery takes a few seconds. It can be |
| 42 | confusing when an AoE device is not present the first time the a |
| 43 | command is run but appears a second later. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 44 | |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 45 | Using Device Nodes |
| 46 | ================== |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 47 | |
| 48 | "cat /dev/etherd/err" blocks, waiting for error diagnostic output, |
| 49 | like any retransmitted packets. |
| 50 | |
| 51 | "echo eth2 eth4 > /dev/etherd/interfaces" tells the aoe driver to |
| 52 | limit ATA over Ethernet traffic to eth2 and eth4. AoE traffic from |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 53 | untrusted networks should be ignored as a matter of security. See |
| 54 | also the aoe_iflist driver option described below. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 55 | |
| 56 | "echo > /dev/etherd/discover" tells the driver to find out what AoE |
| 57 | devices are available. |
| 58 | |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 59 | In the future these character devices may disappear and be replaced |
| 60 | by sysfs counterparts. Using the commands in aoetools insulates |
| 61 | users from these implementation details. |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 62 | |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 63 | The block devices are named like this:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 64 | |
| 65 | e{shelf}.{slot} |
| 66 | e{shelf}.{slot}p{part} |
| 67 | |
| 68 | ... so that "e0.2" is the third blade from the left (slot 2) in the |
| 69 | first shelf (shelf address zero). That's the whole disk. The first |
| 70 | partition on that disk would be "e0.2p1". |
| 71 | |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 72 | Using sysfs |
| 73 | =========== |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 74 | |
| 75 | Each aoe block device in /sys/block has the extra attributes of |
| 76 | state, mac, and netif. The state attribute is "up" when the device |
| 77 | is ready for I/O and "down" if detected but unusable. The |
| 78 | "down,closewait" state shows that the device is still open and |
| 79 | cannot come up again until it has been closed. |
| 80 | |
| 81 | The mac attribute is the ethernet address of the remote AoE device. |
| 82 | The netif attribute is the network interface on the localhost |
| 83 | through which we are communicating with the remote AoE device. |
| 84 | |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 85 | There is a script in this directory that formats this information in |
| 86 | a convenient way. Users with aoetools should use the aoe-stat |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 87 | command:: |
Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 88 | |
Mauro Carvalho Chehab | 83bbf6e | 2019-06-27 12:59:40 -0300 | [diff] [blame] | 89 | root@makki root# sh Documentation/admin-guide/aoe/status.sh |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 90 | e10.0 eth3 up |
| 91 | e10.1 eth3 up |
| 92 | e10.2 eth3 up |
| 93 | e10.3 eth3 up |
| 94 | e10.4 eth3 up |
| 95 | e10.5 eth3 up |
| 96 | e10.6 eth3 up |
| 97 | e10.7 eth3 up |
| 98 | e10.8 eth3 up |
| 99 | e10.9 eth3 up |
| 100 | e4.0 eth1 up |
| 101 | e4.1 eth1 up |
| 102 | e4.2 eth1 up |
| 103 | e4.3 eth1 up |
| 104 | e4.4 eth1 up |
| 105 | e4.5 eth1 up |
| 106 | e4.6 eth1 up |
| 107 | e4.7 eth1 up |
| 108 | e4.8 eth1 up |
| 109 | e4.9 eth1 up |
Ed L Cashin | 03c41c4 | 2005-04-29 10:24:03 -0400 | [diff] [blame] | 110 | |
| 111 | Use /sys/module/aoe/parameters/aoe_iflist (or better, the driver |
| 112 | option discussed below) instead of /dev/etherd/interfaces to limit |
| 113 | AoE traffic to the network interfaces in the given |
| 114 | whitespace-separated list. Unlike the old character device, the |
| 115 | sysfs entry can be read from as well as written to. |
| 116 | |
| 117 | It's helpful to trigger discovery after setting the list of allowed |
Ed L Cashin | 67d9f84 | 2005-04-29 10:24:18 -0400 | [diff] [blame] | 118 | interfaces. The aoetools package provides an aoe-discover script |
| 119 | for this purpose. You can also directly use the |
| 120 | /dev/etherd/discover special file described above. |
Ed L Cashin | 03c41c4 | 2005-04-29 10:24:03 -0400 | [diff] [blame] | 121 | |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 122 | Driver Options |
| 123 | ============== |
Ed L Cashin | 03c41c4 | 2005-04-29 10:24:03 -0400 | [diff] [blame] | 124 | |
| 125 | There is a boot option for the built-in aoe driver and a |
| 126 | corresponding module parameter, aoe_iflist. Without this option, |
| 127 | all network interfaces may be used for ATA over Ethernet. Here is a |
Mauro Carvalho Chehab | 305a99eb | 2019-06-12 14:52:37 -0300 | [diff] [blame] | 128 | usage example for the module parameter:: |
Ed L Cashin | 03c41c4 | 2005-04-29 10:24:03 -0400 | [diff] [blame] | 129 | |
| 130 | modprobe aoe_iflist="eth1 eth3" |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 131 | |
| 132 | The aoe_deadsecs module parameter determines the maximum number of |
| 133 | seconds that the driver will wait for an AoE device to provide a |
| 134 | response to an AoE command. After aoe_deadsecs seconds have |
Ed Cashin | c450ba0 | 2012-12-17 16:04:14 -0800 | [diff] [blame] | 135 | elapsed, the AoE device will be marked as "down". A value of zero |
| 136 | is supported for testing purposes and makes the aoe driver keep |
| 137 | trying AoE commands forever. |
Ed Cashin | 5b8c4be | 2012-10-04 17:16:49 -0700 | [diff] [blame] | 138 | |
| 139 | The aoe_maxout module parameter has a default of 128. This is the |
| 140 | maximum number of unresponded packets that will be sent to an AoE |
| 141 | target at one time. |
| 142 | |
| 143 | The aoe_dyndevs module parameter defaults to 1, meaning that the |
| 144 | driver will assign a block device minor number to a discovered AoE |
| 145 | target based on the order of its discovery. With dynamic minor |
| 146 | device numbers in use, a greater range of AoE shelf and slot |
| 147 | addresses can be supported. Users with udev will never have to |
| 148 | think about minor numbers. Using aoe_dyndevs=0 allows device nodes |
| 149 | to be pre-created using a static minor-number scheme with the |
| 150 | aoe-mkshelf script in the aoetools. |