blob: adc314639085b273d9e99dd3fab97f387e4bffaf [file] [log] [blame]
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001.. SPDX-License-Identifier: GPL-2.0
Linus Torvalds1da177e2005-04-16 15:20:36 -07002
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02003===================================
4Linux Ethernet Bonding Driver HOWTO
5===================================
Jay Vosburgh00354cf2005-07-21 12:18:02 -07006
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02007Latest update: 27 April 2011
Linus Torvalds1da177e2005-04-16 15:20:36 -07008
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02009Initial release: Thomas Davis <tadavis at lbl.gov>
10
11Corrections, HA extensions: 2000/10/03-15:
12
Linus Torvalds1da177e2005-04-16 15:20:36 -070013 - Willy Tarreau <willy at meta-x.org>
14 - Constantine Gavrilov <const-g at xpert.com>
15 - Chad N. Tindel <ctindel at ieee dot org>
16 - Janice Girouard <girouard at us dot ibm dot com>
17 - Jay Vosburgh <fubar at us dot ibm dot com>
18
19Reorganized and updated Feb 2005 by Jay Vosburgh
Auke Kok6224e012006-06-08 11:15:35 -070020Added Sysfs information: 2006/04/24
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020021
Auke Kok6224e012006-06-08 11:15:35 -070022 - Mitch Williams <mitch.a.williams at intel.com>
Linus Torvalds1da177e2005-04-16 15:20:36 -070023
Jay Vosburgh00354cf2005-07-21 12:18:02 -070024Introduction
25============
Linus Torvalds1da177e2005-04-16 15:20:36 -070026
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020027The Linux bonding driver provides a method for aggregating
Jay Vosburgh00354cf2005-07-21 12:18:02 -070028multiple network interfaces into a single logical "bonded" interface.
29The behavior of the bonded interfaces depends upon the mode; generally
30speaking, modes provide either hot standby or load balancing services.
31Additionally, link integrity monitoring may be performed.
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020032
33The bonding driver originally came from Donald Becker's
Jay Vosburgh00354cf2005-07-21 12:18:02 -070034beowulf patches for kernel 2.0. It has changed quite a bit since, and
35the original tools from extreme-linux and beowulf sites will not work
36with this version of the driver.
37
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020038For new versions of the driver, updated userspace tools, and
Jay Vosburgh00354cf2005-07-21 12:18:02 -070039who to ask for help, please follow the links at the end of this file.
Linus Torvalds1da177e2005-04-16 15:20:36 -070040
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020041.. Table of Contents
Linus Torvalds1da177e2005-04-16 15:20:36 -070042
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020043 1. Bonding Driver Installation
Linus Torvalds1da177e2005-04-16 15:20:36 -070044
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020045 2. Bonding Driver Options
Linus Torvalds1da177e2005-04-16 15:20:36 -070046
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020047 3. Configuring Bonding Devices
48 3.1 Configuration with Sysconfig Support
49 3.1.1 Using DHCP with Sysconfig
50 3.1.2 Configuring Multiple Bonds with Sysconfig
51 3.2 Configuration with Initscripts Support
52 3.2.1 Using DHCP with Initscripts
53 3.2.2 Configuring Multiple Bonds with Initscripts
54 3.3 Configuring Bonding Manually with Ifenslave
55 3.3.1 Configuring Multiple Bonds Manually
56 3.4 Configuring Bonding Manually via Sysfs
57 3.5 Configuration with Interfaces Support
58 3.6 Overriding Configuration for Special Cases
59 3.7 Configuring LACP for 802.3ad mode in a more secure way
Linus Torvalds1da177e2005-04-16 15:20:36 -070060
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020061 4. Querying Bonding Configuration
62 4.1 Bonding Configuration
63 4.2 Network Configuration
Linus Torvalds1da177e2005-04-16 15:20:36 -070064
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020065 5. Switch Configuration
Linus Torvalds1da177e2005-04-16 15:20:36 -070066
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020067 6. 802.1q VLAN Support
Linus Torvalds1da177e2005-04-16 15:20:36 -070068
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020069 7. Link Monitoring
70 7.1 ARP Monitor Operation
71 7.2 Configuring Multiple ARP Targets
72 7.3 MII Monitor Operation
Linus Torvalds1da177e2005-04-16 15:20:36 -070073
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020074 8. Potential Trouble Sources
75 8.1 Adventures in Routing
76 8.2 Ethernet Device Renaming
77 8.3 Painfully Slow Or No Failed Link Detection By Miimon
Linus Torvalds1da177e2005-04-16 15:20:36 -070078
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020079 9. SNMP agents
Linus Torvalds1da177e2005-04-16 15:20:36 -070080
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020081 10. Promiscuous mode
Linus Torvalds1da177e2005-04-16 15:20:36 -070082
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020083 11. Configuring Bonding for High Availability
84 11.1 High Availability in a Single Switch Topology
85 11.2 High Availability in a Multiple Switch Topology
86 11.2.1 HA Bonding Mode Selection for Multiple Switch Topology
87 11.2.2 HA Link Monitoring for Multiple Switch Topology
Linus Torvalds1da177e2005-04-16 15:20:36 -070088
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020089 12. Configuring Bonding for Maximum Throughput
90 12.1 Maximum Throughput in a Single Switch Topology
91 12.1.1 MT Bonding Mode Selection for Single Switch Topology
92 12.1.2 MT Link Monitoring for Single Switch Topology
93 12.2 Maximum Throughput in a Multiple Switch Topology
94 12.2.1 MT Bonding Mode Selection for Multiple Switch Topology
95 12.2.2 MT Link Monitoring for Multiple Switch Topology
Linus Torvalds1da177e2005-04-16 15:20:36 -070096
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020097 13. Switch Behavior Issues
98 13.1 Link Establishment and Failover Delays
99 13.2 Duplicated Incoming Packets
Linus Torvalds1da177e2005-04-16 15:20:36 -0700100
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200101 14. Hardware Specific Considerations
102 14.1 IBM BladeCenter
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700103
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200104 15. Frequently Asked Questions
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700105
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200106 16. Resources and Links
Linus Torvalds1da177e2005-04-16 15:20:36 -0700107
108
1091. Bonding Driver Installation
110==============================
111
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200112Most popular distro kernels ship with the bonding driver
Cong Wangb1098bb2013-05-27 15:49:16 +0000113already available as a module. If your distro does not, or you
Linus Torvalds1da177e2005-04-16 15:20:36 -0700114have need to compile bonding from source (e.g., configuring and
115installing a mainline kernel from kernel.org), you'll need to perform
116the following steps:
117
1181.1 Configure and build the kernel with bonding
119-----------------------------------------------
120
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200121The current version of the bonding driver is available in the
Linus Torvalds1da177e2005-04-16 15:20:36 -0700122drivers/net/bonding subdirectory of the most recent kernel source
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700123(which is available on http://kernel.org). Most users "rolling their
124own" will want to use the most recent kernel from kernel.org.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700125
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200126Configure kernel with "make menuconfig" (or "make xconfig" or
Linus Torvalds1da177e2005-04-16 15:20:36 -0700127"make config"), then select "Bonding driver support" in the "Network
128device support" section. It is recommended that you configure the
129driver as module since it is currently the only way to pass parameters
130to the driver or configure more than one bonding device.
131
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200132Build and install the new kernel and modules.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700133
Cong Wangb1098bb2013-05-27 15:49:16 +00001341.2 Bonding Control Utility
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200135---------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -0700136
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200137It is recommended to configure bonding via iproute2 (netlink)
Cong Wangb1098bb2013-05-27 15:49:16 +0000138or sysfs, the old ifenslave control utility is obsolete.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700139
1402. Bonding Driver Options
141=========================
142
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200143Options for the bonding driver are supplied as parameters to the
Jay Vosburgh9a6c6862007-11-13 20:25:48 -0800144bonding module at load time, or are specified via sysfs.
145
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200146Module options may be given as command line arguments to the
Jay Vosburgh9a6c6862007-11-13 20:25:48 -0800147insmod or modprobe command, but are usually specified in either the
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200148``/etc/modprobe.d/*.conf`` configuration files, or in a distro-specific
Lucas De Marchi970e2482012-03-30 13:37:16 -0700149configuration file (some of which are detailed in the next section).
Jay Vosburgh9a6c6862007-11-13 20:25:48 -0800150
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200151Details on bonding support for sysfs is provided in the
Jay Vosburgh9a6c6862007-11-13 20:25:48 -0800152"Configuring Bonding Manually via Sysfs" section, below.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700153
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200154The available bonding driver parameters are listed below. If a
Linus Torvalds1da177e2005-04-16 15:20:36 -0700155parameter is not specified the default value is used. When initially
156configuring a bond, it is recommended "tail -f /var/log/messages" be
157run in a separate window to watch for bonding driver error messages.
158
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200159It is critical that either the miimon or arp_interval and
Linus Torvalds1da177e2005-04-16 15:20:36 -0700160arp_ip_target parameters be specified, otherwise serious network
161degradation will occur during link failures. Very few devices do not
162support at least miimon, so there is really no reason not to use it.
163
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200164Options with textual values will accept either the text name
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700165or, for backwards compatibility, the option value. E.g.,
166"mode=802.3ad" and "mode=4" set the same mode.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700167
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200168The parameters are as follows:
Linus Torvalds1da177e2005-04-16 15:20:36 -0700169
Nicolas de Pesloüan1ba9ac72011-12-26 13:35:24 +0000170active_slave
171
172 Specifies the new active slave for modes that support it
173 (active-backup, balance-alb and balance-tlb). Possible values
174 are the name of any currently enslaved interface, or an empty
175 string. If a name is given, the slave and its link must be up in order
176 to be selected as the new active slave. If an empty string is
177 specified, the current active slave is cleared, and a new active
178 slave is selected automatically.
179
180 Note that this is only available through the sysfs interface. No module
181 parameter by this name exists.
182
183 The normal value of this option is the name of the currently
184 active slave, or the empty string if there is no active slave or
185 the current mode does not use an active slave.
186
Mahesh Bandewar6791e462015-05-09 00:01:55 -0700187ad_actor_sys_prio
188
189 In an AD system, this specifies the system priority. The allowed range
190 is 1 - 65535. If the value is not specified, it takes 65535 as the
191 default value.
192
193 This parameter has effect only in 802.3ad mode and is available through
194 SysFs interface.
195
Mahesh Bandewar74514952015-05-09 00:01:56 -0700196ad_actor_system
197
198 In an AD system, this specifies the mac-address for the actor in
199 protocol packet exchanges (LACPDUs). The value cannot be NULL or
200 multicast. It is preferred to have the local-admin bit set for this
201 mac but driver does not enforce it. If the value is not given then
202 system defaults to using the masters' mac address as actors' system
203 address.
204
205 This parameter has effect only in 802.3ad mode and is available through
206 SysFs interface.
207
Jay Vosburghfd989c82008-11-04 17:51:16 -0800208ad_select
209
210 Specifies the 802.3ad aggregation selection logic to use. The
211 possible values and their effects are:
212
213 stable or 0
214
215 The active aggregator is chosen by largest aggregate
216 bandwidth.
217
218 Reselection of the active aggregator occurs only when all
219 slaves of the active aggregator are down or the active
220 aggregator has no slaves.
221
222 This is the default value.
223
224 bandwidth or 1
225
226 The active aggregator is chosen by largest aggregate
227 bandwidth. Reselection occurs if:
228
229 - A slave is added to or removed from the bond
230
231 - Any slave's link state changes
232
233 - Any slave's 802.3ad association state changes
234
Matt LaPlante19f59462009-04-27 15:06:31 +0200235 - The bond's administrative state changes to up
Jay Vosburghfd989c82008-11-04 17:51:16 -0800236
237 count or 2
238
239 The active aggregator is chosen by the largest number of
240 ports (slaves). Reselection occurs as described under the
241 "bandwidth" setting, above.
242
243 The bandwidth and count selection policies permit failover of
244 802.3ad aggregations when partial failure of the active aggregator
245 occurs. This keeps the aggregator with the highest availability
246 (either in bandwidth or in number of ports) active at all times.
247
248 This option was added in bonding version 3.4.0.
249
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -0700250ad_user_port_key
251
252 In an AD system, the port-key has three parts as shown below -
253
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200254 ===== ============
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -0700255 Bits Use
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200256 ===== ============
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -0700257 00 Duplex
258 01-05 Speed
259 06-15 User-defined
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200260 ===== ============
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -0700261
262 This defines the upper 10 bits of the port key. The values can be
263 from 0 - 1023. If not given, the system defaults to 0.
264
265 This parameter has effect only in 802.3ad mode and is available through
266 SysFs interface.
267
Nicolas de Pesloüan025890b2011-08-06 07:06:39 +0000268all_slaves_active
269
270 Specifies that duplicate frames (received on inactive ports) should be
271 dropped (0) or delivered (1).
272
273 Normally, bonding will drop duplicate frames (received on inactive
274 ports), which is desirable for most users. But there are some times
275 it is nice to allow duplicate frames to be delivered.
276
277 The default value is 0 (drop duplicate frames received on inactive
278 ports).
279
Linus Torvalds1da177e2005-04-16 15:20:36 -0700280arp_interval
281
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700282 Specifies the ARP link monitoring frequency in milliseconds.
Jay Vosburghf5b2b962006-09-22 21:54:53 -0700283
284 The ARP monitor works by periodically checking the slave
285 devices to determine whether they have sent or received
286 traffic recently (the precise criteria depends upon the
287 bonding mode, and the state of the slave). Regular traffic is
288 generated via ARP probes issued for the addresses specified by
289 the arp_ip_target option.
290
291 This behavior can be modified by the arp_validate option,
292 below.
293
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700294 If ARP monitoring is used in an etherchannel compatible mode
295 (modes 0 and 2), the switch should be configured in a mode
296 that evenly distributes packets across all links. If the
297 switch is configured to distribute the packets in an XOR
Linus Torvalds1da177e2005-04-16 15:20:36 -0700298 fashion, all replies from the ARP targets will be received on
299 the same link which could cause the other team members to
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700300 fail. ARP monitoring should not be used in conjunction with
301 miimon. A value of 0 disables ARP monitoring. The default
Linus Torvalds1da177e2005-04-16 15:20:36 -0700302 value is 0.
303
304arp_ip_target
305
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700306 Specifies the IP addresses to use as ARP monitoring peers when
307 arp_interval is > 0. These are the targets of the ARP request
308 sent to determine the health of the link to the targets.
309 Specify these values in ddd.ddd.ddd.ddd format. Multiple IP
310 addresses must be separated by a comma. At least one IP
311 address must be given for ARP monitoring to function. The
312 maximum number of targets that can be specified is 16. The
313 default value is no IP addresses.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700314
Jay Vosburghf5b2b962006-09-22 21:54:53 -0700315arp_validate
316
317 Specifies whether or not ARP probes and replies should be
Veaceslav Falico52f65ef2014-02-18 07:48:41 +0100318 validated in any mode that supports arp monitoring, or whether
319 non-ARP traffic should be filtered (disregarded) for link
320 monitoring purposes.
Jay Vosburghf5b2b962006-09-22 21:54:53 -0700321
322 Possible values are:
323
324 none or 0
325
Veaceslav Falico52f65ef2014-02-18 07:48:41 +0100326 No validation or filtering is performed.
Jay Vosburghf5b2b962006-09-22 21:54:53 -0700327
328 active or 1
329
330 Validation is performed only for the active slave.
331
332 backup or 2
333
334 Validation is performed only for backup slaves.
335
336 all or 3
337
338 Validation is performed for all slaves.
339
Veaceslav Falico52f65ef2014-02-18 07:48:41 +0100340 filter or 4
Jay Vosburghf5b2b962006-09-22 21:54:53 -0700341
Veaceslav Falico52f65ef2014-02-18 07:48:41 +0100342 Filtering is applied to all slaves. No validation is
343 performed.
Veaceslav Falicod7d35c62013-06-24 11:49:33 +0200344
Veaceslav Falico52f65ef2014-02-18 07:48:41 +0100345 filter_active or 5
346
347 Filtering is applied to all slaves, validation is performed
348 only for the active slave.
349
350 filter_backup or 6
351
352 Filtering is applied to all slaves, validation is performed
353 only for backup slaves.
354
355 Validation:
356
357 Enabling validation causes the ARP monitor to examine the incoming
358 ARP requests and replies, and only consider a slave to be up if it
359 is receiving the appropriate ARP traffic.
360
361 For an active slave, the validation checks ARP replies to confirm
362 that they were generated by an arp_ip_target. Since backup slaves
363 do not typically receive these replies, the validation performed
364 for backup slaves is on the broadcast ARP request sent out via the
365 active slave. It is possible that some switch or network
366 configurations may result in situations wherein the backup slaves
367 do not receive the ARP requests; in such a situation, validation
368 of backup slaves must be disabled.
369
370 The validation of ARP requests on backup slaves is mainly helping
371 bonding to decide which slaves are more likely to work in case of
372 the active slave failure, it doesn't really guarantee that the
373 backup slave will work if it's selected as the next active slave.
374
375 Validation is useful in network configurations in which multiple
376 bonding hosts are concurrently issuing ARPs to one or more targets
377 beyond a common switch. Should the link between the switch and
378 target fail (but not the switch itself), the probe traffic
379 generated by the multiple bonding instances will fool the standard
380 ARP monitor into considering the links as still up. Use of
381 validation can resolve this, as the ARP monitor will only consider
382 ARP requests and replies associated with its own instance of
383 bonding.
384
385 Filtering:
386
387 Enabling filtering causes the ARP monitor to only use incoming ARP
388 packets for link availability purposes. Arriving packets that are
389 not ARPs are delivered normally, but do not count when determining
390 if a slave is available.
391
392 Filtering operates by only considering the reception of ARP
393 packets (any ARP packet, regardless of source or destination) when
394 determining if a slave has received traffic for link availability
395 purposes.
396
397 Filtering is useful in network configurations in which significant
398 levels of third party broadcast traffic would fool the standard
399 ARP monitor into considering the links as still up. Use of
400 filtering can resolve this, as only ARP traffic is considered for
401 link availability purposes.
Jay Vosburghf5b2b962006-09-22 21:54:53 -0700402
403 This option was added in bonding version 3.1.0.
404
Veaceslav Falico8599b522013-06-24 11:49:34 +0200405arp_all_targets
406
407 Specifies the quantity of arp_ip_targets that must be reachable
408 in order for the ARP monitor to consider a slave as being up.
409 This option affects only active-backup mode for slaves with
410 arp_validation enabled.
411
412 Possible values are:
413
414 any or 0
415
416 consider the slave up only when any of the arp_ip_targets
417 is reachable
418
419 all or 1
420
421 consider the slave up only when all of the arp_ip_targets
422 are reachable
423
Linus Torvalds1da177e2005-04-16 15:20:36 -0700424downdelay
425
426 Specifies the time, in milliseconds, to wait before disabling
427 a slave after a link failure has been detected. This option
428 is only valid for the miimon link monitor. The downdelay
429 value should be a multiple of the miimon value; if not, it
430 will be rounded down to the nearest multiple. The default
431 value is 0.
432
Jay Vosburghdd957c52007-10-09 19:57:24 -0700433fail_over_mac
434
435 Specifies whether active-backup mode should set all slaves to
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700436 the same MAC address at enslavement (the traditional
437 behavior), or, when enabled, perform special handling of the
438 bond's MAC address in accordance with the selected policy.
Jay Vosburghdd957c52007-10-09 19:57:24 -0700439
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700440 Possible values are:
Jay Vosburghdd957c52007-10-09 19:57:24 -0700441
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700442 none or 0
Jay Vosburghdd957c52007-10-09 19:57:24 -0700443
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700444 This setting disables fail_over_mac, and causes
445 bonding to set all slaves of an active-backup bond to
446 the same MAC address at enslavement time. This is the
447 default.
Jay Vosburghdd957c52007-10-09 19:57:24 -0700448
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700449 active or 1
Jay Vosburghdd957c52007-10-09 19:57:24 -0700450
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700451 The "active" fail_over_mac policy indicates that the
452 MAC address of the bond should always be the MAC
453 address of the currently active slave. The MAC
454 address of the slaves is not changed; instead, the MAC
455 address of the bond changes during a failover.
456
457 This policy is useful for devices that cannot ever
458 alter their MAC address, or for devices that refuse
459 incoming broadcasts with their own source MAC (which
460 interferes with the ARP monitor).
461
462 The down side of this policy is that every device on
463 the network must be updated via gratuitous ARP,
464 vs. just updating a switch or set of switches (which
465 often takes place for any traffic, not just ARP
466 traffic, if the switch snoops incoming traffic to
467 update its tables) for the traditional method. If the
468 gratuitous ARP is lost, communication may be
469 disrupted.
470
Lucas De Marchi25985ed2011-03-30 22:57:33 -0300471 When this policy is used in conjunction with the mii
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700472 monitor, devices which assert link up prior to being
473 able to actually transmit and receive are particularly
Matt LaPlante19f59462009-04-27 15:06:31 +0200474 susceptible to loss of the gratuitous ARP, and an
Jay Vosburgh3915c1e82008-05-17 21:10:14 -0700475 appropriate updelay setting may be required.
476
477 follow or 2
478
479 The "follow" fail_over_mac policy causes the MAC
480 address of the bond to be selected normally (normally
481 the MAC address of the first slave added to the bond).
482 However, the second and subsequent slaves are not set
483 to this MAC address while they are in a backup role; a
484 slave is programmed with the bond's MAC address at
485 failover time (and the formerly active slave receives
486 the newly active slave's MAC address).
487
488 This policy is useful for multiport devices that
489 either become confused or incur a performance penalty
490 when multiple ports are programmed with the same MAC
491 address.
492
493
494 The default policy is none, unless the first slave cannot
495 change its MAC address, in which case the active policy is
496 selected by default.
497
498 This option may be modified via sysfs only when no slaves are
499 present in the bond.
500
501 This option was added in bonding version 3.2.0. The "follow"
502 policy was added in bonding version 3.3.0.
Jay Vosburghdd957c52007-10-09 19:57:24 -0700503
Linus Torvalds1da177e2005-04-16 15:20:36 -0700504lacp_rate
505
506 Option specifying the rate in which we'll ask our link partner
507 to transmit LACPDU packets in 802.3ad mode. Possible values
508 are:
509
510 slow or 0
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700511 Request partner to transmit LACPDUs every 30 seconds
Linus Torvalds1da177e2005-04-16 15:20:36 -0700512
513 fast or 1
514 Request partner to transmit LACPDUs every 1 second
515
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700516 The default is slow.
517
Linus Torvalds1da177e2005-04-16 15:20:36 -0700518max_bonds
519
520 Specifies the number of bonding devices to create for this
521 instance of the bonding driver. E.g., if max_bonds is 3, and
522 the bonding driver is not already loaded, then bond0, bond1
Jay Vosburghb8a97872008-06-13 18:12:04 -0700523 and bond2 will be created. The default value is 1. Specifying
524 a value of 0 will load bonding, but will not create any devices.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700525
526miimon
527
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700528 Specifies the MII link monitoring frequency in milliseconds.
529 This determines how often the link state of each slave is
530 inspected for link failures. A value of zero disables MII
531 link monitoring. A value of 100 is a good starting point.
532 The use_carrier option, below, affects how the link state is
Linus Torvalds1da177e2005-04-16 15:20:36 -0700533 determined. See the High Availability section for additional
534 information. The default value is 0.
535
Nicolas de Pesloüan025890b2011-08-06 07:06:39 +0000536min_links
537
538 Specifies the minimum number of links that must be active before
539 asserting carrier. It is similar to the Cisco EtherChannel min-links
540 feature. This allows setting the minimum number of member ports that
541 must be up (link-up state) before marking the bond device as up
542 (carrier on). This is useful for situations where higher level services
543 such as clustering want to ensure a minimum number of low bandwidth
544 links are active before switchover. This option only affect 802.3ad
545 mode.
546
547 The default value is 0. This will cause carrier to be asserted (for
548 802.3ad mode) whenever there is an active aggregator, regardless of the
549 number of available links in that aggregator. Note that, because an
550 aggregator cannot be active without at least one available link,
551 setting this option to 0 or to 1 has the exact same effect.
552
Linus Torvalds1da177e2005-04-16 15:20:36 -0700553mode
554
555 Specifies one of the bonding policies. The default is
556 balance-rr (round robin). Possible values are:
557
558 balance-rr or 0
559
560 Round-robin policy: Transmit packets in sequential
561 order from the first available slave through the
562 last. This mode provides load balancing and fault
563 tolerance.
564
565 active-backup or 1
566
567 Active-backup policy: Only one slave in the bond is
568 active. A different slave becomes active if, and only
569 if, the active slave fails. The bond's MAC address is
570 externally visible on only one port (network adapter)
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700571 to avoid confusing the switch.
572
573 In bonding version 2.6.2 or later, when a failover
574 occurs in active-backup mode, bonding will issue one
575 or more gratuitous ARPs on the newly active slave.
Auke Kok6224e012006-06-08 11:15:35 -0700576 One gratuitous ARP is issued for the bonding master
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700577 interface and each VLAN interfaces configured above
578 it, provided that the interface has at least one IP
579 address configured. Gratuitous ARPs issued for VLAN
580 interfaces are tagged with the appropriate VLAN id.
581
582 This mode provides fault tolerance. The primary
583 option, documented below, affects the behavior of this
584 mode.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700585
586 balance-xor or 2
587
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700588 XOR policy: Transmit based on the selected transmit
589 hash policy. The default policy is a simple [(source
Jianhua Xie92abf752014-07-17 14:16:26 +0800590 MAC address XOR'd with destination MAC address XOR
591 packet type ID) modulo slave count]. Alternate transmit
592 policies may be selected via the xmit_hash_policy option,
593 described below.
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700594
595 This mode provides load balancing and fault tolerance.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700596
597 broadcast or 3
598
599 Broadcast policy: transmits everything on all slave
600 interfaces. This mode provides fault tolerance.
601
602 802.3ad or 4
603
604 IEEE 802.3ad Dynamic link aggregation. Creates
605 aggregation groups that share the same speed and
606 duplex settings. Utilizes all slaves in the active
607 aggregator according to the 802.3ad specification.
608
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700609 Slave selection for outgoing traffic is done according
610 to the transmit hash policy, which may be changed from
611 the default simple XOR policy via the xmit_hash_policy
612 option, documented below. Note that not all transmit
613 policies may be 802.3ad compliant, particularly in
614 regards to the packet mis-ordering requirements of
615 section 43.2.4 of the 802.3ad standard. Differing
616 peer implementations will have varying tolerances for
617 noncompliance.
618
619 Prerequisites:
Linus Torvalds1da177e2005-04-16 15:20:36 -0700620
621 1. Ethtool support in the base drivers for retrieving
622 the speed and duplex of each slave.
623
624 2. A switch that supports IEEE 802.3ad Dynamic link
625 aggregation.
626
627 Most switches will require some type of configuration
628 to enable 802.3ad mode.
629
630 balance-tlb or 5
631
632 Adaptive transmit load balancing: channel bonding that
Mahesh Bandeware9f0fb82014-04-22 16:30:22 -0700633 does not require any special switch support.
634
635 In tlb_dynamic_lb=1 mode; the outgoing traffic is
636 distributed according to the current load (computed
637 relative to the speed) on each slave.
638
639 In tlb_dynamic_lb=0 mode; the load balancing based on
640 current load is disabled and the load is distributed
641 only using the hash distribution.
642
643 Incoming traffic is received by the current slave.
644 If the receiving slave fails, another slave takes over
645 the MAC address of the failed receiving slave.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700646
647 Prerequisite:
648
649 Ethtool support in the base drivers for retrieving the
650 speed of each slave.
651
652 balance-alb or 6
653
654 Adaptive load balancing: includes balance-tlb plus
655 receive load balancing (rlb) for IPV4 traffic, and
656 does not require any special switch support. The
657 receive load balancing is achieved by ARP negotiation.
658 The bonding driver intercepts the ARP Replies sent by
659 the local system on their way out and overwrites the
660 source hardware address with the unique hardware
661 address of one of the slaves in the bond such that
662 different peers use different hardware addresses for
663 the server.
664
665 Receive traffic from connections created by the server
666 is also balanced. When the local system sends an ARP
667 Request the bonding driver copies and saves the peer's
668 IP information from the ARP packet. When the ARP
669 Reply arrives from the peer, its hardware address is
670 retrieved and the bonding driver initiates an ARP
671 reply to this peer assigning it to one of the slaves
672 in the bond. A problematic outcome of using ARP
673 negotiation for balancing is that each time that an
674 ARP request is broadcast it uses the hardware address
675 of the bond. Hence, peers learn the hardware address
676 of the bond and the balancing of receive traffic
677 collapses to the current slave. This is handled by
678 sending updates (ARP Replies) to all the peers with
679 their individually assigned hardware address such that
680 the traffic is redistributed. Receive traffic is also
681 redistributed when a new slave is added to the bond
682 and when an inactive slave is re-activated. The
683 receive load is distributed sequentially (round robin)
684 among the group of highest speed slaves in the bond.
685
686 When a link is reconnected or a new slave joins the
687 bond the receive traffic is redistributed among all
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700688 active slaves in the bond by initiating ARP Replies
Auke Kok6224e012006-06-08 11:15:35 -0700689 with the selected MAC address to each of the
Linus Torvalds1da177e2005-04-16 15:20:36 -0700690 clients. The updelay parameter (detailed below) must
691 be set to a value equal or greater than the switch's
692 forwarding delay so that the ARP Replies sent to the
693 peers will not be blocked by the switch.
694
695 Prerequisites:
696
697 1. Ethtool support in the base drivers for retrieving
698 the speed of each slave.
699
700 2. Base driver support for setting the hardware
701 address of a device while it is open. This is
702 required so that there will always be one slave in the
703 team using the bond hardware address (the
704 curr_active_slave) while having a unique hardware
705 address for each slave in the bond. If the
706 curr_active_slave fails its hardware address is
707 swapped with the new curr_active_slave that was
708 chosen.
709
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200710num_grat_arp,
Brian Haley305d5522008-11-04 17:51:14 -0800711num_unsol_na
712
Ben Hutchingsad246c992011-04-26 15:25:52 +0000713 Specify the number of peer notifications (gratuitous ARPs and
714 unsolicited IPv6 Neighbor Advertisements) to be issued after a
715 failover event. As soon as the link is up on the new slave
716 (possibly immediately) a peer notification is sent on the
Vincent Bernat0307d582019-07-13 16:35:27 +0200717 bonding device and each VLAN sub-device. This is repeated at
718 the rate specified by peer_notif_delay if the number is
719 greater than 1.
Brian Haley305d5522008-11-04 17:51:14 -0800720
Ben Hutchingsad246c992011-04-26 15:25:52 +0000721 The valid range is 0 - 255; the default value is 1. These options
722 affect only the active-backup mode. These options were added for
723 bonding versions 3.3.0 and 3.4.0 respectively.
724
Jesper Juhl8fb4e132011-08-01 17:59:44 -0700725 From Linux 3.0 and bonding version 3.7.1, these notifications
Ben Hutchingsad246c992011-04-26 15:25:52 +0000726 are generated by the ipv4 and ipv6 code and the numbers of
727 repetitions cannot be set independently.
Brian Haley305d5522008-11-04 17:51:14 -0800728
Nikolay Aleksandrov12465fb2013-11-05 13:51:42 +0100729packets_per_slave
730
731 Specify the number of packets to transmit through a slave before
732 moving to the next one. When set to 0 then a slave is chosen at
733 random.
734
735 The valid range is 0 - 65535; the default value is 1. This option
736 has effect only in balance-rr mode.
737
Vincent Bernat0307d582019-07-13 16:35:27 +0200738peer_notif_delay
739
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200740 Specify the delay, in milliseconds, between each peer
741 notification (gratuitous ARP and unsolicited IPv6 Neighbor
742 Advertisement) when they are issued after a failover event.
743 This delay should be a multiple of the link monitor interval
744 (arp_interval or miimon, whichever is active). The default
745 value is 0 which means to match the value of the link monitor
746 interval.
Vincent Bernat0307d582019-07-13 16:35:27 +0200747
Linus Torvalds1da177e2005-04-16 15:20:36 -0700748primary
749
750 A string (eth0, eth2, etc) specifying which slave is the
751 primary device. The specified device will always be the
752 active slave while it is available. Only when the primary is
753 off-line will alternate devices be used. This is useful when
754 one slave is preferred over another, e.g., when one slave has
755 higher throughput than another.
756
dingtianhonge1d206a2014-01-18 16:28:57 +0800757 The primary option is only valid for active-backup(1),
758 balance-tlb (5) and balance-alb (6) mode.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700759
Jiri Pirkoa5499522009-09-25 03:28:09 +0000760primary_reselect
761
762 Specifies the reselection policy for the primary slave. This
763 affects how the primary slave is chosen to become the active slave
764 when failure of the active slave or recovery of the primary slave
765 occurs. This option is designed to prevent flip-flopping between
766 the primary slave and other slaves. Possible values are:
767
768 always or 0 (default)
769
770 The primary slave becomes the active slave whenever it
771 comes back up.
772
773 better or 1
774
775 The primary slave becomes the active slave when it comes
776 back up, if the speed and duplex of the primary slave is
777 better than the speed and duplex of the current active
778 slave.
779
780 failure or 2
781
782 The primary slave becomes the active slave only if the
783 current active slave fails and the primary slave is up.
784
785 The primary_reselect setting is ignored in two cases:
786
787 If no slaves are active, the first slave to recover is
788 made the active slave.
789
790 When initially enslaved, the primary slave is always made
791 the active slave.
792
793 Changing the primary_reselect policy via sysfs will cause an
794 immediate selection of the best active slave according to the new
795 policy. This may or may not result in a change of the active
796 slave, depending upon the circumstances.
797
798 This option was added for bonding version 3.6.0.
799
Mahesh Bandeware9f0fb82014-04-22 16:30:22 -0700800tlb_dynamic_lb
801
802 Specifies if dynamic shuffling of flows is enabled in tlb
803 mode. The value has no effect on any other modes.
804
805 The default behavior of tlb mode is to shuffle active flows across
806 slaves based on the load in that interval. This gives nice lb
807 characteristics but can cause packet reordering. If re-ordering is
808 a concern use this variable to disable flow shuffling and rely on
809 load balancing provided solely by the hash distribution.
810 xmit-hash-policy can be used to select the appropriate hashing for
811 the setup.
812
813 The sysfs entry can be used to change the setting per bond device
814 and the initial value is derived from the module parameter. The
815 sysfs entry is allowed to be changed only if the bond device is
816 down.
817
818 The default value is "1" that enables flow shuffling while value "0"
819 disables it. This option was added in bonding driver 3.7.1
820
821
Linus Torvalds1da177e2005-04-16 15:20:36 -0700822updelay
823
824 Specifies the time, in milliseconds, to wait before enabling a
825 slave after a link recovery has been detected. This option is
826 only valid for the miimon link monitor. The updelay value
827 should be a multiple of the miimon value; if not, it will be
828 rounded down to the nearest multiple. The default value is 0.
829
830use_carrier
831
832 Specifies whether or not miimon should use MII or ETHTOOL
833 ioctls vs. netif_carrier_ok() to determine the link
834 status. The MII or ETHTOOL ioctls are less efficient and
835 utilize a deprecated calling sequence within the kernel. The
836 netif_carrier_ok() relies on the device driver to maintain its
837 state with netif_carrier_on/off; at this writing, most, but
838 not all, device drivers support this facility.
839
840 If bonding insists that the link is up when it should not be,
841 it may be that your network device driver does not support
842 netif_carrier_on/off. The default state for netif_carrier is
843 "carrier on," so if a driver does not support netif_carrier,
844 it will appear as if the link is always up. In this case,
845 setting use_carrier to 0 will cause bonding to revert to the
846 MII / ETHTOOL ioctl method to determine the link state.
847
848 A value of 1 enables the use of netif_carrier_ok(), a value of
Debabrata Banerjeeb3c898e2018-05-16 14:02:13 -0400849 0 will use the deprecated MII / ETHTOOL ioctls. The default
850 value is 1.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700851
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700852xmit_hash_policy
853
854 Selects the transmit hash policy to use for slave selection in
Mahesh Bandewarf05b42e2014-04-22 16:30:20 -0700855 balance-xor, 802.3ad, and tlb modes. Possible values are:
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700856
857 layer2
858
Jianhua Xie92abf752014-07-17 14:16:26 +0800859 Uses XOR of hardware MAC addresses and packet type ID
860 field to generate the hash. The formula is
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700861
Jianhua Xie92abf752014-07-17 14:16:26 +0800862 hash = source MAC XOR destination MAC XOR packet type ID
863 slave number = hash modulo slave count
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700864
865 This algorithm will place all traffic to a particular
866 network peer on the same slave.
867
868 This algorithm is 802.3ad compliant.
869
Jay Vosburgh6f6652b2007-12-06 23:40:34 -0800870 layer2+3
871
872 This policy uses a combination of layer2 and layer3
873 protocol information to generate the hash.
874
875 Uses XOR of hardware MAC addresses and IP addresses to
Nikolay Aleksandrov7a6afab2013-10-02 13:39:26 +0200876 generate the hash. The formula is
Jay Vosburgh6f6652b2007-12-06 23:40:34 -0800877
Jianhua Xie92abf752014-07-17 14:16:26 +0800878 hash = source MAC XOR destination MAC XOR packet type ID
Nikolay Aleksandrov7a6afab2013-10-02 13:39:26 +0200879 hash = hash XOR source IP XOR destination IP
880 hash = hash XOR (hash RSHIFT 16)
881 hash = hash XOR (hash RSHIFT 8)
882 And then hash is reduced modulo slave count.
Jay Vosburgh6f6652b2007-12-06 23:40:34 -0800883
Nikolay Aleksandrov7a6afab2013-10-02 13:39:26 +0200884 If the protocol is IPv6 then the source and destination
885 addresses are first hashed using ipv6_addr_hash.
John Eaglesham6b923cb2012-08-21 20:43:35 +0000886
Jay Vosburgh6f6652b2007-12-06 23:40:34 -0800887 This algorithm will place all traffic to a particular
888 network peer on the same slave. For non-IP traffic,
889 the formula is the same as for the layer2 transmit
890 hash policy.
891
892 This policy is intended to provide a more balanced
893 distribution of traffic than layer2 alone, especially
894 in environments where a layer3 gateway device is
895 required to reach most destinations.
896
Matt LaPlanted9195882008-07-25 19:45:33 -0700897 This algorithm is 802.3ad compliant.
Jay Vosburgh6f6652b2007-12-06 23:40:34 -0800898
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700899 layer3+4
900
901 This policy uses upper layer protocol information,
902 when available, to generate the hash. This allows for
903 traffic to a particular network peer to span multiple
904 slaves, although a single connection will not span
905 multiple slaves.
906
Nikolay Aleksandrov7a6afab2013-10-02 13:39:26 +0200907 The formula for unfragmented TCP and UDP packets is
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700908
Nikolay Aleksandrov7a6afab2013-10-02 13:39:26 +0200909 hash = source port, destination port (as in the header)
910 hash = hash XOR source IP XOR destination IP
911 hash = hash XOR (hash RSHIFT 16)
912 hash = hash XOR (hash RSHIFT 8)
913 And then hash is reduced modulo slave count.
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700914
Nikolay Aleksandrov7a6afab2013-10-02 13:39:26 +0200915 If the protocol is IPv6 then the source and destination
916 addresses are first hashed using ipv6_addr_hash.
John Eaglesham6b923cb2012-08-21 20:43:35 +0000917
918 For fragmented TCP or UDP packets and all other IPv4 and
919 IPv6 protocol traffic, the source and destination port
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700920 information is omitted. For non-IP traffic, the
921 formula is the same as for the layer2 transmit hash
922 policy.
923
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700924 This algorithm is not fully 802.3ad compliant. A
925 single TCP or UDP conversation containing both
926 fragmented and unfragmented packets will see packets
927 striped across two interfaces. This may result in out
928 of order delivery. Most traffic types will not meet
929 this criteria, as TCP rarely fragments traffic, and
930 most UDP traffic is not involved in extended
931 conversations. Other implementations of 802.3ad may
932 or may not tolerate this noncompliance.
933
Nikolay Aleksandrov7a6afab2013-10-02 13:39:26 +0200934 encap2+3
935
936 This policy uses the same formula as layer2+3 but it
937 relies on skb_flow_dissect to obtain the header fields
938 which might result in the use of inner headers if an
939 encapsulation protocol is used. For example this will
940 improve the performance for tunnel users because the
941 packets will be distributed according to the encapsulated
942 flows.
943
944 encap3+4
945
946 This policy uses the same formula as layer3+4 but it
947 relies on skb_flow_dissect to obtain the header fields
948 which might result in the use of inner headers if an
949 encapsulation protocol is used. For example this will
950 improve the performance for tunnel users because the
951 packets will be distributed according to the encapsulated
952 flows.
953
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700954 The default value is layer2. This option was added in bonding
Jay Vosburgh6f6652b2007-12-06 23:40:34 -0800955 version 2.6.3. In earlier versions of bonding, this parameter
956 does not exist, and the layer2 policy is the only policy. The
957 layer2+3 value was added for bonding version 3.2.2.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700958
Flavio Leitnerc2952c32010-10-05 14:23:59 +0000959resend_igmp
960
961 Specifies the number of IGMP membership reports to be issued after
962 a failover event. One membership report is issued immediately after
963 the failover, subsequent packets are sent in each 200ms interval.
964
Flavio Leitner94265cf2011-05-25 08:38:58 +0000965 The valid range is 0 - 255; the default value is 1. A value of 0
966 prevents the IGMP membership report from being issued in response
967 to the failover event.
968
969 This option is useful for bonding modes balance-rr (0), active-backup
970 (1), balance-tlb (5) and balance-alb (6), in which a failover can
971 switch the IGMP traffic from one slave to another. Therefore a fresh
972 IGMP report must be issued to cause the switch to forward the incoming
973 IGMP traffic over the newly selected slave.
974
975 This option was added for bonding version 3.7.0.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700976
dingtianhong84a6a0a2013-12-21 14:40:22 +0800977lp_interval
978
979 Specifies the number of seconds between instances where the bonding
980 driver sends learning packets to each slaves peer switch.
981
982 The valid range is 1 - 0x7fffffff; the default value is 1. This Option
983 has effect only in balance-tlb and balance-alb modes.
984
Linus Torvalds1da177e2005-04-16 15:20:36 -07009853. Configuring Bonding Devices
986==============================
987
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200988You can configure bonding using either your distro's network
Cong Wangb1098bb2013-05-27 15:49:16 +0000989initialization scripts, or manually using either iproute2 or the
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +0000990sysfs interface. Distros generally use one of three packages for the
991network initialization scripts: initscripts, sysconfig or interfaces.
992Recent versions of these packages have support for bonding, while older
Auke Kok6224e012006-06-08 11:15:35 -0700993versions do not.
Linus Torvalds1da177e2005-04-16 15:20:36 -0700994
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +0200995We will first describe the options for configuring bonding for
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +0000996distros using versions of initscripts, sysconfig and interfaces with full
997or partial support for bonding, then provide information on enabling
Linus Torvalds1da177e2005-04-16 15:20:36 -0700998bonding without support from the network initialization scripts (i.e.,
999older versions of initscripts or sysconfig).
1000
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001001If you're unsure whether your distro uses sysconfig,
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001002initscripts or interfaces, or don't know if it's new enough, have no fear.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001003Determining this is fairly straightforward.
1004
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001005First, look for a file called interfaces in /etc/network directory.
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001006If this file is present in your system, then your system use interfaces. See
1007Configuration with Interfaces Support.
1008
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001009Else, issue the command::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001010
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001011 $ rpm -qf /sbin/ifup
Linus Torvalds1da177e2005-04-16 15:20:36 -07001012
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001013It will respond with a line of text starting with either
Linus Torvalds1da177e2005-04-16 15:20:36 -07001014"initscripts" or "sysconfig," followed by some numbers. This is the
1015package that provides your network initialization scripts.
1016
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001017Next, to determine if your installation supports bonding,
1018issue the command::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001019
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001020 $ grep ifenslave /sbin/ifup
Linus Torvalds1da177e2005-04-16 15:20:36 -07001021
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001022If this returns any matches, then your initscripts or
Linus Torvalds1da177e2005-04-16 15:20:36 -07001023sysconfig has support for bonding.
1024
Auke Kok6224e012006-06-08 11:15:35 -070010253.1 Configuration with Sysconfig Support
Linus Torvalds1da177e2005-04-16 15:20:36 -07001026----------------------------------------
1027
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001028This section applies to distros using a version of sysconfig
Linus Torvalds1da177e2005-04-16 15:20:36 -07001029with bonding support, for example, SuSE Linux Enterprise Server 9.
1030
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001031SuSE SLES 9's networking configuration system does support
Linus Torvalds1da177e2005-04-16 15:20:36 -07001032bonding, however, at this writing, the YaST system configuration
Auke Kok6224e012006-06-08 11:15:35 -07001033front end does not provide any means to work with bonding devices.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001034Bonding devices can be managed by hand, however, as follows.
1035
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001036First, if they have not already been configured, configure the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001037slave devices. On SLES 9, this is most easily done by running the
1038yast2 sysconfig configuration utility. The goal is for to create an
1039ifcfg-id file for each slave device. The simplest way to accomplish
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001040this is to configure the devices for DHCP (this is only to get the
1041file ifcfg-id file created; see below for some issues with DHCP). The
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001042name of the configuration file for each device will be of the form::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001043
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001044 ifcfg-id-xx:xx:xx:xx:xx:xx
Linus Torvalds1da177e2005-04-16 15:20:36 -07001045
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001046Where the "xx" portion will be replaced with the digits from
Linus Torvalds1da177e2005-04-16 15:20:36 -07001047the device's permanent MAC address.
1048
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001049Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
Linus Torvalds1da177e2005-04-16 15:20:36 -07001050created, it is necessary to edit the configuration files for the slave
1051devices (the MAC addresses correspond to those of the slave devices).
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001052Before editing, the file will contain multiple lines, and will look
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001053something like this::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001054
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001055 BOOTPROTO='dhcp'
1056 STARTMODE='on'
1057 USERCTL='no'
1058 UNIQUE='XNzu.WeZGOGF+4wE'
1059 _nm_name='bus-pci-0001:61:01.0'
Linus Torvalds1da177e2005-04-16 15:20:36 -07001060
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001061Change the BOOTPROTO and STARTMODE lines to the following::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001062
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001063 BOOTPROTO='none'
1064 STARTMODE='off'
Linus Torvalds1da177e2005-04-16 15:20:36 -07001065
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001066Do not alter the UNIQUE or _nm_name lines. Remove any other
Linus Torvalds1da177e2005-04-16 15:20:36 -07001067lines (USERCTL, etc).
1068
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001069Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified,
Linus Torvalds1da177e2005-04-16 15:20:36 -07001070it's time to create the configuration file for the bonding device
1071itself. This file is named ifcfg-bondX, where X is the number of the
1072bonding device to create, starting at 0. The first such file is
1073ifcfg-bond0, the second is ifcfg-bond1, and so on. The sysconfig
1074network configuration system will correctly start multiple instances
1075of bonding.
1076
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001077The contents of the ifcfg-bondX file is as follows::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001078
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001079 BOOTPROTO="static"
1080 BROADCAST="10.0.2.255"
1081 IPADDR="10.0.2.10"
1082 NETMASK="255.255.0.0"
1083 NETWORK="10.0.2.0"
1084 REMOTE_IPADDR=""
1085 STARTMODE="onboot"
1086 BONDING_MASTER="yes"
1087 BONDING_MODULE_OPTS="mode=active-backup miimon=100"
1088 BONDING_SLAVE0="eth0"
1089 BONDING_SLAVE1="bus-pci-0000:06:08.1"
Linus Torvalds1da177e2005-04-16 15:20:36 -07001090
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001091Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
Linus Torvalds1da177e2005-04-16 15:20:36 -07001092values with the appropriate values for your network.
1093
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001094The STARTMODE specifies when the device is brought online.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001095The possible values are:
1096
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001097 ======== ======================================================
1098 onboot The device is started at boot time. If you're not
Linus Torvalds1da177e2005-04-16 15:20:36 -07001099 sure, this is probably what you want.
1100
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001101 manual The device is started only when ifup is called
Linus Torvalds1da177e2005-04-16 15:20:36 -07001102 manually. Bonding devices may be configured this
1103 way if you do not wish them to start automatically
1104 at boot for some reason.
1105
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001106 hotplug The device is started by a hotplug event. This is not
Linus Torvalds1da177e2005-04-16 15:20:36 -07001107 a valid choice for a bonding device.
1108
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001109 off or The device configuration is ignored.
1110 ignore
1111 ======== ======================================================
Linus Torvalds1da177e2005-04-16 15:20:36 -07001112
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001113The line BONDING_MASTER='yes' indicates that the device is a
Linus Torvalds1da177e2005-04-16 15:20:36 -07001114bonding master device. The only useful value is "yes."
1115
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001116The contents of BONDING_MODULE_OPTS are supplied to the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001117instance of the bonding module for this device. Specify the options
1118for the bonding mode, link monitoring, and so on here. Do not include
1119the max_bonds bonding parameter; this will confuse the configuration
1120system if you have multiple bonding devices.
1121
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001122Finally, supply one BONDING_SLAVEn="slave device" for each
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001123slave. where "n" is an increasing value, one for each slave. The
1124"slave device" is either an interface name, e.g., "eth0", or a device
1125specifier for the network device. The interface name is easier to
1126find, but the ethN names are subject to change at boot time if, e.g.,
1127a device early in the sequence has failed. The device specifiers
1128(bus-pci-0000:06:08.1 in the example above) specify the physical
1129network device, and will not change unless the device's bus location
1130changes (for example, it is moved from one PCI slot to another). The
1131example above uses one of each type for demonstration purposes; most
1132configurations will choose one or the other for all slave devices.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001133
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001134When all configuration files have been modified or created,
Linus Torvalds1da177e2005-04-16 15:20:36 -07001135networking must be restarted for the configuration changes to take
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001136effect. This can be accomplished via the following::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001137
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001138 # /etc/init.d/network restart
Linus Torvalds1da177e2005-04-16 15:20:36 -07001139
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001140Note that the network control script (/sbin/ifdown) will
Linus Torvalds1da177e2005-04-16 15:20:36 -07001141remove the bonding module as part of the network shutdown processing,
1142so it is not necessary to remove the module by hand if, e.g., the
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001143module parameters have changed.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001144
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001145Also, at this writing, YaST/YaST2 will not manage bonding
Linus Torvalds1da177e2005-04-16 15:20:36 -07001146devices (they do not show bonding interfaces on its list of network
1147devices). It is necessary to edit the configuration file by hand to
1148change the bonding configuration.
1149
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001150Additional general options and details of the ifcfg file
1151format can be found in an example ifcfg template file::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001152
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001153 /etc/sysconfig/network/ifcfg.template
Linus Torvalds1da177e2005-04-16 15:20:36 -07001154
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001155Note that the template does not document the various ``BONDING_*``
Linus Torvalds1da177e2005-04-16 15:20:36 -07001156settings described above, but does describe many of the other options.
1157
Auke Kok6224e012006-06-08 11:15:35 -070011583.1.1 Using DHCP with Sysconfig
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001159-------------------------------
1160
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001161Under sysconfig, configuring a device with BOOTPROTO='dhcp'
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001162will cause it to query DHCP for its IP address information. At this
1163writing, this does not function for bonding devices; the scripts
1164attempt to obtain the device address from DHCP prior to adding any of
1165the slave devices. Without active slaves, the DHCP requests are not
1166sent to the network.
1167
Auke Kok6224e012006-06-08 11:15:35 -070011683.1.2 Configuring Multiple Bonds with Sysconfig
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001169-----------------------------------------------
1170
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001171The sysconfig network initialization system is capable of
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001172handling multiple bonding devices. All that is necessary is for each
1173bonding instance to have an appropriately configured ifcfg-bondX file
1174(as described above). Do not specify the "max_bonds" parameter to any
1175instance of bonding, as this will confuse sysconfig. If you require
1176multiple bonding devices with identical parameters, create multiple
1177ifcfg-bondX files.
1178
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001179Because the sysconfig scripts supply the bonding module
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001180options in the ifcfg-bondX file, it is not necessary to add them to
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001181the system ``/etc/modules.d/*.conf`` configuration files.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001182
Auke Kok6224e012006-06-08 11:15:35 -070011833.2 Configuration with Initscripts Support
Linus Torvalds1da177e2005-04-16 15:20:36 -07001184------------------------------------------
1185
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001186This section applies to distros using a recent version of
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001187initscripts with bonding support, for example, Red Hat Enterprise Linux
1188version 3 or later, Fedora, etc. On these systems, the network
1189initialization scripts have knowledge of bonding, and can be configured to
1190control bonding devices. Note that older versions of the initscripts
1191package have lower levels of support for bonding; this will be noted where
1192applicable.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001193
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001194These distros will not automatically load the network adapter
Linus Torvalds1da177e2005-04-16 15:20:36 -07001195driver unless the ethX device is configured with an IP address.
1196Because of this constraint, users must manually configure a
1197network-script file for all physical adapters that will be members of
1198a bondX link. Network script files are located in the directory:
1199
1200/etc/sysconfig/network-scripts
1201
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001202The file name must be prefixed with "ifcfg-eth" and suffixed
Linus Torvalds1da177e2005-04-16 15:20:36 -07001203with the adapter's physical adapter number. For example, the script
1204for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0.
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001205Place the following text in the file::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001206
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001207 DEVICE=eth0
1208 USERCTL=no
1209 ONBOOT=yes
1210 MASTER=bond0
1211 SLAVE=yes
1212 BOOTPROTO=none
Linus Torvalds1da177e2005-04-16 15:20:36 -07001213
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001214The DEVICE= line will be different for every ethX device and
Linus Torvalds1da177e2005-04-16 15:20:36 -07001215must correspond with the name of the file, i.e., ifcfg-eth1 must have
1216a device line of DEVICE=eth1. The setting of the MASTER= line will
1217also depend on the final bonding interface name chosen for your bond.
1218As with other network devices, these typically start at 0, and go up
1219one for each device, i.e., the first bonding instance is bond0, the
1220second is bond1, and so on.
1221
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001222Next, create a bond network script. The file name for this
Linus Torvalds1da177e2005-04-16 15:20:36 -07001223script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is
1224the number of the bond. For bond0 the file is named "ifcfg-bond0",
1225for bond1 it is named "ifcfg-bond1", and so on. Within that file,
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001226place the following text::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001227
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001228 DEVICE=bond0
1229 IPADDR=192.168.1.1
1230 NETMASK=255.255.255.0
1231 NETWORK=192.168.1.0
1232 BROADCAST=192.168.1.255
1233 ONBOOT=yes
1234 BOOTPROTO=none
1235 USERCTL=no
Linus Torvalds1da177e2005-04-16 15:20:36 -07001236
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001237Be sure to change the networking specific lines (IPADDR,
Linus Torvalds1da177e2005-04-16 15:20:36 -07001238NETMASK, NETWORK and BROADCAST) to match your network configuration.
1239
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001240For later versions of initscripts, such as that found with Fedora
Andy Gospodarek3f8b4b12008-10-22 11:19:48 +000012417 (or later) and Red Hat Enterprise Linux version 5 (or later), it is possible,
1242and, indeed, preferable, to specify the bonding options in the ifcfg-bond0
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001243file, e.g. a line of the format::
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001244
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001245 BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=192.168.1.254"
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001246
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001247will configure the bond with the specified options. The options
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001248specified in BONDING_OPTS are identical to the bonding module parameters
Andy Gospodarek3f8b4b12008-10-22 11:19:48 +00001249except for the arp_ip_target field when using versions of initscripts older
1250than and 8.57 (Fedora 8) and 8.45.19 (Red Hat Enterprise Linux 5.2). When
1251using older versions each target should be included as a separate option and
1252should be preceded by a '+' to indicate it should be added to the list of
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001253queried targets, e.g.,::
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001254
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001255 arp_ip_target=+192.168.1.1 arp_ip_target=+192.168.1.2
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001256
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001257is the proper syntax to specify multiple targets. When specifying
1258options via BONDING_OPTS, it is not necessary to edit
1259``/etc/modprobe.d/*.conf``.
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001260
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001261For even older versions of initscripts that do not support
Lucas De Marchi970e2482012-03-30 13:37:16 -07001262BONDING_OPTS, it is necessary to edit /etc/modprobe.d/*.conf, depending upon
1263your distro) to load the bonding module with your desired options when the
1264bond0 interface is brought up. The following lines in /etc/modprobe.d/*.conf
1265will load the bonding module, and select its options:
Linus Torvalds1da177e2005-04-16 15:20:36 -07001266
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001267 alias bond0 bonding
1268 options bond0 mode=balance-alb miimon=100
Linus Torvalds1da177e2005-04-16 15:20:36 -07001269
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001270Replace the sample parameters with the appropriate set of
Linus Torvalds1da177e2005-04-16 15:20:36 -07001271options for your configuration.
1272
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001273Finally run "/etc/rc.d/init.d/network restart" as root. This
Linus Torvalds1da177e2005-04-16 15:20:36 -07001274will restart the networking subsystem and your bond link should be now
1275up and running.
1276
Auke Kok6224e012006-06-08 11:15:35 -070012773.2.1 Using DHCP with Initscripts
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001278---------------------------------
1279
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001280Recent versions of initscripts (the versions supplied with Fedora
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001281Core 3 and Red Hat Enterprise Linux 4, or later versions, are reported to
1282work) have support for assigning IP information to bonding devices via
1283DHCP.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001284
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001285To configure bonding for DHCP, configure it as described
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001286above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp"
1287and add a line consisting of "TYPE=Bonding". Note that the TYPE value
1288is case sensitive.
1289
Auke Kok6224e012006-06-08 11:15:35 -070012903.2.2 Configuring Multiple Bonds with Initscripts
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001291-------------------------------------------------
1292
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001293Initscripts packages that are included with Fedora 7 and Red Hat
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001294Enterprise Linux 5 support multiple bonding interfaces by simply
1295specifying the appropriate BONDING_OPTS= in ifcfg-bondX where X is the
1296number of the bond. This support requires sysfs support in the kernel,
1297and a bonding driver of version 3.0.0 or later. Other configurations may
1298not support this method for specifying multiple bonding interfaces; for
1299those instances, see the "Configuring Multiple Bonds Manually" section,
1300below.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001301
Cong Wangb1098bb2013-05-27 15:49:16 +000013023.3 Configuring Bonding Manually with iproute2
Auke Kok6224e012006-06-08 11:15:35 -07001303-----------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07001304
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001305This section applies to distros whose network initialization
Linus Torvalds1da177e2005-04-16 15:20:36 -07001306scripts (the sysconfig or initscripts package) do not have specific
1307knowledge of bonding. One such distro is SuSE Linux Enterprise Server
1308version 8.
1309
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001310The general method for these systems is to place the bonding
Lucas De Marchi970e2482012-03-30 13:37:16 -07001311module parameters into a config file in /etc/modprobe.d/ (as
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001312appropriate for the installed distro), then add modprobe and/or
Cong Wangb1098bb2013-05-27 15:49:16 +00001313`ip link` commands to the system's global init script. The name of
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001314the global init script differs; for sysconfig, it is
Linus Torvalds1da177e2005-04-16 15:20:36 -07001315/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
1316
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001317For example, if you wanted to make a simple bond of two e100
Linus Torvalds1da177e2005-04-16 15:20:36 -07001318devices (presumed to be eth0 and eth1), and have it persist across
1319reboots, edit the appropriate file (/etc/init.d/boot.local or
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001320/etc/rc.d/rc.local), and add the following::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001321
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001322 modprobe bonding mode=balance-alb miimon=100
1323 modprobe e100
1324 ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
1325 ip link set eth0 master bond0
1326 ip link set eth1 master bond0
Linus Torvalds1da177e2005-04-16 15:20:36 -07001327
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001328Replace the example bonding module parameters and bond0
Linus Torvalds1da177e2005-04-16 15:20:36 -07001329network configuration (IP address, netmask, etc) with the appropriate
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001330values for your configuration.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001331
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001332Unfortunately, this method will not provide support for the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001333ifup and ifdown scripts on the bond devices. To reload the bonding
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001334configuration, it is necessary to run the initialization script, e.g.,::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001335
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001336 # /etc/init.d/boot.local
Linus Torvalds1da177e2005-04-16 15:20:36 -07001337
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001338or::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001339
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001340 # /etc/rc.d/rc.local
Linus Torvalds1da177e2005-04-16 15:20:36 -07001341
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001342It may be desirable in such a case to create a separate script
Linus Torvalds1da177e2005-04-16 15:20:36 -07001343which only initializes the bonding configuration, then call that
1344separate script from within boot.local. This allows for bonding to be
1345enabled without re-running the entire global init script.
1346
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001347To shut down the bonding devices, it is necessary to first
Linus Torvalds1da177e2005-04-16 15:20:36 -07001348mark the bonding device itself as being down, then remove the
1349appropriate device driver modules. For our example above, you can do
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001350the following::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001351
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001352 # ifconfig bond0 down
1353 # rmmod bonding
1354 # rmmod e100
Linus Torvalds1da177e2005-04-16 15:20:36 -07001355
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001356Again, for convenience, it may be desirable to create a script
Linus Torvalds1da177e2005-04-16 15:20:36 -07001357with these commands.
1358
1359
Jay Vosburgh00354cf2005-07-21 12:18:02 -070013603.3.1 Configuring Multiple Bonds Manually
1361-----------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07001362
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001363This section contains information on configuring multiple
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001364bonding devices with differing options for those systems whose network
1365initialization scripts lack support for configuring multiple bonds.
1366
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001367If you require multiple bonding devices, but all with the same
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001368options, you may wish to use the "max_bonds" module parameter,
1369documented above.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001370
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001371To create multiple bonding devices with differing options, it is
Rick Jonesf8b72d32012-07-20 10:51:37 +00001372preferable to use bonding parameters exported by sysfs, documented in the
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001373section below.
Linus Torvalds1da177e2005-04-16 15:20:36 -07001374
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001375For versions of bonding without sysfs support, the only means to
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001376provide multiple instances of bonding with differing options is to load
1377the bonding driver multiple times. Note that current versions of the
1378sysconfig network initialization scripts handle this automatically; if
1379your distro uses these scripts, no special action is needed. See the
1380section Configuring Bonding Devices, above, if you're not sure about your
1381network initialization scripts.
1382
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001383To load multiple instances of the module, it is necessary to
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001384specify a different name for each instance (the module loading system
1385requires that every loaded module, even multiple instances of the same
1386module, have a unique name). This is accomplished by supplying multiple
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001387sets of bonding options in ``/etc/modprobe.d/*.conf``, for example::
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001388
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001389 alias bond0 bonding
1390 options bond0 -o bond0 mode=balance-rr miimon=100
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001391
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001392 alias bond1 bonding
1393 options bond1 -o bond1 mode=balance-alb miimon=50
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001394
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001395will load the bonding module two times. The first instance is
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001396named "bond0" and creates the bond0 device in balance-rr mode with an
1397miimon of 100. The second instance is named "bond1" and creates the
1398bond1 device in balance-alb mode with an miimon of 50.
1399
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001400In some circumstances (typically with older distributions),
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001401the above does not work, and the second bonding instance never sees
1402its options. In that case, the second options line can be substituted
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001403as follows::
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001404
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001405 install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \
1406 mode=balance-alb miimon=50
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001407
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001408This may be repeated any number of times, specifying a new and
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001409unique name in place of bond1 for each subsequent instance.
1410
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001411It has been observed that some Red Hat supplied kernels are unable
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08001412to rename modules at load time (the "-o bond1" part). Attempts to pass
1413that option to modprobe will produce an "Operation not permitted" error.
1414This has been reported on some Fedora Core kernels, and has been seen on
1415RHEL 4 as well. On kernels exhibiting this problem, it will be impossible
1416to configure multiple bonds with differing parameters (as they are older
1417kernels, and also lack sysfs support).
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001418
Auke Kok6224e012006-06-08 11:15:35 -070014193.4 Configuring Bonding Manually via Sysfs
1420------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07001421
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001422Starting with version 3.0.0, Channel Bonding may be configured
Auke Kok6224e012006-06-08 11:15:35 -07001423via the sysfs interface. This interface allows dynamic configuration
1424of all bonds in the system without unloading the module. It also
1425allows for adding and removing bonds at runtime. Ifenslave is no
1426longer required, though it is still supported.
1427
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001428Use of the sysfs interface allows you to use multiple bonds
Auke Kok6224e012006-06-08 11:15:35 -07001429with different configurations without having to reload the module.
1430It also allows you to use multiple, differently configured bonds when
1431bonding is compiled into the kernel.
1432
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001433You must have the sysfs filesystem mounted to configure
Auke Kok6224e012006-06-08 11:15:35 -07001434bonding this way. The examples in this document assume that you
1435are using the standard mount point for sysfs, e.g. /sys. If your
1436sysfs filesystem is mounted elsewhere, you will need to adjust the
1437example paths accordingly.
1438
1439Creating and Destroying Bonds
1440-----------------------------
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001441To add a new bond foo::
Auke Kok6224e012006-06-08 11:15:35 -07001442
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001443 # echo +foo > /sys/class/net/bonding_masters
Auke Kok6224e012006-06-08 11:15:35 -07001444
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001445To remove an existing bond bar::
Auke Kok6224e012006-06-08 11:15:35 -07001446
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001447 # echo -bar > /sys/class/net/bonding_masters
1448
1449To show all existing bonds::
1450
1451 # cat /sys/class/net/bonding_masters
1452
1453.. note::
1454
1455 due to 4K size limitation of sysfs files, this list may be
1456 truncated if you have more than a few hundred bonds. This is unlikely
1457 to occur under normal operating conditions.
Auke Kok6224e012006-06-08 11:15:35 -07001458
1459Adding and Removing Slaves
1460--------------------------
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001461Interfaces may be enslaved to a bond using the file
Auke Kok6224e012006-06-08 11:15:35 -07001462/sys/class/net/<bond>/bonding/slaves. The semantics for this file
1463are the same as for the bonding_masters file.
1464
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001465To enslave interface eth0 to bond bond0::
Auke Kok6224e012006-06-08 11:15:35 -07001466
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001467 # ifconfig bond0 up
1468 # echo +eth0 > /sys/class/net/bond0/bonding/slaves
Auke Kok6224e012006-06-08 11:15:35 -07001469
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001470To free slave eth0 from bond bond0::
1471
1472 # echo -eth0 > /sys/class/net/bond0/bonding/slaves
1473
1474When an interface is enslaved to a bond, symlinks between the
Auke Kok6224e012006-06-08 11:15:35 -07001475two are created in the sysfs filesystem. In this case, you would get
1476/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and
1477/sys/class/net/eth0/master pointing to /sys/class/net/bond0.
1478
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001479This means that you can tell quickly whether or not an
Auke Kok6224e012006-06-08 11:15:35 -07001480interface is enslaved by looking for the master symlink. Thus:
1481# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves
1482will free eth0 from whatever bond it is enslaved to, regardless of
1483the name of the bond interface.
1484
1485Changing a Bond's Configuration
1486-------------------------------
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001487Each bond may be configured individually by manipulating the
Auke Kok6224e012006-06-08 11:15:35 -07001488files located in /sys/class/net/<bond name>/bonding
1489
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001490The names of these files correspond directly with the command-
Paolo Ornati670e9f32006-10-03 22:57:56 +02001491line parameters described elsewhere in this file, and, with the
Auke Kok6224e012006-06-08 11:15:35 -07001492exception of arp_ip_target, they accept the same values. To see the
1493current setting, simply cat the appropriate file.
1494
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001495A few examples will be given here; for specific usage
Auke Kok6224e012006-06-08 11:15:35 -07001496guidelines for each parameter, see the appropriate section in this
1497document.
1498
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001499To configure bond0 for balance-alb mode::
Auke Kok6224e012006-06-08 11:15:35 -07001500
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001501 # ifconfig bond0 down
1502 # echo 6 > /sys/class/net/bond0/bonding/mode
1503 - or -
1504 # echo balance-alb > /sys/class/net/bond0/bonding/mode
Auke Kok6224e012006-06-08 11:15:35 -07001505
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001506.. note::
Auke Kok6224e012006-06-08 11:15:35 -07001507
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001508 The bond interface must be down before the mode can be changed.
Auke Kok6224e012006-06-08 11:15:35 -07001509
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001510To enable MII monitoring on bond0 with a 1 second interval::
1511
1512 # echo 1000 > /sys/class/net/bond0/bonding/miimon
1513
1514.. note::
1515
1516 If ARP monitoring is enabled, it will disabled when MII
1517 monitoring is enabled, and vice-versa.
1518
1519To add ARP targets::
1520
1521 # echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
1522 # echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target
1523
1524.. note::
1525
1526 up to 16 target addresses may be specified.
1527
1528To remove an ARP target::
1529
1530 # echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
1531
1532To configure the interval between learning packet transmits::
1533
1534 # echo 12 > /sys/class/net/bond0/bonding/lp_interval
1535
1536.. note::
1537
1538 the lp_interval is the number of seconds between instances where
1539 the bonding driver sends learning packets to each slaves peer switch. The
1540 default interval is 1 second.
Neil Horman7eacd032013-09-13 11:05:33 -04001541
Auke Kok6224e012006-06-08 11:15:35 -07001542Example Configuration
1543---------------------
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001544We begin with the same example that is shown in section 3.3,
Auke Kok6224e012006-06-08 11:15:35 -07001545executed with sysfs, and without using ifenslave.
1546
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001547To make a simple bond of two e100 devices (presumed to be eth0
Auke Kok6224e012006-06-08 11:15:35 -07001548and eth1), and have it persist across reboots, edit the appropriate
1549file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001550following::
Auke Kok6224e012006-06-08 11:15:35 -07001551
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001552 modprobe bonding
1553 modprobe e100
1554 echo balance-alb > /sys/class/net/bond0/bonding/mode
1555 ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
1556 echo 100 > /sys/class/net/bond0/bonding/miimon
1557 echo +eth0 > /sys/class/net/bond0/bonding/slaves
1558 echo +eth1 > /sys/class/net/bond0/bonding/slaves
Auke Kok6224e012006-06-08 11:15:35 -07001559
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001560To add a second bond, with two e1000 interfaces in
Auke Kok6224e012006-06-08 11:15:35 -07001561active-backup mode, using ARP monitoring, add the following lines to
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001562your init script::
Auke Kok6224e012006-06-08 11:15:35 -07001563
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001564 modprobe e1000
1565 echo +bond1 > /sys/class/net/bonding_masters
1566 echo active-backup > /sys/class/net/bond1/bonding/mode
1567 ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up
1568 echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target
1569 echo 2000 > /sys/class/net/bond1/bonding/arp_interval
1570 echo +eth2 > /sys/class/net/bond1/bonding/slaves
1571 echo +eth3 > /sys/class/net/bond1/bonding/slaves
Auke Kok6224e012006-06-08 11:15:35 -07001572
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +000015733.5 Configuration with Interfaces Support
1574-----------------------------------------
1575
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001576This section applies to distros which use /etc/network/interfaces file
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001577to describe network interface configuration, most notably Debian and it's
1578derivatives.
1579
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001580The ifup and ifdown commands on Debian don't support bonding out of
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001581the box. The ifenslave-2.6 package should be installed to provide bonding
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001582support. Once installed, this package will provide ``bond-*`` options
1583to be used into /etc/network/interfaces.
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001584
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001585Note that ifenslave-2.6 package will load the bonding module and use
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001586the ifenslave command when appropriate.
1587
1588Example Configurations
1589----------------------
1590
1591In /etc/network/interfaces, the following stanza will configure bond0, in
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001592active-backup mode, with eth0 and eth1 as slaves::
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001593
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001594 auto bond0
1595 iface bond0 inet dhcp
1596 bond-slaves eth0 eth1
1597 bond-mode active-backup
1598 bond-miimon 100
1599 bond-primary eth0 eth1
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001600
1601If the above configuration doesn't work, you might have a system using
1602upstart for system startup. This is most notably true for recent
1603Ubuntu versions. The following stanza in /etc/network/interfaces will
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001604produce the same result on those systems::
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001605
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001606 auto bond0
1607 iface bond0 inet dhcp
1608 bond-slaves none
1609 bond-mode active-backup
1610 bond-miimon 100
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001611
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001612 auto eth0
1613 iface eth0 inet manual
1614 bond-master bond0
1615 bond-primary eth0 eth1
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001616
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001617 auto eth1
1618 iface eth1 inet manual
1619 bond-master bond0
1620 bond-primary eth0 eth1
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001621
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001622For a full list of ``bond-*`` supported options in /etc/network/interfaces and
1623some more advanced examples tailored to you particular distros, see the files in
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001624/usr/share/doc/ifenslave-2.6.
1625
16263.6 Overriding Configuration for Special Cases
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001627----------------------------------------------
Nicolas de Pesloüande221bd2011-01-24 13:21:37 +00001628
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001629When using the bonding driver, the physical port which transmits a frame is
1630typically selected by the bonding driver, and is not relevant to the user or
1631system administrator. The output port is simply selected using the policies of
1632the selected bonding mode. On occasion however, it is helpful to direct certain
1633classes of traffic to certain physical interfaces on output to implement
1634slightly more complex policies. For example, to reach a web server over a
1635bonded interface in which eth0 connects to a private network, while eth1
1636connects via a public network, it may be desirous to bias the bond to send said
1637traffic over eth0 first, using eth1 only as a fall back, while all other traffic
1638can safely be sent over either interface. Such configurations may be achieved
1639using the traffic control utilities inherent in linux.
Auke Kok6224e012006-06-08 11:15:35 -07001640
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001641By default the bonding driver is multiqueue aware and 16 queues are created
Mauro Carvalho Chehabe98aa6822020-04-30 18:04:01 +02001642when the driver initializes (see Documentation/networking/multiqueue.rst
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001643for details). If more or less queues are desired the module parameter
1644tx_queues can be used to change this value. There is no sysfs parameter
1645available as the allocation is done at module init time.
1646
1647The output of the file /proc/net/bonding/bondX has changed so the output Queue
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001648ID is now printed for each slave::
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001649
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001650 Bonding Mode: fault-tolerance (active-backup)
1651 Primary Slave: None
1652 Currently Active Slave: eth0
1653 MII Status: up
1654 MII Polling Interval (ms): 0
1655 Up Delay (ms): 0
1656 Down Delay (ms): 0
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001657
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001658 Slave Interface: eth0
1659 MII Status: up
1660 Link Failure Count: 0
1661 Permanent HW addr: 00:1a:a0:12:8f:cb
1662 Slave queue ID: 0
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001663
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001664 Slave Interface: eth1
1665 MII Status: up
1666 Link Failure Count: 0
1667 Permanent HW addr: 00:1a:a0:12:8f:cc
1668 Slave queue ID: 2
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001669
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001670The queue_id for a slave can be set using the command::
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001671
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001672 # echo "eth1:2" > /sys/class/net/bond0/bonding/queue_id
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001673
1674Any interface that needs a queue_id set should set it with multiple calls
1675like the one above until proper priorities are set for all interfaces. On
1676distributions that allow configuration via initscripts, multiple 'queue_id'
1677arguments can be added to BONDING_OPTS to set all needed slave queues.
1678
1679These queue id's can be used in conjunction with the tc utility to configure
1680a multiqueue qdisc and filters to bias certain traffic to transmit on certain
1681slave devices. For instance, say we wanted, in the above configuration to
1682force all traffic bound to 192.168.1.100 to use eth1 in the bond as its output
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001683device. The following commands would accomplish this::
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001684
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001685 # tc qdisc add dev bond0 handle 1 root multiq
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001686
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001687 # tc filter add dev bond0 protocol ip parent 1: prio 1 u32 match ip \
1688 dst 192.168.1.100 action skbedit queue_mapping 2
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001689
1690These commands tell the kernel to attach a multiqueue queue discipline to the
1691bond0 interface and filter traffic enqueued to it, such that packets with a dst
1692ip of 192.168.1.100 have their output queue mapping value overwritten to 2.
1693This value is then passed into the driver, causing the normal output path
1694selection policy to be overridden, selecting instead qid 2, which maps to eth1.
1695
1696Note that qid values begin at 1. Qid 0 is reserved to initiate to the driver
1697that normal output policy selection should take place. One benefit to simply
1698leaving the qid for a slave to 0 is the multiqueue awareness in the bonding
1699driver that is now present. This awareness allows tc filters to be placed on
1700slave devices as well as bond devices and the bonding driver will simply act as
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001701a pass-through for selecting output queues on the slave device rather than
Andy Gospodarekbb1d9122010-06-02 08:40:18 +00001702output port selection.
1703
1704This feature first appeared in bonding driver version 3.7.0 and support for
1705output slave selection was limited to round-robin and active-backup modes.
1706
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -070017073.7 Configuring LACP for 802.3ad mode in a more secure way
1708----------------------------------------------------------
1709
1710When using 802.3ad bonding mode, the Actor (host) and Partner (switch)
1711exchange LACPDUs. These LACPDUs cannot be sniffed, because they are
1712destined to link local mac addresses (which switches/bridges are not
1713supposed to forward). However, most of the values are easily predictable
1714or are simply the machine's MAC address (which is trivially known to all
1715other hosts in the same L2). This implies that other machines in the L2
1716domain can spoof LACPDU packets from other hosts to the switch and potentially
1717cause mayhem by joining (from the point of view of the switch) another
1718machine's aggregate, thus receiving a portion of that hosts incoming
1719traffic and / or spoofing traffic from that machine themselves (potentially
1720even successfully terminating some portion of flows). Though this is not
1721a likely scenario, one could avoid this possibility by simply configuring
1722few bonding parameters:
1723
1724 (a) ad_actor_system : You can set a random mac-address that can be used for
1725 these LACPDU exchanges. The value can not be either NULL or Multicast.
1726 Also it's preferable to set the local-admin bit. Following shell code
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001727 generates a random mac-address as described above::
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -07001728
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001729 # sys_mac_addr=$(printf '%02x:%02x:%02x:%02x:%02x:%02x' \
1730 $(( (RANDOM & 0xFE) | 0x02 )) \
1731 $(( RANDOM & 0xFF )) \
1732 $(( RANDOM & 0xFF )) \
1733 $(( RANDOM & 0xFF )) \
1734 $(( RANDOM & 0xFF )) \
1735 $(( RANDOM & 0xFF )))
1736 # echo $sys_mac_addr > /sys/class/net/bond0/bonding/ad_actor_system
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -07001737
1738 (b) ad_actor_sys_prio : Randomize the system priority. The default value
1739 is 65535, but system can take the value from 1 - 65535. Following shell
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001740 code generates random priority and sets it::
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -07001741
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001742 # sys_prio=$(( 1 + RANDOM + RANDOM ))
1743 # echo $sys_prio > /sys/class/net/bond0/bonding/ad_actor_sys_prio
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -07001744
1745 (c) ad_user_port_key : Use the user portion of the port-key. The default
1746 keeps this empty. These are the upper 10 bits of the port-key and value
1747 ranges from 0 - 1023. Following shell code generates these 10 bits and
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001748 sets it::
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -07001749
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001750 # usr_port_key=$(( RANDOM & 0x3FF ))
1751 # echo $usr_port_key > /sys/class/net/bond0/bonding/ad_user_port_key
Mahesh Bandeward22a5fc2015-05-09 00:01:57 -07001752
1753
Andy Gospodarekbb1d9122010-06-02 08:40:18 +000017544 Querying Bonding Configuration
Linus Torvalds1da177e2005-04-16 15:20:36 -07001755=================================
1756
Auke Kok6224e012006-06-08 11:15:35 -070017574.1 Bonding Configuration
Linus Torvalds1da177e2005-04-16 15:20:36 -07001758-------------------------
1759
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001760Each bonding device has a read-only file residing in the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001761/proc/net/bonding directory. The file contents include information
1762about the bonding configuration, options and state of each slave.
1763
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001764For example, the contents of /proc/net/bonding/bond0 after the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001765driver is loaded with parameters of mode=0 and miimon=1000 is
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001766generally as follows::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001767
1768 Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004)
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001769 Bonding Mode: load balancing (round-robin)
1770 Currently Active Slave: eth0
1771 MII Status: up
1772 MII Polling Interval (ms): 1000
1773 Up Delay (ms): 0
1774 Down Delay (ms): 0
Linus Torvalds1da177e2005-04-16 15:20:36 -07001775
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001776 Slave Interface: eth1
1777 MII Status: up
1778 Link Failure Count: 1
Linus Torvalds1da177e2005-04-16 15:20:36 -07001779
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001780 Slave Interface: eth0
1781 MII Status: up
1782 Link Failure Count: 1
Linus Torvalds1da177e2005-04-16 15:20:36 -07001783
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001784The precise format and contents will change depending upon the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001785bonding configuration, state, and version of the bonding driver.
1786
Auke Kok6224e012006-06-08 11:15:35 -070017874.2 Network configuration
Linus Torvalds1da177e2005-04-16 15:20:36 -07001788-------------------------
1789
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001790The network configuration can be inspected using the ifconfig
Linus Torvalds1da177e2005-04-16 15:20:36 -07001791command. Bonding devices will have the MASTER flag set; Bonding slave
1792devices will have the SLAVE flag set. The ifconfig output does not
1793contain information on which slaves are associated with which masters.
1794
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001795In the example below, the bond0 interface is the master
Linus Torvalds1da177e2005-04-16 15:20:36 -07001796(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of
1797bond0 have the same MAC address (HWaddr) as bond0 for all modes except
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001798TLB and ALB that require a unique MAC address for each slave::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001799
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001800 # /sbin/ifconfig
1801 bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
1802 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
1803 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
1804 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
1805 TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
1806 collisions:0 txqueuelen:0
Linus Torvalds1da177e2005-04-16 15:20:36 -07001807
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001808 eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
1809 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
1810 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
1811 TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
1812 collisions:0 txqueuelen:100
1813 Interrupt:10 Base address:0x1080
Linus Torvalds1da177e2005-04-16 15:20:36 -07001814
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001815 eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
1816 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
1817 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
1818 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
1819 collisions:0 txqueuelen:100
1820 Interrupt:9 Base address:0x1400
Linus Torvalds1da177e2005-04-16 15:20:36 -07001821
Auke Kok6224e012006-06-08 11:15:35 -070018225. Switch Configuration
Linus Torvalds1da177e2005-04-16 15:20:36 -07001823=======================
1824
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001825For this section, "switch" refers to whatever system the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001826bonded devices are directly connected to (i.e., where the other end of
1827the cable plugs into). This may be an actual dedicated switch device,
1828or it may be another regular system (e.g., another computer running
1829Linux),
1830
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001831The active-backup, balance-tlb and balance-alb modes do not
Linus Torvalds1da177e2005-04-16 15:20:36 -07001832require any specific configuration of the switch.
1833
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001834The 802.3ad mode requires that the switch have the appropriate
Linus Torvalds1da177e2005-04-16 15:20:36 -07001835ports configured as an 802.3ad aggregation. The precise method used
1836to configure this varies from switch to switch, but, for example, a
1837Cisco 3550 series switch requires that the appropriate ports first be
1838grouped together in a single etherchannel instance, then that
1839etherchannel is set to mode "lacp" to enable 802.3ad (instead of
1840standard EtherChannel).
1841
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001842The balance-rr, balance-xor and broadcast modes generally
Linus Torvalds1da177e2005-04-16 15:20:36 -07001843require that the switch have the appropriate ports grouped together.
1844The nomenclature for such a group differs between switches, it may be
1845called an "etherchannel" (as in the Cisco example, above), a "trunk
1846group" or some other similar variation. For these modes, each switch
1847will also have its own configuration options for the switch's transmit
1848policy to the bond. Typical choices include XOR of either the MAC or
1849IP addresses. The transmit policy of the two peers does not need to
1850match. For these three modes, the bonding mode really selects a
1851transmit policy for an EtherChannel group; all three will interoperate
1852with another EtherChannel group.
1853
1854
Auke Kok6224e012006-06-08 11:15:35 -070018556. 802.1q VLAN Support
Linus Torvalds1da177e2005-04-16 15:20:36 -07001856======================
1857
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001858It is possible to configure VLAN devices over a bond interface
Linus Torvalds1da177e2005-04-16 15:20:36 -07001859using the 8021q driver. However, only packets coming from the 8021q
1860driver and passing through bonding will be tagged by default. Self
1861generated packets, for example, bonding's learning packets or ARP
1862packets generated by either ALB mode or the ARP monitor mechanism, are
1863tagged internally by bonding itself. As a result, bonding must
1864"learn" the VLAN IDs configured above it, and use those IDs to tag
1865self generated packets.
1866
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001867For reasons of simplicity, and to support the use of adapters
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001868that can do VLAN hardware acceleration offloading, the bonding
1869interface declares itself as fully hardware offloading capable, it gets
Linus Torvalds1da177e2005-04-16 15:20:36 -07001870the add_vid/kill_vid notifications to gather the necessary
1871information, and it propagates those actions to the slaves. In case
1872of mixed adapter types, hardware accelerated tagged packets that
1873should go through an adapter that is not offloading capable are
1874"un-accelerated" by the bonding driver so the VLAN tag sits in the
1875regular location.
1876
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001877VLAN interfaces *must* be added on top of a bonding interface
Linus Torvalds1da177e2005-04-16 15:20:36 -07001878only after enslaving at least one slave. The bonding interface has a
1879hardware address of 00:00:00:00:00:00 until the first slave is added.
1880If the VLAN interface is created prior to the first enslavement, it
1881would pick up the all-zeroes hardware address. Once the first slave
1882is attached to the bond, the bond device itself will pick up the
1883slave's hardware address, which is then available for the VLAN device.
1884
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001885Also, be aware that a similar problem can occur if all slaves
Linus Torvalds1da177e2005-04-16 15:20:36 -07001886are released from a bond that still has one or more VLAN interfaces on
1887top of it. When a new slave is added, the bonding interface will
1888obtain its hardware address from the first slave, which might not
1889match the hardware address of the VLAN interfaces (which was
1890ultimately copied from an earlier slave).
1891
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001892There are two methods to insure that the VLAN device operates
Linus Torvalds1da177e2005-04-16 15:20:36 -07001893with the correct hardware address if all slaves are removed from a
1894bond interface:
1895
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020018961. Remove all VLAN interfaces then recreate them
Linus Torvalds1da177e2005-04-16 15:20:36 -07001897
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +020018982. Set the bonding interface's hardware address so that it
Linus Torvalds1da177e2005-04-16 15:20:36 -07001899matches the hardware address of the VLAN interfaces.
1900
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001901Note that changing a VLAN interface's HW address would set the
Jay Vosburgh00354cf2005-07-21 12:18:02 -07001902underlying device -- i.e. the bonding interface -- to promiscuous
Linus Torvalds1da177e2005-04-16 15:20:36 -07001903mode, which might not be what you want.
1904
1905
Auke Kok6224e012006-06-08 11:15:35 -070019067. Link Monitoring
Linus Torvalds1da177e2005-04-16 15:20:36 -07001907==================
1908
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001909The bonding driver at present supports two schemes for
Linus Torvalds1da177e2005-04-16 15:20:36 -07001910monitoring a slave device's link state: the ARP monitor and the MII
1911monitor.
1912
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001913At the present time, due to implementation restrictions in the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001914bonding driver itself, it is not possible to enable both ARP and MII
1915monitoring simultaneously.
1916
Auke Kok6224e012006-06-08 11:15:35 -070019177.1 ARP Monitor Operation
Linus Torvalds1da177e2005-04-16 15:20:36 -07001918-------------------------
1919
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001920The ARP monitor operates as its name suggests: it sends ARP
Linus Torvalds1da177e2005-04-16 15:20:36 -07001921queries to one or more designated peer systems on the network, and
1922uses the response as an indication that the link is operating. This
1923gives some assurance that traffic is actually flowing to and from one
1924or more peers on the local network.
1925
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001926The ARP monitor relies on the device driver itself to verify
Linus Torvalds1da177e2005-04-16 15:20:36 -07001927that traffic is flowing. In particular, the driver must keep up to
Florian Westphal5c2a9642016-05-04 22:51:47 +02001928date the last receive time, dev->last_rx. Drivers that use NETIF_F_LLTX
1929flag must also update netdev_queue->trans_start. If they do not, then the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001930ARP monitor will immediately fail any slaves using that driver, and
1931those slaves will stay down. If networking monitoring (tcpdump, etc)
1932shows the ARP requests and replies on the network, then it may be that
1933your device driver is not updating last_rx and trans_start.
1934
Auke Kok6224e012006-06-08 11:15:35 -070019357.2 Configuring Multiple ARP Targets
Linus Torvalds1da177e2005-04-16 15:20:36 -07001936------------------------------------
1937
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001938While ARP monitoring can be done with just one target, it can
Linus Torvalds1da177e2005-04-16 15:20:36 -07001939be useful in a High Availability setup to have several targets to
1940monitor. In the case of just one target, the target itself may go
1941down or have a problem making it unresponsive to ARP requests. Having
1942an additional target (or several) increases the reliability of the ARP
1943monitoring.
1944
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001945Multiple ARP targets must be separated by commas as follows::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001946
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001947 # example options for ARP monitoring with three targets
1948 alias bond0 bonding
1949 options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
Linus Torvalds1da177e2005-04-16 15:20:36 -07001950
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001951For just a single target the options would resemble::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001952
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001953 # example options for ARP monitoring with one target
1954 alias bond0 bonding
1955 options bond0 arp_interval=60 arp_ip_target=192.168.0.100
Linus Torvalds1da177e2005-04-16 15:20:36 -07001956
1957
Auke Kok6224e012006-06-08 11:15:35 -070019587.3 MII Monitor Operation
Linus Torvalds1da177e2005-04-16 15:20:36 -07001959-------------------------
1960
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001961The MII monitor monitors only the carrier state of the local
Linus Torvalds1da177e2005-04-16 15:20:36 -07001962network interface. It accomplishes this in one of three ways: by
1963depending upon the device driver to maintain its carrier state, by
1964querying the device's MII registers, or by making an ethtool query to
1965the device.
1966
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001967If the use_carrier module parameter is 1 (the default value),
Linus Torvalds1da177e2005-04-16 15:20:36 -07001968then the MII monitor will rely on the driver for carrier state
1969information (via the netif_carrier subsystem). As explained in the
1970use_carrier parameter information, above, if the MII monitor fails to
1971detect carrier loss on the device (e.g., when the cable is physically
1972disconnected), it may be that the driver does not support
1973netif_carrier.
1974
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001975If use_carrier is 0, then the MII monitor will first query the
Linus Torvalds1da177e2005-04-16 15:20:36 -07001976device's (via ioctl) MII registers and check the link state. If that
1977request fails (not just that it returns carrier down), then the MII
1978monitor will make an ethtool ETHOOL_GLINK request to attempt to obtain
1979the same information. If both methods fail (i.e., the driver either
1980does not support or had some error in processing both the MII register
1981and ethtool requests), then the MII monitor will assume the link is
1982up.
1983
Auke Kok6224e012006-06-08 11:15:35 -070019848. Potential Sources of Trouble
Linus Torvalds1da177e2005-04-16 15:20:36 -07001985===============================
1986
Auke Kok6224e012006-06-08 11:15:35 -070019878.1 Adventures in Routing
Linus Torvalds1da177e2005-04-16 15:20:36 -07001988-------------------------
1989
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001990When bonding is configured, it is important that the slave
Auke Kok6224e012006-06-08 11:15:35 -07001991devices not have routes that supersede routes of the master (or,
Linus Torvalds1da177e2005-04-16 15:20:36 -07001992generally, not have routes at all). For example, suppose the bonding
1993device bond0 has two slaves, eth0 and eth1, and the routing table is
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001994as follows::
Linus Torvalds1da177e2005-04-16 15:20:36 -07001995
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02001996 Kernel IP routing table
1997 Destination Gateway Genmask Flags MSS Window irtt Iface
1998 10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0
1999 10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1
2000 10.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0
2001 127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo
Linus Torvalds1da177e2005-04-16 15:20:36 -07002002
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002003This routing configuration will likely still update the
Linus Torvalds1da177e2005-04-16 15:20:36 -07002004receive/transmit times in the driver (needed by the ARP monitor), but
2005may bypass the bonding driver (because outgoing traffic to, in this
2006case, another host on network 10 would use eth0 or eth1 before bond0).
2007
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002008The ARP monitor (and ARP itself) may become confused by this
Linus Torvalds1da177e2005-04-16 15:20:36 -07002009configuration, because ARP requests (generated by the ARP monitor)
2010will be sent on one interface (bond0), but the corresponding reply
2011will arrive on a different interface (eth0). This reply looks to ARP
2012as an unsolicited ARP reply (because ARP matches replies on an
2013interface basis), and is discarded. The MII monitor is not affected
2014by the state of the routing table.
2015
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002016The solution here is simply to insure that slaves do not have
Linus Torvalds1da177e2005-04-16 15:20:36 -07002017routes of their own, and if for some reason they must, those routes do
Auke Kok6224e012006-06-08 11:15:35 -07002018not supersede routes of their master. This should generally be the
Linus Torvalds1da177e2005-04-16 15:20:36 -07002019case, but unusual configurations or errant manual or automatic static
2020route additions may cause trouble.
2021
Auke Kok6224e012006-06-08 11:15:35 -070020228.2 Ethernet Device Renaming
Linus Torvalds1da177e2005-04-16 15:20:36 -07002023----------------------------
2024
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002025On systems with network configuration scripts that do not
Linus Torvalds1da177e2005-04-16 15:20:36 -07002026associate physical devices directly with network interface names (so
2027that the same physical device always has the same "ethX" name), it may
Lucas De Marchi970e2482012-03-30 13:37:16 -07002028be necessary to add some special logic to config files in
2029/etc/modprobe.d/.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002030
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002031For example, given a modules.conf containing the following::
Linus Torvalds1da177e2005-04-16 15:20:36 -07002032
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002033 alias bond0 bonding
2034 options bond0 mode=some-mode miimon=50
2035 alias eth0 tg3
2036 alias eth1 tg3
2037 alias eth2 e1000
2038 alias eth3 e1000
Linus Torvalds1da177e2005-04-16 15:20:36 -07002039
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002040If neither eth0 and eth1 are slaves to bond0, then when the
Linus Torvalds1da177e2005-04-16 15:20:36 -07002041bond0 interface comes up, the devices may end up reordered. This
2042happens because bonding is loaded first, then its slave device's
2043drivers are loaded next. Since no other drivers have been loaded,
2044when the e1000 driver loads, it will receive eth0 and eth1 for its
2045devices, but the bonding configuration tries to enslave eth2 and eth3
2046(which may later be assigned to the tg3 devices).
2047
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002048Adding the following::
Linus Torvalds1da177e2005-04-16 15:20:36 -07002049
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002050 add above bonding e1000 tg3
Linus Torvalds1da177e2005-04-16 15:20:36 -07002051
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002052causes modprobe to load e1000 then tg3, in that order, when
Linus Torvalds1da177e2005-04-16 15:20:36 -07002053bonding is loaded. This command is fully documented in the
2054modules.conf manual page.
2055
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002056On systems utilizing modprobe an equivalent problem can occur.
Lucas De Marchi970e2482012-03-30 13:37:16 -07002057In this case, the following can be added to config files in
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002058/etc/modprobe.d/ as::
Linus Torvalds1da177e2005-04-16 15:20:36 -07002059
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002060 softdep bonding pre: tg3 e1000
Linus Torvalds1da177e2005-04-16 15:20:36 -07002061
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002062This will load tg3 and e1000 modules before loading the bonding one.
Lucas De Marchi970e2482012-03-30 13:37:16 -07002063Full documentation on this can be found in the modprobe.d and modprobe
2064manual pages.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002065
Auke Kok6224e012006-06-08 11:15:35 -070020668.3. Painfully Slow Or No Failed Link Detection By Miimon
Linus Torvalds1da177e2005-04-16 15:20:36 -07002067---------------------------------------------------------
2068
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002069By default, bonding enables the use_carrier option, which
Linus Torvalds1da177e2005-04-16 15:20:36 -07002070instructs bonding to trust the driver to maintain carrier state.
2071
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002072As discussed in the options section, above, some drivers do
Linus Torvalds1da177e2005-04-16 15:20:36 -07002073not support the netif_carrier_on/_off link state tracking system.
2074With use_carrier enabled, bonding will always see these links as up,
2075regardless of their actual state.
2076
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002077Additionally, other drivers do support netif_carrier, but do
Linus Torvalds1da177e2005-04-16 15:20:36 -07002078not maintain it in real time, e.g., only polling the link state at
2079some fixed interval. In this case, miimon will detect failures, but
2080only after some long period of time has expired. If it appears that
2081miimon is very slow in detecting link failures, try specifying
2082use_carrier=0 to see if that improves the failure detection time. If
2083it does, then it may be that the driver checks the carrier state at a
2084fixed interval, but does not cache the MII register values (so the
2085use_carrier=0 method of querying the registers directly works). If
2086use_carrier=0 does not improve the failover, then the driver may cache
2087the registers, or the problem may be elsewhere.
2088
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002089Also, remember that miimon only checks for the device's
Linus Torvalds1da177e2005-04-16 15:20:36 -07002090carrier state. It has no way to determine the state of devices on or
2091beyond other ports of a switch, or if a switch is refusing to pass
2092traffic while still maintaining carrier on.
2093
Auke Kok6224e012006-06-08 11:15:35 -070020949. SNMP agents
Linus Torvalds1da177e2005-04-16 15:20:36 -07002095===============
2096
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002097If running SNMP agents, the bonding driver should be loaded
Linus Torvalds1da177e2005-04-16 15:20:36 -07002098before any network drivers participating in a bond. This requirement
Tobias Klauserd533f672005-09-10 00:26:46 -07002099is due to the interface index (ipAdEntIfIndex) being associated to
Linus Torvalds1da177e2005-04-16 15:20:36 -07002100the first interface found with a given IP address. That is, there is
2101only one ipAdEntIfIndex for each IP address. For example, if eth0 and
2102eth1 are slaves of bond0 and the driver for eth0 is loaded before the
2103bonding driver, the interface for the IP address will be associated
2104with the eth0 interface. This configuration is shown below, the IP
2105address 192.168.1.1 has an interface index of 2 which indexes to eth0
2106in the ifDescr table (ifDescr.2).
2107
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002108::
2109
Linus Torvalds1da177e2005-04-16 15:20:36 -07002110 interfaces.ifTable.ifEntry.ifDescr.1 = lo
2111 interfaces.ifTable.ifEntry.ifDescr.2 = eth0
2112 interfaces.ifTable.ifEntry.ifDescr.3 = eth1
2113 interfaces.ifTable.ifEntry.ifDescr.4 = eth2
2114 interfaces.ifTable.ifEntry.ifDescr.5 = eth3
2115 interfaces.ifTable.ifEntry.ifDescr.6 = bond0
2116 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
2117 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
2118 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
2119 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
2120
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002121This problem is avoided by loading the bonding driver before
Linus Torvalds1da177e2005-04-16 15:20:36 -07002122any network drivers participating in a bond. Below is an example of
2123loading the bonding driver first, the IP address 192.168.1.1 is
2124correctly associated with ifDescr.2.
2125
2126 interfaces.ifTable.ifEntry.ifDescr.1 = lo
2127 interfaces.ifTable.ifEntry.ifDescr.2 = bond0
2128 interfaces.ifTable.ifEntry.ifDescr.3 = eth0
2129 interfaces.ifTable.ifEntry.ifDescr.4 = eth1
2130 interfaces.ifTable.ifEntry.ifDescr.5 = eth2
2131 interfaces.ifTable.ifEntry.ifDescr.6 = eth3
2132 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
2133 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
2134 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
2135 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
2136
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002137While some distributions may not report the interface name in
Linus Torvalds1da177e2005-04-16 15:20:36 -07002138ifDescr, the association between the IP address and IfIndex remains
2139and SNMP functions such as Interface_Scan_Next will report that
2140association.
2141
Auke Kok6224e012006-06-08 11:15:35 -0700214210. Promiscuous mode
Linus Torvalds1da177e2005-04-16 15:20:36 -07002143====================
2144
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002145When running network monitoring tools, e.g., tcpdump, it is
Linus Torvalds1da177e2005-04-16 15:20:36 -07002146common to enable promiscuous mode on the device, so that all traffic
2147is seen (instead of seeing only traffic destined for the local host).
2148The bonding driver handles promiscuous mode changes to the bonding
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002149master device (e.g., bond0), and propagates the setting to the slave
Linus Torvalds1da177e2005-04-16 15:20:36 -07002150devices.
2151
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002152For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002153the promiscuous mode setting is propagated to all slaves.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002154
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002155For the active-backup, balance-tlb and balance-alb modes, the
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002156promiscuous mode setting is propagated only to the active slave.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002157
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002158For balance-tlb mode, the active slave is the slave currently
Linus Torvalds1da177e2005-04-16 15:20:36 -07002159receiving inbound traffic.
2160
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002161For balance-alb mode, the active slave is the slave used as a
Linus Torvalds1da177e2005-04-16 15:20:36 -07002162"primary." This slave is used for mode-specific control traffic, for
2163sending to peers that are unassigned or if the load is unbalanced.
2164
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002165For the active-backup, balance-tlb and balance-alb modes, when
Linus Torvalds1da177e2005-04-16 15:20:36 -07002166the active slave changes (e.g., due to a link failure), the
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002167promiscuous setting will be propagated to the new active slave.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002168
Auke Kok6224e012006-06-08 11:15:35 -0700216911. Configuring Bonding for High Availability
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002170=============================================
Linus Torvalds1da177e2005-04-16 15:20:36 -07002171
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002172High Availability refers to configurations that provide
Linus Torvalds1da177e2005-04-16 15:20:36 -07002173maximum network availability by having redundant or backup devices,
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002174links or switches between the host and the rest of the world. The
2175goal is to provide the maximum availability of network connectivity
2176(i.e., the network always works), even though other configurations
2177could provide higher throughput.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002178
Auke Kok6224e012006-06-08 11:15:35 -0700217911.1 High Availability in a Single Switch Topology
Linus Torvalds1da177e2005-04-16 15:20:36 -07002180--------------------------------------------------
2181
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002182If two hosts (or a host and a single switch) are directly
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002183connected via multiple physical links, then there is no availability
2184penalty to optimizing for maximum bandwidth. In this case, there is
2185only one switch (or peer), so if it fails, there is no alternative
2186access to fail over to. Additionally, the bonding load balance modes
2187support link monitoring of their members, so if individual links fail,
2188the load will be rebalanced across the remaining devices.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002189
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002190See Section 12, "Configuring Bonding for Maximum Throughput"
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002191for information on configuring bonding with one peer device.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002192
Auke Kok6224e012006-06-08 11:15:35 -0700219311.2 High Availability in a Multiple Switch Topology
Linus Torvalds1da177e2005-04-16 15:20:36 -07002194----------------------------------------------------
2195
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002196With multiple switches, the configuration of bonding and the
Linus Torvalds1da177e2005-04-16 15:20:36 -07002197network changes dramatically. In multiple switch topologies, there is
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002198a trade off between network availability and usable bandwidth.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002199
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002200Below is a sample network, configured to maximize the
2201availability of the network::
Linus Torvalds1da177e2005-04-16 15:20:36 -07002202
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002203 | |
2204 |port3 port3|
2205 +-----+----+ +-----+----+
2206 | |port2 ISL port2| |
2207 | switch A +--------------------------+ switch B |
2208 | | | |
2209 +-----+----+ +-----++---+
2210 |port1 port1|
2211 | +-------+ |
2212 +-------------+ host1 +---------------+
2213 eth0 +-------+ eth1
Linus Torvalds1da177e2005-04-16 15:20:36 -07002214
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002215In this configuration, there is a link between the two
Linus Torvalds1da177e2005-04-16 15:20:36 -07002216switches (ISL, or inter switch link), and multiple ports connecting to
2217the outside world ("port3" on each switch). There is no technical
2218reason that this could not be extended to a third switch.
2219
Auke Kok6224e012006-06-08 11:15:35 -0700222011.2.1 HA Bonding Mode Selection for Multiple Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002221-------------------------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002222
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002223In a topology such as the example above, the active-backup and
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002224broadcast modes are the only useful bonding modes when optimizing for
2225availability; the other modes require all links to terminate on the
2226same peer for them to behave rationally.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002227
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002228active-backup:
2229 This is generally the preferred mode, particularly if
Linus Torvalds1da177e2005-04-16 15:20:36 -07002230 the switches have an ISL and play together well. If the
2231 network configuration is such that one switch is specifically
2232 a backup switch (e.g., has lower capacity, higher cost, etc),
2233 then the primary option can be used to insure that the
2234 preferred link is always used when it is available.
2235
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002236broadcast:
2237 This mode is really a special purpose mode, and is suitable
Linus Torvalds1da177e2005-04-16 15:20:36 -07002238 only for very specific needs. For example, if the two
2239 switches are not connected (no ISL), and the networks beyond
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002240 them are totally independent. In this case, if it is
Linus Torvalds1da177e2005-04-16 15:20:36 -07002241 necessary for some specific one-way traffic to reach both
2242 independent networks, then the broadcast mode may be suitable.
2243
Auke Kok6224e012006-06-08 11:15:35 -0700224411.2.2 HA Link Monitoring Selection for Multiple Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002245----------------------------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002246
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002247The choice of link monitoring ultimately depends upon your
Linus Torvalds1da177e2005-04-16 15:20:36 -07002248switch. If the switch can reliably fail ports in response to other
2249failures, then either the MII or ARP monitors should work. For
2250example, in the above example, if the "port3" link fails at the remote
2251end, the MII monitor has no direct means to detect this. The ARP
2252monitor could be configured with a target at the remote end of port3,
2253thus detecting that failure without switch support.
2254
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002255In general, however, in a multiple switch topology, the ARP
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002256monitor can provide a higher level of reliability in detecting end to
2257end connectivity failures (which may be caused by the failure of any
2258individual component to pass traffic for any reason). Additionally,
2259the ARP monitor should be configured with multiple targets (at least
2260one for each switch in the network). This will insure that,
Linus Torvalds1da177e2005-04-16 15:20:36 -07002261regardless of which switch is active, the ARP monitor has a suitable
2262target to query.
2263
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002264Note, also, that of late many switches now support a functionality
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002265generally referred to as "trunk failover." This is a feature of the
2266switch that causes the link state of a particular switch port to be set
2267down (or up) when the state of another switch port goes down (or up).
Matt LaPlante19f59462009-04-27 15:06:31 +02002268Its purpose is to propagate link failures from logically "exterior" ports
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002269to the logically "interior" ports that bonding is able to monitor via
2270miimon. Availability and configuration for trunk failover varies by
2271switch, but this can be a viable alternative to the ARP monitor when using
2272suitable switches.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002273
Auke Kok6224e012006-06-08 11:15:35 -0700227412. Configuring Bonding for Maximum Throughput
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002275==============================================
Linus Torvalds1da177e2005-04-16 15:20:36 -07002276
Auke Kok6224e012006-06-08 11:15:35 -0700227712.1 Maximizing Throughput in a Single Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002278------------------------------------------------------
2279
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002280In a single switch configuration, the best method to maximize
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002281throughput depends upon the application and network environment. The
2282various load balancing modes each have strengths and weaknesses in
2283different environments, as detailed below.
2284
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002285For this discussion, we will break down the topologies into
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002286two categories. Depending upon the destination of most traffic, we
2287categorize them into either "gatewayed" or "local" configurations.
2288
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002289In a gatewayed configuration, the "switch" is acting primarily
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002290as a router, and the majority of traffic passes through this router to
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002291other networks. An example would be the following::
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002292
2293
2294 +----------+ +----------+
2295 | |eth0 port1| | to other networks
2296 | Host A +---------------------+ router +------------------->
2297 | +---------------------+ | Hosts B and C are out
2298 | |eth1 port2| | here somewhere
2299 +----------+ +----------+
2300
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002301The router may be a dedicated router device, or another host
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002302acting as a gateway. For our discussion, the important point is that
2303the majority of traffic from Host A will pass through the router to
2304some other network before reaching its final destination.
2305
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002306In a gatewayed network configuration, although Host A may
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002307communicate with many other systems, all of its traffic will be sent
2308and received via one other peer on the local network, the router.
2309
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002310Note that the case of two systems connected directly via
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002311multiple physical links is, for purposes of configuring bonding, the
2312same as a gatewayed configuration. In that case, it happens that all
2313traffic is destined for the "gateway" itself, not some other network
2314beyond the gateway.
2315
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002316In a local configuration, the "switch" is acting primarily as
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002317a switch, and the majority of traffic passes through this switch to
2318reach other stations on the same network. An example would be the
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002319following::
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002320
2321 +----------+ +----------+ +--------+
2322 | |eth0 port1| +-------+ Host B |
2323 | Host A +------------+ switch |port3 +--------+
2324 | +------------+ | +--------+
2325 | |eth1 port2| +------------------+ Host C |
2326 +----------+ +----------+port4 +--------+
2327
2328
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002329Again, the switch may be a dedicated switch device, or another
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002330host acting as a gateway. For our discussion, the important point is
2331that the majority of traffic from Host A is destined for other hosts
2332on the same local network (Hosts B and C in the above example).
2333
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002334In summary, in a gatewayed configuration, traffic to and from
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002335the bonded device will be to the same MAC level peer on the network
2336(the gateway itself, i.e., the router), regardless of its final
2337destination. In a local configuration, traffic flows directly to and
2338from the final destinations, thus, each destination (Host B, Host C)
2339will be addressed directly by their individual MAC addresses.
2340
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002341This distinction between a gatewayed and a local network
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002342configuration is important because many of the load balancing modes
2343available use the MAC addresses of the local network source and
2344destination to make load balancing decisions. The behavior of each
2345mode is described below.
2346
2347
Auke Kok6224e012006-06-08 11:15:35 -0700234812.1.1 MT Bonding Mode Selection for Single Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002349-----------------------------------------------------------
2350
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002351This configuration is the easiest to set up and to understand,
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002352although you will have to decide which bonding mode best suits your
2353needs. The trade offs for each mode are detailed below:
2354
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002355balance-rr:
2356 This mode is the only mode that will permit a single
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002357 TCP/IP connection to stripe traffic across multiple
2358 interfaces. It is therefore the only mode that will allow a
2359 single TCP/IP stream to utilize more than one interface's
2360 worth of throughput. This comes at a cost, however: the
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002361 striping generally results in peer systems receiving packets out
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002362 of order, causing TCP/IP's congestion control system to kick
2363 in, often by retransmitting segments.
2364
2365 It is possible to adjust TCP/IP's congestion limits by
2366 altering the net.ipv4.tcp_reordering sysctl parameter. The
Eric Dumazetdca145f2014-10-27 21:45:24 -07002367 usual default value is 3. But keep in mind TCP stack is able
2368 to automatically increase this when it detects reorders.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002369
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002370 Note that the fraction of packets that will be delivered out of
2371 order is highly variable, and is unlikely to be zero. The level
2372 of reordering depends upon a variety of factors, including the
2373 networking interfaces, the switch, and the topology of the
2374 configuration. Speaking in general terms, higher speed network
2375 cards produce more reordering (due to factors such as packet
2376 coalescing), and a "many to many" topology will reorder at a
2377 higher rate than a "many slow to one fast" configuration.
2378
2379 Many switches do not support any modes that stripe traffic
2380 (instead choosing a port based upon IP or MAC level addresses);
2381 for those devices, traffic for a particular connection flowing
2382 through the switch to a balance-rr bond will not utilize greater
2383 than one interface's worth of bandwidth.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002384
2385 If you are utilizing protocols other than TCP/IP, UDP for
2386 example, and your application can tolerate out of order
2387 delivery, then this mode can allow for single stream datagram
2388 performance that scales near linearly as interfaces are added
2389 to the bond.
2390
2391 This mode requires the switch to have the appropriate ports
2392 configured for "etherchannel" or "trunking."
2393
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002394active-backup:
2395 There is not much advantage in this network topology to
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002396 the active-backup mode, as the inactive backup devices are all
2397 connected to the same peer as the primary. In this case, a
2398 load balancing mode (with link monitoring) will provide the
2399 same level of network availability, but with increased
2400 available bandwidth. On the plus side, active-backup mode
2401 does not require any configuration of the switch, so it may
2402 have value if the hardware available does not support any of
2403 the load balance modes.
2404
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002405balance-xor:
2406 This mode will limit traffic such that packets destined
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002407 for specific peers will always be sent over the same
2408 interface. Since the destination is determined by the MAC
2409 addresses involved, this mode works best in a "local" network
2410 configuration (as described above), with destinations all on
2411 the same local network. This mode is likely to be suboptimal
2412 if all your traffic is passed through a single router (i.e., a
2413 "gatewayed" network configuration, as described above).
2414
2415 As with balance-rr, the switch ports need to be configured for
2416 "etherchannel" or "trunking."
2417
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002418broadcast:
2419 Like active-backup, there is not much advantage to this
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002420 mode in this type of network topology.
2421
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002422802.3ad:
2423 This mode can be a good choice for this type of network
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002424 topology. The 802.3ad mode is an IEEE standard, so all peers
2425 that implement 802.3ad should interoperate well. The 802.3ad
2426 protocol includes automatic configuration of the aggregates,
2427 so minimal manual configuration of the switch is needed
2428 (typically only to designate that some set of devices is
2429 available for 802.3ad). The 802.3ad standard also mandates
2430 that frames be delivered in order (within certain limits), so
2431 in general single connections will not see misordering of
2432 packets. The 802.3ad mode does have some drawbacks: the
2433 standard mandates that all devices in the aggregate operate at
2434 the same speed and duplex. Also, as with all bonding load
2435 balance modes other than balance-rr, no single connection will
2436 be able to utilize more than a single interface's worth of
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002437 bandwidth.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002438
2439 Additionally, the linux bonding 802.3ad implementation
Jianhua Xie92abf752014-07-17 14:16:26 +08002440 distributes traffic by peer (using an XOR of MAC addresses
2441 and packet type ID), so in a "gatewayed" configuration, all
2442 outgoing traffic will generally use the same device. Incoming
2443 traffic may also end up on a single device, but that is
Axel Beckert00a534e2017-10-05 22:00:33 +02002444 dependent upon the balancing policy of the peer's 802.3ad
Jianhua Xie92abf752014-07-17 14:16:26 +08002445 implementation. In a "local" configuration, traffic will be
2446 distributed across the devices in the bond.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002447
2448 Finally, the 802.3ad mode mandates the use of the MII monitor,
2449 therefore, the ARP monitor is not available in this mode.
2450
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002451balance-tlb:
2452 The balance-tlb mode balances outgoing traffic by peer.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002453 Since the balancing is done according to MAC address, in a
2454 "gatewayed" configuration (as described above), this mode will
2455 send all traffic across a single device. However, in a
2456 "local" network configuration, this mode balances multiple
2457 local network peers across devices in a vaguely intelligent
2458 manner (not a simple XOR as in balance-xor or 802.3ad mode),
2459 so that mathematically unlucky MAC addresses (i.e., ones that
2460 XOR to the same value) will not all "bunch up" on a single
2461 interface.
2462
2463 Unlike 802.3ad, interfaces may be of differing speeds, and no
2464 special switch configuration is required. On the down side,
2465 in this mode all incoming traffic arrives over a single
2466 interface, this mode requires certain ethtool support in the
2467 network device driver of the slave interfaces, and the ARP
2468 monitor is not available.
2469
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002470balance-alb:
2471 This mode is everything that balance-tlb is, and more.
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002472 It has all of the features (and restrictions) of balance-tlb,
2473 and will also balance incoming traffic from local network
2474 peers (as described in the Bonding Module Options section,
2475 above).
2476
2477 The only additional down side to this mode is that the network
2478 device driver must support changing the hardware address while
2479 the device is open.
2480
Auke Kok6224e012006-06-08 11:15:35 -0700248112.1.2 MT Link Monitoring for Single Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002482----------------------------------------------------
2483
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002484The choice of link monitoring may largely depend upon which
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002485mode you choose to use. The more advanced load balancing modes do not
2486support the use of the ARP monitor, and are thus restricted to using
2487the MII monitor (which does not provide as high a level of end to end
2488assurance as the ARP monitor).
2489
Auke Kok6224e012006-06-08 11:15:35 -0700249012.2 Maximum Throughput in a Multiple Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002491-----------------------------------------------------
2492
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002493Multiple switches may be utilized to optimize for throughput
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002494when they are configured in parallel as part of an isolated network
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002495between two or more systems, for example::
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002496
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002497 +-----------+
2498 | Host A |
2499 +-+---+---+-+
2500 | | |
2501 +--------+ | +---------+
2502 | | |
2503 +------+---+ +-----+----+ +-----+----+
2504 | Switch A | | Switch B | | Switch C |
2505 +------+---+ +-----+----+ +-----+----+
2506 | | |
2507 +--------+ | +---------+
2508 | | |
2509 +-+---+---+-+
2510 | Host B |
2511 +-----------+
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002512
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002513In this configuration, the switches are isolated from one
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002514another. One reason to employ a topology such as this is for an
2515isolated network with many hosts (a cluster configured for high
2516performance, for example), using multiple smaller switches can be more
2517cost effective than a single larger switch, e.g., on a network with 24
2518hosts, three 24 port switches can be significantly less expensive than
2519a single 72 port switch.
2520
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002521If access beyond the network is required, an individual host
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002522can be equipped with an additional network device connected to an
2523external network; this host then additionally acts as a gateway.
2524
Auke Kok6224e012006-06-08 11:15:35 -0700252512.2.1 MT Bonding Mode Selection for Multiple Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002526-------------------------------------------------------------
2527
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002528In actual practice, the bonding mode typically employed in
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002529configurations of this type is balance-rr. Historically, in this
2530network configuration, the usual caveats about out of order packet
2531delivery are mitigated by the use of network adapters that do not do
2532any kind of packet coalescing (via the use of NAPI, or because the
2533device itself does not generate interrupts until some number of
2534packets has arrived). When employed in this fashion, the balance-rr
2535mode allows individual connections between two hosts to effectively
2536utilize greater than one interface's bandwidth.
2537
Auke Kok6224e012006-06-08 11:15:35 -0700253812.2.2 MT Link Monitoring for Multiple Switch Topology
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002539------------------------------------------------------
2540
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002541Again, in actual practice, the MII monitor is most often used
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002542in this configuration, as performance is given preference over
2543availability. The ARP monitor will function in this topology, but its
2544advantages over the MII monitor are mitigated by the volume of probes
2545needed as the number of systems involved grows (remember that each
2546host in the network is configured with bonding).
2547
Auke Kok6224e012006-06-08 11:15:35 -0700254813. Switch Behavior Issues
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002549==========================
2550
Auke Kok6224e012006-06-08 11:15:35 -0700255113.1 Link Establishment and Failover Delays
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002552-------------------------------------------
2553
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002554Some switches exhibit undesirable behavior with regard to the
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002555timing of link up and down reporting by the switch.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002556
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002557First, when a link comes up, some switches may indicate that
Linus Torvalds1da177e2005-04-16 15:20:36 -07002558the link is up (carrier available), but not pass traffic over the
2559interface for some period of time. This delay is typically due to
2560some type of autonegotiation or routing protocol, but may also occur
2561during switch initialization (e.g., during recovery after a switch
2562failure). If you find this to be a problem, specify an appropriate
2563value to the updelay bonding module option to delay the use of the
2564relevant interface(s).
2565
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002566Second, some switches may "bounce" the link state one or more
Linus Torvalds1da177e2005-04-16 15:20:36 -07002567times while a link is changing state. This occurs most commonly while
2568the switch is initializing. Again, an appropriate updelay value may
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002569help.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002570
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002571Note that when a bonding interface has no active links, the
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002572driver will immediately reuse the first link that goes up, even if the
2573updelay parameter has been specified (the updelay is ignored in this
2574case). If there are slave interfaces waiting for the updelay timeout
2575to expire, the interface that first went into that state will be
2576immediately reused. This reduces down time of the network if the
2577value of updelay has been overestimated, and since this occurs only in
2578cases with no connectivity, there is no additional penalty for
2579ignoring the updelay.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002580
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002581In addition to the concerns about switch timings, if your
Linus Torvalds1da177e2005-04-16 15:20:36 -07002582switches take a long time to go into backup mode, it may be desirable
2583to not activate a backup interface immediately after a link goes down.
2584Failover may be delayed via the downdelay bonding module option.
2585
Auke Kok6224e012006-06-08 11:15:35 -0700258613.2 Duplicated Incoming Packets
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002587--------------------------------
2588
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002589NOTE: Starting with version 3.0.2, the bonding driver has logic to
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002590suppress duplicate packets, which should largely eliminate this problem.
2591The following description is kept for reference.
2592
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002593It is not uncommon to observe a short burst of duplicated
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002594traffic when the bonding device is first used, or after it has been
2595idle for some period of time. This is most easily observed by issuing
2596a "ping" to some other host on the network, and noticing that the
2597output from ping flags duplicates (typically one per slave).
2598
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002599For example, on a bond in active-backup mode with five slaves
2600all connected to one switch, the output may appear as follows::
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002601
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002602 # ping -n 10.0.4.2
2603 PING 10.0.4.2 (10.0.4.2) from 10.0.3.10 : 56(84) bytes of data.
2604 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.7 ms
2605 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2606 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2607 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2608 64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2609 64 bytes from 10.0.4.2: icmp_seq=2 ttl=64 time=0.216 ms
2610 64 bytes from 10.0.4.2: icmp_seq=3 ttl=64 time=0.267 ms
2611 64 bytes from 10.0.4.2: icmp_seq=4 ttl=64 time=0.222 ms
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002612
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002613This is not due to an error in the bonding driver, rather, it
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002614is a side effect of how many switches update their MAC forwarding
2615tables. Initially, the switch does not associate the MAC address in
2616the packet with a particular switch port, and so it may send the
2617traffic to all ports until its MAC forwarding table is updated. Since
2618the interfaces attached to the bond may occupy multiple ports on a
2619single switch, when the switch (temporarily) floods the traffic to all
2620ports, the bond device receives multiple copies of the same packet
2621(one per slave device).
2622
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002623The duplicated packet behavior is switch dependent, some
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002624switches exhibit this, and some do not. On switches that display this
2625behavior, it can be induced by clearing the MAC forwarding table (on
2626most Cisco switches, the privileged command "clear mac address-table
2627dynamic" will accomplish this).
2628
Auke Kok6224e012006-06-08 11:15:35 -0700262914. Hardware Specific Considerations
Linus Torvalds1da177e2005-04-16 15:20:36 -07002630====================================
2631
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002632This section contains additional information for configuring
Linus Torvalds1da177e2005-04-16 15:20:36 -07002633bonding on specific hardware platforms, or for interfacing bonding
2634with particular switches or other devices.
2635
Auke Kok6224e012006-06-08 11:15:35 -0700263614.1 IBM BladeCenter
Linus Torvalds1da177e2005-04-16 15:20:36 -07002637--------------------
2638
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002639This applies to the JS20 and similar systems.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002640
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002641On the JS20 blades, the bonding driver supports only
Linus Torvalds1da177e2005-04-16 15:20:36 -07002642balance-rr, active-backup, balance-tlb and balance-alb modes. This is
2643largely due to the network topology inside the BladeCenter, detailed
2644below.
2645
2646JS20 network adapter information
2647--------------------------------
2648
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002649All JS20s come with two Broadcom Gigabit Ethernet ports
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002650integrated on the planar (that's "motherboard" in IBM-speak). In the
2651BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to
2652I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2.
2653An add-on Broadcom daughter card can be installed on a JS20 to provide
2654two more Gigabit Ethernet ports. These ports, eth2 and eth3, are
2655wired to I/O Modules 3 and 4, respectively.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002656
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002657Each I/O Module may contain either a switch or a passthrough
Linus Torvalds1da177e2005-04-16 15:20:36 -07002658module (which allows ports to be directly connected to an external
2659switch). Some bonding modes require a specific BladeCenter internal
2660network topology in order to function; these are detailed below.
2661
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002662Additional BladeCenter-specific networking information can be
Linus Torvalds1da177e2005-04-16 15:20:36 -07002663found in two IBM Redbooks (www.ibm.com/redbooks):
2664
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002665- "IBM eServer BladeCenter Networking Options"
2666- "IBM eServer BladeCenter Layer 2-7 Network Switching"
Linus Torvalds1da177e2005-04-16 15:20:36 -07002667
2668BladeCenter networking configuration
2669------------------------------------
2670
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002671Because a BladeCenter can be configured in a very large number
Linus Torvalds1da177e2005-04-16 15:20:36 -07002672of ways, this discussion will be confined to describing basic
2673configurations.
2674
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002675Normally, Ethernet Switch Modules (ESMs) are used in I/O
Linus Torvalds1da177e2005-04-16 15:20:36 -07002676modules 1 and 2. In this configuration, the eth0 and eth1 ports of a
2677JS20 will be connected to different internal switches (in the
2678respective I/O modules).
2679
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002680A passthrough module (OPM or CPM, optical or copper,
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002681passthrough module) connects the I/O module directly to an external
2682switch. By using PMs in I/O module #1 and #2, the eth0 and eth1
2683interfaces of a JS20 can be redirected to the outside world and
2684connected to a common external switch.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002685
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002686Depending upon the mix of ESMs and PMs, the network will
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002687appear to bonding as either a single switch topology (all PMs) or as a
2688multiple switch topology (one or more ESMs, zero or more PMs). It is
2689also possible to connect ESMs together, resulting in a configuration
2690much like the example in "High Availability in a Multiple Switch
2691Topology," above.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002692
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002693Requirements for specific modes
2694-------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002695
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002696The balance-rr mode requires the use of passthrough modules
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002697for devices in the bond, all connected to an common external switch.
2698That switch must be configured for "etherchannel" or "trunking" on the
Linus Torvalds1da177e2005-04-16 15:20:36 -07002699appropriate ports, as is usual for balance-rr.
2700
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002701The balance-alb and balance-tlb modes will function with
Linus Torvalds1da177e2005-04-16 15:20:36 -07002702either switch modules or passthrough modules (or a mix). The only
2703specific requirement for these modes is that all network interfaces
2704must be able to reach all destinations for traffic sent over the
2705bonding device (i.e., the network must converge at some point outside
2706the BladeCenter).
2707
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002708The active-backup mode has no additional requirements.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002709
2710Link monitoring issues
2711----------------------
2712
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002713When an Ethernet Switch Module is in place, only the ARP
Linus Torvalds1da177e2005-04-16 15:20:36 -07002714monitor will reliably detect link loss to an external switch. This is
2715nothing unusual, but examination of the BladeCenter cabinet would
2716suggest that the "external" network ports are the ethernet ports for
2717the system, when it fact there is a switch between these "external"
2718ports and the devices on the JS20 system itself. The MII monitor is
2719only able to detect link failures between the ESM and the JS20 system.
2720
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002721When a passthrough module is in place, the MII monitor does
Linus Torvalds1da177e2005-04-16 15:20:36 -07002722detect failures to the "external" port, which is then directly
2723connected to the JS20 system.
2724
2725Other concerns
2726--------------
2727
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002728The Serial Over LAN (SoL) link is established over the primary
Linus Torvalds1da177e2005-04-16 15:20:36 -07002729ethernet (eth0) only, therefore, any loss of link to eth0 will result
2730in losing your SoL connection. It will not fail over with other
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002731network traffic, as the SoL system is beyond the control of the
2732bonding driver.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002733
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002734It may be desirable to disable spanning tree on the switch
Linus Torvalds1da177e2005-04-16 15:20:36 -07002735(either the internal Ethernet Switch Module, or an external switch) to
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002736avoid fail-over delay issues when using bonding.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002737
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002738
Auke Kok6224e012006-06-08 11:15:35 -0700273915. Frequently Asked Questions
Linus Torvalds1da177e2005-04-16 15:20:36 -07002740==============================
2741
27421. Is it SMP safe?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002743-------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002744
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002745Yes. The old 2.0.xx channel bonding patch was not SMP safe.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002746The new driver was designed to be SMP safe from the start.
2747
27482. What type of cards will work with it?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002749-----------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002750
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002751Any Ethernet type cards (you can even mix cards - a Intel
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002752EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes,
2753devices need not be of the same speed.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002754
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002755Starting with version 3.2.1, bonding also supports Infiniband
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002756slaves in active-backup mode.
2757
Linus Torvalds1da177e2005-04-16 15:20:36 -070027583. How many bonding devices can I have?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002759----------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002760
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002761There is no limit.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002762
27634. How many slaves can a bonding device have?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002764----------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002765
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002766This is limited only by the number of network interfaces Linux
Linus Torvalds1da177e2005-04-16 15:20:36 -07002767supports and/or the number of network cards you can place in your
2768system.
2769
27705. What happens when a slave link dies?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002771----------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002772
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002773If link monitoring is enabled, then the failing device will be
Linus Torvalds1da177e2005-04-16 15:20:36 -07002774disabled. The active-backup mode will fail over to a backup link, and
2775other modes will ignore the failed link. The link will continue to be
2776monitored, and should it recover, it will rejoin the bond (in whatever
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002777manner is appropriate for the mode). See the sections on High
2778Availability and the documentation for each mode for additional
2779information.
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002780
2781Link monitoring can be enabled via either the miimon or
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002782arp_interval parameters (described in the module parameters section,
Linus Torvalds1da177e2005-04-16 15:20:36 -07002783above). In general, miimon monitors the carrier state as sensed by
2784the underlying network device, and the arp monitor (arp_interval)
2785monitors connectivity to another host on the local network.
2786
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002787If no link monitoring is configured, the bonding driver will
Linus Torvalds1da177e2005-04-16 15:20:36 -07002788be unable to detect link failures, and will assume that all links are
2789always available. This will likely result in lost packets, and a
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002790resulting degradation of performance. The precise performance loss
Linus Torvalds1da177e2005-04-16 15:20:36 -07002791depends upon the bonding mode and network configuration.
2792
27936. Can bonding be used for High Availability?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002794----------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002795
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002796Yes. See the section on High Availability for details.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002797
27987. Which switches/systems does it work with?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002799---------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002800
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002801The full answer to this depends upon the desired mode.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002802
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002803In the basic balance modes (balance-rr and balance-xor), it
Linus Torvalds1da177e2005-04-16 15:20:36 -07002804works with any system that supports etherchannel (also called
2805trunking). Most managed switches currently available have such
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002806support, and many unmanaged switches as well.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002807
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002808The advanced balance modes (balance-tlb and balance-alb) do
Linus Torvalds1da177e2005-04-16 15:20:36 -07002809not have special switch requirements, but do need device drivers that
2810support specific features (described in the appropriate section under
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002811module parameters, above).
Linus Torvalds1da177e2005-04-16 15:20:36 -07002812
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002813In 802.3ad mode, it works with systems that support IEEE
Linus Torvalds1da177e2005-04-16 15:20:36 -07002814802.3ad Dynamic Link Aggregation. Most managed and many unmanaged
2815switches currently available support 802.3ad.
2816
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002817The active-backup mode should work with any Layer-II switch.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002818
28198. Where does a bonding device get its MAC address from?
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002820---------------------------------------------------------
Linus Torvalds1da177e2005-04-16 15:20:36 -07002821
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002822When using slave devices that have fixed MAC addresses, or when
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002823the fail_over_mac option is enabled, the bonding device's MAC address is
2824the MAC address of the active slave.
2825
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002826For other configurations, if not explicitly configured (with
Jay Vosburgh9a6c6862007-11-13 20:25:48 -08002827ifconfig or ip link), the MAC address of the bonding device is taken from
2828its first slave device. This MAC address is then passed to all following
2829slaves and remains persistent (even if the first slave is removed) until
2830the bonding device is brought down or reconfigured.
Linus Torvalds1da177e2005-04-16 15:20:36 -07002831
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002832If you wish to change the MAC address, you can set it with
2833ifconfig or ip link::
Linus Torvalds1da177e2005-04-16 15:20:36 -07002834
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002835 # ifconfig bond0 hw ether 00:11:22:33:44:55
Linus Torvalds1da177e2005-04-16 15:20:36 -07002836
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002837 # ip link set bond0 address 66:77:88:99:aa:bb
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002838
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002839The MAC address can be also changed by bringing down/up the
2840device and then changing its slaves (or their order)::
Linus Torvalds1da177e2005-04-16 15:20:36 -07002841
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002842 # ifconfig bond0 down ; modprobe -r bonding
2843 # ifconfig bond0 .... up
2844 # ifenslave bond0 eth...
Linus Torvalds1da177e2005-04-16 15:20:36 -07002845
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002846This method will automatically take the address from the next
Linus Torvalds1da177e2005-04-16 15:20:36 -07002847slave that is added.
2848
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002849To restore your slaves' MAC addresses, you need to detach them
2850from the bond (``ifenslave -d bond0 eth0``). The bonding driver will
Linus Torvalds1da177e2005-04-16 15:20:36 -07002851then restore the MAC addresses that the slaves had before they were
2852enslaved.
2853
Jay Vosburgh00354cf2005-07-21 12:18:02 -0700285416. Resources and Links
Linus Torvalds1da177e2005-04-16 15:20:36 -07002855=======================
2856
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002857The latest version of the bonding driver can be found in the latest
Linus Torvalds1da177e2005-04-16 15:20:36 -07002858version of the linux kernel, found on http://kernel.org
2859
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002860The latest version of this document can be found in the latest kernel
2861source (named Documentation/networking/bonding.rst).
Jay Vosburgh00354cf2005-07-21 12:18:02 -07002862
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002863Discussions regarding the development of the bonding driver take place
Nicolas de Pesloüana23c37f12011-03-13 10:34:22 +00002864on the main Linux network mailing list, hosted at vger.kernel.org. The list
2865address is:
2866
2867netdev@vger.kernel.org
2868
Mauro Carvalho Chehaba3620322020-04-28 00:01:24 +02002869The administrative interface (to subscribe or unsubscribe) can
Nicolas de Pesloüana23c37f12011-03-13 10:34:22 +00002870be found at:
2871
2872http://vger.kernel.org/vger-lists.html#netdev