1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
|
.\" Copyright (c) 2007 Matthew Jacob
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\" $FreeBSD$
.\"
.Dd July 17, 2015
.Dt GMULTIPATH 8
.Os
.Sh NAME
.Nm gmultipath
.Nd "disk multipath control utility"
.Sh SYNOPSIS
.Nm
.Cm create
.Op Fl ARv
.Ar name
.Ar prov ...
.Nm
.Cm label
.Op Fl ARv
.Ar name
.Ar prov ...
.Nm
.Cm configure
.Op Fl APRv
.Ar name
.Nm
.Cm add
.Op Fl v
.Ar name prov
.Nm
.Cm remove
.Op Fl v
.Ar name prov
.Nm
.Cm fail
.Op Fl v
.Ar name prov
.Nm
.Cm restore
.Op Fl v
.Ar name prov
.Nm
.Cm rotate
.Op Fl v
.Ar name
.Nm
.Cm prefer
.Op Fl v
.Ar name
.Ar prov
.Nm
.Cm getactive
.Op Fl v
.Ar name
.Nm
.Cm destroy
.Op Fl v
.Ar name
.Nm
.Cm stop
.Op Fl v
.Ar name
.Nm
.Cm clear
.Op Fl v
.Ar prov ...
.Nm
.Cm list
.Nm
.Cm status
.Nm
.Cm load
.Nm
.Cm unload
.Sh DESCRIPTION
The
.Nm
utility is used for device multipath configuration.
.Pp
The multipath device can be configured using two different methods:
.Dq manual
or
.Dq automatic .
When using the
.Dq manual
method, no metadata are stored on the devices, so the multipath
device has to be configured by hand every time it is needed.
Additional device paths also won't be detected automatically.
The
.Dq automatic
method uses on-disk metadata to detect device and all it's paths.
Metadata use the last sector of the underlying disk device and
include device name and UUID.
The UUID guarantees uniqueness in a shared storage environment
but is in general too cumbersome to use.
The name is what is exported via the device interface.
.Pp
The first argument to
.Nm
indicates an action to be performed:
.Bl -tag -width ".Cm destroy"
.It Cm create
Create multipath device with
.Dq manual
method without writing any on-disk metadata.
It is up to administrator, how to properly identify device paths.
Kernel will only check that all given providers have same media and
sector sizes.
.Pp
.Fl A
option enables Active/Active mode,
.Fl R
option enables Active/Read mode, otherwise Active/Passive mode is used
by default.
.It Cm label
Create multipath device with
.Dq automatic
method.
Label the first given provider with on-disk metadata using the specified
.Ar name .
The rest of given providers will be retasted to detect these metadata.
It reliably protects against specifying unrelated providers.
Providers with no matching metadata detected will not be added to the device.
.Pp
.Fl A
option enables Active/Active mode,
.Fl R
option enables Active/Read mode, otherwise Active/Passive mode is used
by default.
.It Cm configure
Configure the given multipath device.
.Pp
.Fl A
option enables Active/Active mode,
.Fl P
option enables Active/Passive mode,
.Fl R
option enables Active/Read mode.
.It Cm add
Add the given provider as a path to the given multipath device.
Should normally be used only for devices created with
.Dq manual
method, unless you know what you are doing (you are sure that it is another
device path, but tasting its metadata in regular
.Dq automatic
way is not possible).
.It Cm remove
Remove the given provider as a path from the given multipath device.
If the last path removed, the multipath device will be destroyed.
.It Cm fail
Mark specified provider as a path of the specified multipath device as failed.
If there are other paths present, new requests will be forwarded there.
.It Cm restore
Mark specified provider as a path of the specified multipath device as
operational, allowing it to handle requests.
.It Cm rotate
Change the active provider/path to the next available provider in Active/Passive mode.
.It Cm prefer
Change the active provider/path to the specified provider in Active/Passive mode.
.It Cm getactive
Get the currently active provider(s)/path(s).
.It Cm destroy
Destroy the given multipath device clearing metadata.
.It Cm stop
Stop the given multipath device without clearing metadata.
.It Cm clear
Clear metadata on the given provider.
.It Cm list
See
.Xr geom 8 .
.It Cm status
See
.Xr geom 8 .
.It Cm load
See
.Xr geom 8 .
.It Cm unload
See
.Xr geom 8 .
.El
.Sh SYSCTL VARIABLES
The following
.Xr sysctl 8
variable can be used to control the behavior of the
.Nm MULTIPATH
GEOM class.
.Bl -tag -width indent
.It Va kern.geom.multipath.debug : No 0
Debug level of the
.Nm MULTIPATH
GEOM class.
This can be set to 0 (default) or 1 to disable or enable various
forms of chattiness.
.It Va kern.geom.multipath.exclusive : No 1
Open underlying providers exclusively, preventing individual paths access.
.El
.Sh EXIT STATUS
Exit status is 0 on success, and 1 if the command fails.
.Sh MULTIPATH ARCHITECTURE
This is a multiple path architecture with no device knowledge or
presumptions other than size matching built in.
Therefore the user must exercise some care
in selecting providers that do indeed represent multiple paths to the
same underlying disk device.
The reason for this is that there are several
criteria across multiple underlying transport types that can
.Ar indicate
identity, but in all respects such identity can rarely be considered
.Ar definitive .
.Pp
For example, if you use the World Word Port Name of a Fibre Channel
disk object you might believe that two disks that have the same WWPN
on different paths (or even disjoint fabrics) might be considered
the same disk.
Nearly always this would be a safe assumption, until
you realize that a WWPN, like an Ethernet MAC address, is a soft
programmable entity, and that a misconfigured Director Class switch
could lead you to believe incorrectly that you have found multiple
paths to the same device.
This is an extreme and theoretical case, but
it is possible enough to indicate that the policy for deciding which
of multiple pathnames refer to the same device should be left to the
system operator who will use tools and knowledge of their own storage
subsystem to make the correct configuration selection.
.Pp
There are Active/Passive, Active/Read and Active/Active operation modes
supported.
In Active/Passive mode only one path has I/O moving on it
at any point in time.
This I/O continues until an I/O is returned with
a generic I/O error or a "Nonexistent Device" error.
When this occurs, that path is marked FAIL, the next path
in a list is selected as active and the failed I/O reissued.
In Active/Active mode all paths not marked FAIL may handle I/O at the same time.
Requests are distributed between paths to equalize load.
For capable devices it allows to utilize the bandwidth of all paths.
In Active/Read mode all paths not marked FAIL may handle reads at the same time,
but unlike in Active/Active mode only one path handles write requests at any
point in time.
It allows to closer follow the original write request order if the layer above
needs it for data consistency (not waiting for requisite write completion
before sending dependent write).
.Pp
When new devices are added to the system the
.Nm MULTIPATH
GEOM class is given an opportunity to taste these new devices.
If a new
device has a
.Nm MULTIPATH
on-disk metadata label, the device is either used to create a new
.Nm MULTIPATH
GEOM, or added to the list of paths for an existing
.Nm MULTIPATH
GEOM.
.Pp
It is this mechanism that works reasonably with
.Xr isp 4
and
.Xr mpt 4
based Fibre Channel disk devices.
For these devices, when a device disappears
(due to e.g., a cable pull or power failure to a switch), the device is
proactively marked as gone and I/O to it failed.
This causes the
.Nm MULTIPATH
failure event just described.
.Pp
When Fibre Channel events inform either
.Xr isp 4
or
.Xr mpt 4
host bus adapters that new devices may have arrived (e.g., the arrival
of an RSCN event from the Fabric Domain Controller), they can cause
a rescan to occur and cause the attachment and configuration of any
(now) new devices to occur, causing the taste event described above.
.Pp
This means that this multipath architecture is not a one-shot path
failover, but can be considered to be steady state as long as failed
paths are repaired (automatically or otherwise).
.Pp
Automatic rescanning is not a requirement.
Nor is Fibre Channel.
The
same failover mechanisms work equally well for traditional "Parallel"
SCSI but may require manual intervention with
.Xr camcontrol 8
to cause the reattachment of repaired device links.
.Sh EXAMPLES
The following example shows how to use
.Xr camcontrol 8
to find possible multiple path devices and to create a
.Nm MULTIPATH
GEOM class for them.
.Bd -literal -offset indent
mysys# camcontrol devlist
<ECNCTX @WESTVILLE > at scbus0 target 0 lun 0 (da0,pass0)
<ECNCTX @WESTVILLE > at scbus0 target 0 lun 1 (da1,pass1)
<ECNCTX @WESTVILLE > at scbus1 target 0 lun 0 (da2,pass2)
<ECNCTX @WESTVILLE > at scbus1 target 0 lun 1 (da3,pass3)
mysys# camcontrol inquiry da0 -S
ECNTX0LUN000000SER10ac0d01
mysys# camcontrol inquiry da2 -S
ECNTX0LUN000000SER10ac0d01
.Ed
.Pp
Now that you have used the Serial Number to compare two disk paths
it is not entirely unreasonable to conclude that these are multiple
paths to the same device.
However, only the user who is familiar
with their storage is qualified to make this judgement.
.Pp
You can then use the
.Nm
command to label and create a
.Nm MULTIPATH
GEOM provider named
.Ar FRED .
.Bd -literal -offset indent
gmultipath label -v FRED /dev/da0 /dev/da2
disklabel -Brw /dev/multipath/FRED auto
newfs /dev/multipath/FREDa
mount /dev/multipath/FREDa /mnt....
.Ed
.Pp
The resultant console output looks something like:
.Bd -literal -offset indent
GEOM_MULTIPATH: da0 added to FRED
GEOM_MULTIPATH: da0 is now active path in FRED
GEOM_MULTIPATH: da2 added to FRED
.Ed
.Ed
.Pp
To load the
.Nm
Module at boot time, add the following entry to
.Pa /boot/loader.conf:
.Bd -literal -offset ident
geom_multipath_load="YES"
.Ed
.Ed
.Sh SEE ALSO
.Xr geom 4 ,
.Xr isp 4 ,
.Xr mpt 4 ,
.Xr loader.conf 5 ,
.Xr camcontrol 8 ,
.Xr geom 8 ,
.Xr mount 8 ,
.Xr newfs 8 ,
.Xr sysctl 8
.Sh AUTHORS
.An Matthew Jacob Aq Mt mjacob@FreeBSD.org
.An Alexander Motin Aq Mt mav@FreeBSD.org
|