1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
|
.\" $Id: fetch.1,v 1.25 1998/11/08 23:18:47 des Exp $
.Dd July 2, 1996
.Dt FETCH 1
.Os FreeBSD 2.2
.Sh NAME
.Nm fetch
.Nd retrieve a file by Uniform Resource Locator
.Sh SYNOPSIS
.Nm fetch
.Op Fl MPablmnpqrtv
.Op Fl S Ar size
.Op Fl T Ar timeout
.Op Fl o Ar file
.Ar URL
.Op Ar ...
.Nm fetch
.Op Fl MPRlmnpqrv
.Op Fl S Ar size
.Op Fl o Ar file
.Op Fl c Ar dir
.Fl f Ar file
.Fl h Ar host
.Sh DESCRIPTION
.Nm fetch
allows a user to transfer files from a remote network site using
either the
.Tn FTP
or the
.Tn HTTP
protocol. In the first form of the command, the
.Ar URL
may be of the form
.Li http://site.domain/path/to/the/file
or
.Li ftp://site.domain/path/to/the/file .
To denote a local filename to be copied or linked to (see the
.Fl l
flag below), the
.Em file:/path/to/the/file
URL form is used. See URL SYNTAX, below.
.Pp
The second form of the command can be used to get a file using the
.Tn FTP
protocol, specifying the file name and the remote host with the
.Fl h
and the
.Fl f
flags.
.Pp
The following options are available:
.Bl -tag -width Fl
.It Fl a
Automatically retry the transfer upon soft failures.
.It Fl b
Work around a bug in some
.Tn HTTP
servers which fail to correctly implement the
.Tn TCP
protocol.
.It Fl c Ar dir
The file to retrieve is in directory
.Ar dir
on the remote host.
.It Fl f Ar file
The file to retrieve is named
.Ar file
on the remote host.
.It Fl h Ar host
The file to retrieve is located on the host
.Ar host .
.It Fl l
If target is a
.Ar file:/
style of URL, make a link to the target rather than trying
to copy it.
.It Fl M
.It Fl m
Mirror mode: Set the modification time of the file so that it is
identical to the modification time of the file at the remote host.
If the file already exists on the local host and is identical (as
gauged by size and modification time), no transfer is done.
.It Fl n
Don't preserve the modtime of the transfered file, use the current time.
.It Fl o Ar file
Set the output file name to
.Ar file .
By default, a ``pathname'' is extracted from the specified URI, and
its basename is used as the name of the output file. A
.Ar file
argument of
.Sq Li \&-
indicates that results are to be directed to the standard output.
.It Fl P
.It Fl p
Use the passive mode of the
.Tn FTP
protocol. This is useful for crossing certain sorts of firewalls.
.It Fl q
Quiet mode. Do not report transfer progress on the terminal.
.It Fl R
The filenames specified are ``precious'', and should not be deleted
under any circumstances, even if the transfer failed or was incomplete.
.It Fl r
Restart a previously interrupted transfer.
.It Fl S Ar bytes
Require the file size reported by
.Tn FTP
or
.Tn HTTP
server to match the value specified with this option.
On mismatch, a message is printed and the file will not be fetched.
If the server does not support reporting of file sizes, the option
will be ignored and the file will be retrieved anyway.
This option is useful to prevent
.Nm fetch
from downloading a file that is either incomplete or the wrong version,
given the correct size of the file in advance.
.It Fl s
Ask server for size of file in bytes and print it to stdout. Do not
actually fetch the file.
.It Fl T Ar seconds
Set timeout value to
.Ar seconds.
Overrides the environment variables
.Ev FTP_TIMEOUT
for ftp transfers or
.Ev HTTP_TIMEOUT
for http transfers if set.
.It Fl t
Work around a different set of buggy
.Tn TCP
implementations.
.It Fl v
Increase verbosity. More
.Fl v Ns \&'s
result in more information.
.El
.Pp
Many options are also controlled solely by the environment (this is a
bug).
.Sh URL SYNTAX
.Nm
accepts
.Tn http
and
.Tn ftp
URL's, as described in RFC1738. For
.Tn ftp
URL's, a username and password may be specified, using the syntax
.Li ftp://user:password@host/.
If the path is to be absolute, as opposed to relative to the user's
home directory, it must start with %2F, as in
.Li ftp://root:mypass@localhost/%2Fetc/passwd .
.Sh PROXY SERVERS
Many sites use application gateways (``proxy servers'') in their
firewalls in order to allow communication across the firewall using a
trusted protocol. The
.Nm fetch
program can use both the
.Tn FTP
and the
.Tn HTTP
protocol with a proxy server.
.Tn FTP
proxy servers can only relay
.Tn FTP
requests;
.Tn HTTP
proxy servers can relay both
.Tn FTP
and
.Tn HTTP
requests.
A proxy server can be configured by defining an environment variable
named
.Dq Va PROTO Ns Ev _PROXY ,
where
.Va PROTO
is the name of the protocol in upper case. The value of the
environment variable specifies a hostname, optionally followed by a
colon and a port number.
.Pp
The
.Tn FTP
proxy client passes the remote username, host and port as the
.Tn FTP
session's username, in the form
.Do Va remoteuser Ns Li \&@ Ns Va remotehost
.Op Li \^@ Ns Va port
.Dc .
The
.Tn HTTP
proxy client simply passes the originally-requested URI to the remote
server in an
.Tn HTTP
.Dq Li GET
request. HTTP proxy authentication is not yet implemented.
.Sh HTTP AUTHENTICATION
The
.Tn HTTP
protocol includes support for various methods of authentication.
Currently, the
.Dq basic
method, which provides no security from packet-sniffing or
man-in-the-middle attacks, is the only method supported in
.Nm fetch .
Authentication is enabled by the
.Ev HTTP_AUTH
and
.Ev HTTP_PROXY_AUTH
environment variables. Both variables have the same format, which
consists of space-separated list of parameter settings, where each
setting consists of a colon-separated list of parameters. The first
two parameters are always the (case-insensitive) authentication scheme
name and the realm in which authentication is to be performed. If the
realm is specified as
.Sq Li \&* ,
then it will match all realms not specified otherwise.
.Pp
The
.Li basic
authentication scheme uses two additional optional parameters; the
first is a user name, and the second is the password associated with
it. If either the password or both parameters are not specified in
the environment, and the standard input of
.Nm
is connected to a terminal, then
.Nm
will prompt the user to enter the missing parameters. Thus, if the
user is known as
.Dq Li jane
in the
.Dq Li WallyWorld
realm, and has a password of
.Dq Li QghiLx79
there, then she might set her
.Ev HTTP_AUTH
variable to:
.Bl -enum -offset indent
.It
.Dq Li basic:WallyWorld:jane:QghiLx79
.It
.Dq Li basic:WallyWorld:jane ,
or
.It
.Dq Li basic:WallyWorld
.El
.Pp
and
.Nm
will prompt for the missing information if it is required. She might
also specify a realm of
.Dq Li \&*
instead of
.Dq Li WallyWorld
to indicate that the parameters can be applied to any realm. (This is
most commonly used in a construction such as
.Dq Li basic:* ,
which indicates to
.Nm
that it may offer to do
.Li basic
authentication for any realm.
.Sh ERRORS
The
.Nm
command returns zero on success, or a non-zero value from
.Aq Pa sysexits.h
on failure. If multiple URIs are given for retrieval,
.Nm
will attempt all of them and return zero only if all succeeded
(otherwise it will return the error from the last failure).
.Sh ENVIRONMENT
.Bl -tag -width FTP_PASSIVE_MODE -offset indent
.It Ev FTP_TIMEOUT
maximum time, in seconds, to wait before aborting an
.Tn FTP
connection.
.It Ev FTP_LOGIN
the login name used for
.Tn FTP
transfers (default
.Dq Li anonymous )
.It Ev FTP_PASSIVE_MODE
force the use of passive mode FTP
.It Ev FTP_PASSWORD
the password used for
.Tn FTP
transfers (default
.Dq Va yourname Ns Li \&@ Ns Va yourhost )
.It Ev FTP_PROXY
the address (in the form
.Do Va hostname Ns
.Op Li : Ns Va port
.Dc )
of a proxy server which understands
.Tn FTP
.It Ev HTTP_AUTH
defines authentication parameters for
.Tn HTTP
.It Ev HTTP_PROXY
the address (in the form
.Do Va hostname Ns
.Op Li : Ns Va port
.Dc )
of a proxy server which understands
.Tn HTTP
.It Ev HTTP_PROXY_AUTH
defines authentication parameters for
.Tn HTTP
proxy servers
.It Ev HTTP_TIMEOUT
maximum time, in seconds, to wait before aborting an
.Tn HTTP
connection.
.Sh SEE ALSO
.Xr ftp 1 ,
.Xr tftp 1
.Sh HISTORY
The
.Nm fetch
command appeared in
.Fx 2.1.5 .
.Sh AUTHORS
The original implementation of
.Nm
was done by
.An Jean-Marc Zucconi .
It was extensively re-worked for
.Fx 2.2
by
.An Garrett Wollman .
.Sh BUGS
There are too many environment variables and command-line options.
.Pp
The
.Fl a
option is only implemented for certain kinds of
.Tn HTTP
failures, and no
.Tn FTP
failures.
.Pp
Only the
.Dq basic
authentication mode is implemented for
.Tn HTTP .
This should be replaced by digest authentication.
.Pp
Some
.Tn TCP
implementations (other than
.Tn FreeBSD )
fail to correctly implement cases where the
.Dv SYN
and/or
.Dv FIN
control flags are specified in packets which also contain data.
The
.Sq Fl t
flag works around the latter deficiency and the
.Sq Fl b
flag works around the former. Since these are errors of the server's
.Tn TCP
stack, the best we can do is provide these workarounds. Given a correct
server, an optimal
.Tn HTTP
transfer without
.Fl t
and
.Fl b
involves a minimum of two round trips (for small replies), one less than
other implementations.
|