From 9b9369d90eea0c073368daad0cf0e48083929081 Mon Sep 17 00:00:00 2001 From: des Date: Wed, 28 Jun 2000 16:55:15 +0000 Subject: New libfetch-based fetch. --- usr.bin/fetch/fetch.1 | 499 +++++++++++++++----------------------------------- 1 file changed, 147 insertions(+), 352 deletions(-) (limited to 'usr.bin/fetch/fetch.1') diff --git a/usr.bin/fetch/fetch.1 b/usr.bin/fetch/fetch.1 index b2b2ea0..8f59f68 100644 --- a/usr.bin/fetch/fetch.1 +++ b/usr.bin/fetch/fetch.1 @@ -1,378 +1,209 @@ -.\" $FreeBSD$ -.Dd February 22, 1999 +.\"- +.\" Copyright (c) 2000 Dag-Erling Coïdan Smørgrav +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer +.\" in this position and unchanged. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" 3. The name of the author may not be used to endorse or promote products +.\" derived from this software without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR +.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +.\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, +.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT +.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd June 28, 2000 .Dt FETCH 1 -.Os FreeBSD 4.0 +.Os .Sh NAME .Nm fetch .Nd retrieve a file by Uniform Resource Locator .Sh SYNOPSIS -.Nm fetch -.Op Fl AFMPablmnpqrtv -.Op Fl S Ar size -.Op Fl T Ar timeout +.Nm +.Op Fl 146AFHMPRalmnpqrsv +.Op Fl B Ar bytes +.Op Fl S Ar bytes +.Op Fl T Ar seconds .Op Fl o Ar file +.Op Fl w Ar seconds .Ar URL .Op Ar ... -.Nm fetch -.Op Fl FMPRlmnpqrv -.Op Fl S Ar size -.Op Fl o Ar file -.Op Fl c Ar dir -.Fl f Ar file -.Fl h Ar host .Sh DESCRIPTION -.Nm fetch -allows a user to transfer files from a remote network site using -either the -.Tn FTP -or the -.Tn HTTP -protocol. -In the first form of the command, the -.Ar URL -may be of the form -.Li http://site.domain/path/to/the/file -or -.Li ftp://site.domain/path/to/the/file . -To denote a local filename to be copied or linked to (see the -.Fl l -flag below), the -.Em file:/path/to/the/file -URL form is used. See URL SYNTAX, below. -.Pp -The second form of the command can be used to get a file using the -.Tn FTP -protocol, specifying the file name and the remote host with the -.Fl h -and the -.Fl f -flags. +.Nm Fetch +provides a command-line interface to the +.Xr fetch 3 +library. +Its purpose is to retrieve the file(s) pointed to by the URL(s) on the +command line. .Pp The following options are available: .Bl -tag -width Fl +.It Fl \&1 +Stop and return exit code 0 at the first successfully retrieved file. +.It Fl 4 +Forces +.Nm +to use IPv4 addresses only. +.It Fl 6 +Forces +.Nm +to use IPv6 addresses only. .It Fl A -Do not automatically follow ``temporary'' (302) redirects. Some -broken Web sites will return a redirect instead of a not-found error -when the requested object does not exist. +Do not automatically follow ``temporary'' (302) redirects. +Some broken Web sites will return a redirect instead of a not-found +error when the requested object does not exist. .It Fl a Automatically retry the transfer upon soft failures. -.It Fl b -Work around a bug in some -.Tn HTTP -servers which fail to correctly implement the -.Tn TCP -protocol. +.It Fl B Ar bytes +Specify the read buffer size in bytes. +The default is 4096 bytes. +Attempts to set a buffer size lower than this will be silently +ignored. +The number of reads actually performed is reported at verbosity level +two or higher (see the +.Fl v +flag). .It Fl c Ar dir The file to retrieve is in directory .Ar dir on the remote host. .It Fl F -Force restart without checking for the local file's date matching -that of the remote file. Use this with +In combination with the .Fl r -to restart a transfer of a partial file where the modification -time on the local file has been changed and you are sure that the -remote file is still the same, as this will prevent retrieval from -starting anew. +flag, forces a restart even if the local and remote files have +different modification times. .It Fl f Ar file The file to retrieve is named .Ar file on the remote host. +.It Fl H +When using passive FTP, allocate a high port for the data connection. +See +.Xr ip 4 +for details on how to specify which port range this corresponds to. .It Fl h Ar host The file to retrieve is located on the host .Ar host . .It Fl l -If target is a -.Ar file:/ -style of URL, make a link to the target rather than trying -to copy it. +If the target is a file-scheme URL, make a symbolic link to the target +rather than trying to copy it. .It Fl M .It Fl m -Mirror mode: Set the modification time of the file so that it is -identical to the modification time of the file at the remote host. -If the file already exists on the local host and is identical (as -gauged by size and modification time), no transfer is done. +Mirror mode: set the modification time of the local file to that of +the remote file. +If the file already exists locally and has the same size and +modification time as the remote file, it will not be fetched. .It Fl n -Don't preserve the modtime of the transferred file, use the current time. +Don't preserve the modtime of the transfered file, use the current +time. .It Fl o Ar file Set the output file name to .Ar file . By default, a ``pathname'' is extracted from the specified URI, and -its basename is used as the name of the output file. A +its basename is used as the name of the output file. +A .Ar file argument of .Sq Li \&- indicates that results are to be directed to the standard output. .It Fl P .It Fl p -Use the passive mode of the -.Tn FTP -protocol. This is useful for crossing certain sorts of firewalls. +Use passive FTP. +This is useful if you are behind a firewall which blocks incoming +connections. +Try this flag if +.Nm +seems to hang when retrieving FTP URLs. .It Fl q Quiet mode. -Do not report transfer progress on the terminal. +This works by setting the verbosity level to zero; see the +.Fl v +option. .It Fl R -The filenames specified are ``precious'', and should not be deleted -under any circumstances, even if the transfer failed or was incomplete. +The output files are precious, and should not be deleted under any +circumstances, even if the transfer failed or was incomplete. .It Fl r Restart a previously interrupted transfer. .It Fl S Ar bytes -Require the file size reported by -.Tn FTP -or -.Tn HTTP -server to match the value specified with this option. -On mismatch, a message is printed and the file will not be fetched. -If the server does not support reporting of file sizes, the option -will be ignored and the file will be retrieved anyway. -This option is useful to prevent -.Nm fetch -from downloading a file that is either incomplete or the wrong version, -given the correct size of the file in advance. +Require the file size reported by the server to match the specified +value. +If it does not, a message is printed and the file is not fetched. +If the server does not support reporting file sizes, this option is +ignored and the file is fetched unconditionally. .It Fl s -Ask server for size of file in bytes and print it to stdout. -Do not -actually fetch the file. +Print the size in bytes of each requested file, without fetching it. .It Fl T Ar seconds Set timeout value to .Ar seconds. Overrides the environment variables .Ev FTP_TIMEOUT -for ftp transfers or +for FTP transfers or .Ev HTTP_TIMEOUT -for http transfers if set. +for HTTP transfers if set. .It Fl t Work around a different set of buggy .Tn TCP implementations. .It Fl v -Increase verbosity. More -.Fl v Ns \&'s -result in more information. -.El -.Pp -Many options are also controlled solely by the environment (this is a -bug). -.Sh URL SYNTAX -.Nm -accepts -.Tn http -and -.Tn ftp -URL's, as described in RFC1738. For -.Tn ftp -URL's, a username and password may be specified, using the syntax -.Li ftp://user:password@host/. -If the path is to be absolute, as opposed to relative to the user's -home directory, it must start with %2F, as in -.Li ftp://root:mypass@localhost/%2Fetc/passwd . -.Nm Fetch -condenses multiple slashes in an -.Tn ftp -URL into a single slash; literal multiple slashes translate to an -.Tn ftp -protocol error. -.Sh PROXY SERVERS -Many sites use application gateways (``proxy servers'') in their -firewalls in order to allow communication across the firewall using a -trusted protocol. The -.Nm fetch -program can use both the -.Tn FTP -and the -.Tn HTTP -protocol with a proxy server. -.Tn FTP -proxy servers can only relay -.Tn FTP -requests; -.Tn HTTP -proxy servers can relay both -.Tn FTP -and -.Tn HTTP -requests. -A proxy server can be configured by defining an environment variable -named -.Dq Va PROTO Ns Ev _PROXY , -where -.Va PROTO -is the name of the protocol in upper case. The value of the -environment variable specifies a hostname, optionally followed by a -colon and a port number. -.Pp -The -.Tn FTP -proxy client passes the remote username, host and port as the -.Tn FTP -session's username, in the form -.Do Va remoteuser Ns Li \&@ Ns Va remotehost -.Op Li \&@ Ns Va port -.Dc . -The -.Tn HTTP -proxy client simply passes the originally-requested URI to the remote -server in an -.Tn HTTP -.Dq Li GET -request. HTTP proxy authentication is not yet implemented. -.Sh HTTP AUTHENTICATION -The -.Tn HTTP -protocol includes support for various methods of authentication. -Currently, the -.Dq basic -method, which provides no security from packet-sniffing or -man-in-the-middle attacks, is the only method supported in -.Nm fetch . -Authentication is enabled by the -.Ev HTTP_AUTH -and -.Ev HTTP_PROXY_AUTH -environment variables. Both variables have the same format, which -consists of space-separated list of parameter settings, where each -setting consists of a colon-separated list of parameters. The first -two parameters are always the (case-insensitive) authentication scheme -name and the realm in which authentication is to be performed. If the -realm is specified as -.Sq Li \&* , -then it will match all realms not specified otherwise. -.Pp -The -.Li basic -authentication scheme uses two additional optional parameters; the -first is a user name, and the second is the password associated with -it. If either the password or both parameters are not specified in -the environment, and the standard input of -.Nm -is connected to a terminal, then -.Nm -will prompt the user to enter the missing parameters. Thus, if the -user is known as -.Dq Li jane -in the -.Dq Li WallyWorld -realm, and has a password of -.Dq Li QghiLx79 -there, then she might set her -.Ev HTTP_AUTH -variable to: -.Bl -enum -offset indent -.It -.Dq Li basic:WallyWorld:jane:QghiLx79 -.It -.Dq Li basic:WallyWorld:jane , -or -.It -.Dq Li basic:WallyWorld +Each instance of this flag increases the verbosity level by one. +Level one (the default) only gives a summary after each file; level +two show a running count during the transfer, provided that the +standard output goes to a terminal; level three enables messages from +the +.Xr fetch 3 +library. +.It Fl w Ar seconds +When the +.Fl a +flag is specified, wait this many seconds between successive retries. .El -.Pp -and -.Nm -will prompt for any missing information when it is required. She might -also specify a realm of -.Dq Li \&* -instead of -.Dq Li WallyWorld -to indicate that the parameters can be applied to any realm. (This is -most commonly used in a construction such as -.Dq Li basic:* , -which indicates to -.Nm -that it may offer to do -.Li basic -authentication for any realm. -.Sh ERRORS +.Sh DIAGNOSTICS The .Nm -command returns zero on success, or a non-zero value from -.Aq Pa sysexits.h -on failure. If multiple URIs are given for retrieval, +command returns zero on success, or one on failure. +If multiple URLs are listed on the command line, .Nm -will attempt all of them and return zero only if all succeeded -(otherwise it will return the error from the last failure). +will attempt to retrieve them each of them in turn, and return zero +only if they were all successfully retrieved. .Sh ENVIRONMENT .Bl -tag -width FTP_PASSIVE_MODE -offset indent .It Ev FTP_TIMEOUT maximum time, in seconds, to wait before aborting an .Tn FTP connection. -.It Ev FTP_LOGIN -the login name used for -.Tn FTP -transfers (default -.Dq Li anonymous ) -.It Ev FTP_PASSIVE_MODE -force the use of passive mode FTP -.It Ev FTP_PASSWORD -the password used for -.Tn FTP -transfers (default -.Dq Va yourname Ns Li \&@ Ns Va yourhost ) -.It Ev FTP_PROXY -the address (in the form -.Do Va hostname Ns -.Op Li : Ns Va port -.Dc ) -of a proxy server which understands -.Tn FTP -.It Ev HTTP_AUTH -defines authentication parameters for -.Tn HTTP -.It Ev HTTP_PROXY -the address (in the form -.Do Va hostname Ns -.Op Li : Ns Va port -.Dc ) -of a proxy server which understands -.Tn HTTP -.It Ev HTTP_PROXY_AUTH -defines authentication parameters for -.Tn HTTP -proxy servers .It Ev HTTP_TIMEOUT maximum time, in seconds, to wait before aborting an .Tn HTTP connection. +.El +.Pp +All environment variables mentioned in the documentation for the +.Xr fetch 3 +library are supported. .Sh SEE ALSO -.Xr ftp 1 , -.Xr tftp 1 -.Rs -.%A R. Fielding -.%A J. Gettys -.%A J. Mogul -.%A H. Frystyk -.%A T. Berners-Lee -.%T "Hypertext Transfer Protocol \-\- HTTP/1.1" -.%O RFC 2068 -.%D January 1997 -.Re -.Rs -.%A T. Berners-Lee -.%A L. Masinter -.%A M. McCahill -.%T "Uniform Resource Locators (URL)" -.%O RFC 1738 -.%D December 1994 -.Re -.Rs -.%A J. Postel -.%A J.K. Reynolds -.%T "File Transfer Protocol" -.%O RFC 959 / STD 9 -.%D October 1985 -.Re -.Rs -.%A M.R. Horton -.%T "Standard for interchange of USENET messages." -.%O RFC 850 -.%D June 1983 -.Re +.Xr fetch 3 .Sh HISTORY The -.Nm fetch +.Nm command appeared in .Fx 2.1.5 . +This implementation first appeared in +.Fx 4.1 . .Sh AUTHORS The original implementation of .Nm @@ -381,63 +212,27 @@ was done by It was extensively re-worked for .Fx 2.2 by -.An Garrett Wollman . -.Sh BUGS -There are too many environment variables and command-line options. -.Pp +.An Garrett Wollman , +and later completely rewritten to use the +.Xr fetch 3 +library by +.An Dag-Erling Smørgrav . +.Sh NOTES The -.Fl a -option is only implemented for certain kinds of -.Tn HTTP -failures, and no -.Tn FTP -failures. -.Pp -Only the -.Dq basic -authentication mode is implemented for -.Tn HTTP . -This should be replaced by digest authentication. -.Pp -Some -.Tn TCP -implementations (other than -.Tn FreeBSD ) -fail to correctly implement cases where the -.Dv SYN -and/or -.Dv FIN -control flags are specified in packets which also contain data. -The -.Sq Fl t -flag works around the latter deficiency and the -.Sq Fl b -flag works around the former. Since these are errors of the server's -.Tn TCP -stack, the best we can do is provide these workarounds. Given a correct -server, an optimal -.Tn HTTP -transfer without -.Fl t -and .Fl b -involves a minimum of two round trips (for small replies), one less than -other implementations. +and +.Fl t +options are no longer supported and will generate warnings. +They were workarounds for bugs in other OSes which this implementation +does not trigger. .Pp The -.Tn HTTP -standard requires interpretation of the -.Tn RFC 850 -date format, which does not provide a century indication. Versions of -.Nm fetch -prior to -.Fx 3.1 -would interpret all such dates as being in the 1900s. This version of -.Nm fetch -interprets such dates according to the rule given in -.Tn RFC 2068 : -.Bd -literal -offset indent - o HTTP/1.1 clients and caches should assume that an RFC-850 date - which appears to be more than 50 years in the future is in fact - in the past (this helps solve the "year 2000" problem). -.Ed +.Fl f +and +.Fl h +options (used for specifying an file to fetch and a host to fetch +from) are no longer supported and will generate errors. +Use URLs. +RFC1738 is your friend. +.Xr fetch 3 +library. -- cgit v1.1