diff options
Diffstat (limited to 'contrib/awk/doc/gawkinet.texi')
| -rw-r--r-- | contrib/awk/doc/gawkinet.texi | 5075 |
1 files changed, 0 insertions, 5075 deletions
diff --git a/contrib/awk/doc/gawkinet.texi b/contrib/awk/doc/gawkinet.texi deleted file mode 100644 index 2ffb581..0000000 --- a/contrib/awk/doc/gawkinet.texi +++ /dev/null @@ -1,5075 +0,0 @@ -\input texinfo @c -*-texinfo-*- -@c %**start of header (This is for running Texinfo on a region.) -@setfilename gawkinet.info -@settitle TCP/IP Internetworking With @command{gawk} -@c %**end of header (This is for running Texinfo on a region.) - -@c inside ifinfo for older versions of texinfo.tex -@ifinfo -@dircategory GNU Packages -@direntry -* Gawkinet: (gawkinet). TCP/IP Internetworking With @command{gawk}. -@end direntry -@end ifinfo - -@iftex -@set DOCUMENT book -@set CHAPTER chapter -@set SECTION section -@set DARKCORNER @inmargin{@image{lflashlight,1cm}, @image{rflashlight,1cm}} -@end iftex -@ifinfo -@set DOCUMENT Info file -@set CHAPTER major node -@set SECTION node -@set DARKCORNER (d.c.) -@end ifinfo -@ifhtml -@set DOCUMENT web page -@set CHAPTER chapter -@set SECTION section -@set DARKCORNER (d.c.) -@end ifhtml - -@set FSF - -@set FN file name -@set FFN File Name - -@c merge the function and variable indexes into the concept index -@ifinfo -@synindex fn cp -@synindex vr cp -@end ifinfo -@iftex -@syncodeindex fn cp -@syncodeindex vr cp -@end iftex - -@c If "finalout" is commented out, the printed output will show -@c black boxes that mark lines that are too long. Thus, it is -@c unwise to comment it out when running a master in case there are -@c overfulls which are deemed okay. - -@iftex -@finalout -@end iftex - -@smallbook - -@c Special files are described in chapter 6 Printing Output under -@c 6.7 Special File Names in gawk. I think the networking does not -@c fit into that chapter, thus this separate document. At over 50 -@c pages, I think this is the right decision. ADR. - -@set TITLE TCP/IP Internetworking With @command{gawk} -@set EDITION 1.1 -@set UPDATE-MONTH March, 2001 -@c gawk versions: -@set VERSION 3.1 -@set PATCHLEVEL 0 - -@ifinfo -This file documents the networking features in GNU @command{awk}. - -This is Edition @value{EDITION} of @cite{@value{TITLE}}, -for the @value{VERSION}.@value{PATCHLEVEL} (or later) version of the GNU -implementation of AWK. - -Copyright (C) 2000, 2001 Free Software Foundation, Inc. - -Permission is granted to copy, distribute and/or modify this document -under the terms of the GNU Free Documentation License, Version 1.1 or -any later version published by the Free Software Foundation; with the -Invariant Sections being ``GNU General Public License'', the Front-Cover -texts being (a) (see below), and with the Back-Cover Texts being (b) -(see below). A copy of the license is included in the section entitled -``GNU Free Documentation License''. - -@enumerate a -@item -``A GNU Manual'' - -@item -``You have freedom to copy and modify this GNU Manual, like GNU -software. Copies published by the Free Software Foundation raise -funds for GNU development.'' -@end enumerate -@end ifinfo - -@setchapternewpage odd - -@titlepage -@title @value{TITLE} -@subtitle Edition @value{EDITION} -@subtitle @value{UPDATE-MONTH} -@author J@"urgen Kahrs -@author with Arnold D. Robbins - -@c Include the Distribution inside the titlepage environment so -@c that headings are turned off. Headings on and off do not work. - -@page -@vskip 0pt plus 1filll -Copyright @copyright{} 2000, 2001 Free Software Foundation, Inc. -@sp 1 -@b{User Friendly} Copyright @copyright{} 2000 J.D.@: ``Iliad'' Frazier. -Reprinted by permission. -@sp 2 - -This is Edition @value{EDITION} of @cite{@value{TITLE}}, -for the @value{VERSION}.@value{PATCHLEVEL} (or later) version of the GNU -implementation of AWK. - -@sp 2 -Published by: -@sp 1 - -Free Software Foundation @* -59 Temple Place --- Suite 330 @* -Boston, MA 02111-1307 USA @* -Phone: +1-617-542-5942 @* -Fax: +1-617-542-2652 @* -Email: @email{gnu@@gnu.org} @* -URL: @uref{http://www.gnu.org/} @* - -ISBN 1-882114-93-0 @* - -Permission is granted to copy, distribute and/or modify this document -under the terms of the GNU Free Documentation License, Version 1.1 or -any later version published by the Free Software Foundation; with the -Invariant Sections being ``GNU General Public License'', the Front-Cover -texts being (a) (see below), and with the Back-Cover Texts being (b) -(see below). A copy of the license is included in the section entitled -``GNU Free Documentation License''. - -@enumerate a -@item -``A GNU Manual'' - -@item -``You have freedom to copy and modify this GNU Manual, like GNU -software. Copies published by the Free Software Foundation raise -funds for GNU development.'' -@end enumerate -@c @sp 2 -@c Cover art by ?????. -@end titlepage - -@iftex -@headings off -@evenheading @thispage@ @ @ @strong{@value{TITLE}} @| @| -@oddheading @| @| @strong{@thischapter}@ @ @ @thispage -@end iftex - -@ifinfo -@node Top, Preface, (dir), (dir) -@top General Introduction -@comment node-name, next, previous, up - -This file documents the networking features in GNU Awk (@command{gawk}) -version 3.1 and later. -@end ifinfo - -@menu -* Preface:: About this document. -* Introduction:: About networkiing. -* Using Networking:: Some examples. -* Some Applications and Techniques:: More extended examples. -* Links:: Where to find the stuff mentioned in this - document. -* GNU Free Documentation License:: The license for this document. -* Index:: The index. - -@detailmenu -* Stream Communications:: Sending data streams. -* Datagram Communications:: Sending self-contained messages. -* The TCP/IP Protocols:: How these models work in the Internet. -* Basic Protocols:: The basic protocols. -* Ports:: The idea behind ports. -* Making Connections:: Making TCP/IP connections. -* Gawk Special Files:: How to do @command{gawk} networking. -* Special File Fields:: The fields in the special file name. -* Comparing Protocols:: Differences between the protocols. -* File /inet/tcp:: The TCP special file. -* File /inet/udp:: The UDB special file. -* File /inet/raw:: The RAW special file. -* TCP Connecting:: Making a TCP connection. -* Troubleshooting:: Troubleshooting TCP/IP connections. -* Interacting:: Interacting with a service. -* Setting Up:: Setting up a service. -* Email:: Reading email. -* Web page:: Reading a Web page. -* Primitive Service:: A primitive Web service. -* Interacting Service:: A Web service with interaction. -* CGI Lib:: A simple CGI library. -* Simple Server:: A simple Web server. -* Caveats:: Network programming caveats. -* Challenges:: Where to go from here. -* PANIC:: An Emergency Web Server. -* GETURL:: Retrieving Web Pages. -* REMCONF:: Remote Configuration Of Embedded Systems. -* URLCHK:: Look For Changed Web Pages. -* WEBGRAB:: Extract Links From A Page. -* STATIST:: Graphing A Statistical Distribution. -* MAZE:: Walking Through A Maze In Virtual Reality. -* MOBAGWHO:: A Simple Mobile Agent. -* STOXPRED:: Stock Market Prediction As A Service. -* PROTBASE:: Searching Through A Protein Database. -@end detailmenu -@end menu - -@contents - -@node Preface, Introduction, Top, Top -@unnumbered Preface - -In May of 1997, J@"urgen Kahrs felt the need for network access -from @command{awk}, and, with a little help from me, set about adding -features to do this for @command{gawk}. At that time, he -wrote the bulk of this @value{DOCUMENT}. - -The code and documentation were added to the @command{gawk} 3.1 development -tree, and languished somewhat until I could finally get -down to some serious work on that version of @command{gawk}. -This finally happened in the middle of 2000. - -Meantime, J@"urgen wrote an article about the Internet special -files and @samp{|&} operator for @cite{Linux Journal}, and made a -networking patch for the production versions of @command{gawk} -available from his home page. -In August of 2000 (for @command{gawk} 3.0.6), this patch -also made it to the main GNU @command{ftp} distribution site. - -For release with @command{gawk}, I edited J@"urgen's prose -for English grammar and style, as he is not a native English -speaker. I also -rearranged the material somewhat for what I felt was a better order of -presentation, and (re)wrote some of the introductory material. - -The majority of this document and the code are his work, and the -high quality and interesting ideas speak for themselves. It is my -hope that these features will be of significant value to the @command{awk} -community. - -@sp 1 -@noindent -Arnold Robbins @* -Nof Ayalon, ISRAEL @* -March, 2001 - -@node Introduction, Using Networking, Preface, Top -@chapter Networking Concepts - -This @value{CHAPTER} provides a (necessarily) brief intoduction to -computer networking concepts. For many applications of @command{gawk} -to TCP/IP networking, we hope that this is enough. For more -advanced tasks, you will need deeper background, and it may be necessary -to switch to lower-level programming in C or C++. - -There are two real-life models for the way computers send messages -to each other over a network. While the analogies are not perfect, -they are close enough to convey the major concepts. -These two models are the phone system (reliable byte-stream communications), -and the postal system (best-effort datagrams). - -@menu -* Stream Communications:: Sending data streams. -* Datagram Communications:: Sending self-contained messages. -* The TCP/IP Protocols:: How these models work in the Internet. -* Making Connections:: Making TCP/IP connections. -@end menu - -@node Stream Communications, Datagram Communications, Introduction, Introduction -@section Reliable Byte-streams (Phone Calls) - -When you make a phone call, the following steps occur: - -@enumerate -@item -You dial a number. - -@item -The phone system connects to the called party, telling -them there is an incoming call. (Their phone rings.) - -@item -The other party answers the call, or, in the case of a -computer network, refuses to answer the call. - -@item -Assuming the other party answers, the connection between -you is now a @dfn{duplex} (two-way), @dfn{reliable} (no data lost), -sequenced (data comes out in the order sent) data stream. - -@item -You and your friend may now talk freely, with the phone system -moving the data (your voices) from one end to the other. -From your point of view, you have a direct end-to-end -connection with the person on the other end. -@end enumerate - -The same steps occur in a duplex reliable computer networking connection. -There is considerably more overhead in setting up the communications, -but once it's done, data moves in both directions, reliably, in sequence. - -@node Datagram Communications, The TCP/IP Protocols, Stream Communications, Introduction -@section Best-effort Datagrams (Mailed Letters) - -Suppose you mail three different documents to your office on the -other side of the country on two different days. Doing so -entails the following. - -@enumerate -@item -Each document travels in its own envelope. - -@item -Each envelope contains both the sender and the -recipient address. - -@item -Each envelope may travel a different route to its destination. - -@item -The envelopes may arrive in a different order from the one -in which they were sent. - -@item -One or more may get lost in the mail. -(Although, fortunately, this does not occur very often.) - -@item -In a computer network, one or more @dfn{packets} -may also arrive multiple times. (This doesn't happen -with the postal system!) - -@end enumerate - -The important characteristics of datagram communications, like -those of the postal system are thus: - -@itemize @bullet -@item -Delivery is ``best effort;'' the data may never get there. - -@item -Each message is self-contained, including the source and -destination addresses. - -@item -Delivery is @emph{not} sequenced; packets may arrive out -of order, and/or multiple times. - -@item -Unlike the phone system, overhead is considerably lower. -It is not necessary to set up the call first. -@end itemize - -The price the user pays for the lower overhead of datagram communications -is exactly the lower reliability; it is often necessary for user-level -protocols that use datagram communications to add their own reliabilty -features on top of the basic communications. - -@node The TCP/IP Protocols, Making Connections, Datagram Communications, Introduction -@section The Internet Protocols - -The Internet Protocol Suite (usually referred as just TCP/IP)@footnote{ -It should be noted that although the Internet seems to have conquered the -world, there are other networking protocol suites in existence and in use.} -consists of a number of different protocols at different levels or ``layers.'' -For our purposes, three protocols provide the fundamental communications -mechanisms. All other defined protocols are referred to as user-level -protocols (e.g., HTTP, used later in this @value{DOCUMENT}). - -@menu -* Basic Protocols:: The basic protocols. -* Ports:: The idea behind ports. -@end menu - -@node Basic Protocols, Ports, The TCP/IP Protocols, The TCP/IP Protocols -@subsection The Basic Internet Protocols - -@table @asis -@item IP -The Internet Protocol. This protocol is almost never used directly by -applications. It provides the basic packet delivery and routing infrastructure -of the Internet. Much like the phone company's switching centers or the Post -Office's trucks, it is not of much day-to-day interest to the regular user -(or programmer). -It happens to be a best effort datagram protocol. - -@item UDP -The User Datagram Protocol. This is a best effort datagram protocol. -It provides a small amount of extra reliability over IP, and adds -the notion of @dfn{ports}, described in @ref{Ports, ,TCP and UDP Ports}. - -@item TCP -The Transmission Control Protocol. This is a duplex, reliable, sequenced -byte-stream protocol, again layered on top of IP, and also providing the -notion of ports. This is the protocol that you will most likely use -when using @command{gawk} for network programming. -@end table - -All other user-level protocols use either TCP or UDP to do their basic -communications. Examples are SMTP (Simple Mail Transfer Protocol), -FTP (File Transfer Protocol) and HTTP (HyperText Transfer Protocol). -@cindex SMTP -@cindex FTP -@cindex HTTP - -@node Ports, , Basic Protocols, The TCP/IP Protocols -@subsection TCP and UDP Ports - -In the postal system, the address on an envelope indicates a physical -location, such as a residence or office building. But there may be -more than one person at the location; thus you have to further quantify -the recipient by putting a person or company name on the envelope. - -In the phone system, one phone number may represent an entire company, -in which case you need a person's extension number in order to -reach that individual directly. Or, when you call a home, you have to -say, ``May I please speak to ...'' before talking to the person directly. - -IP networking provides the concept of addressing. An IP address represents -a particular computer, but no more. In order to reach the mail service -on a system, or the FTP or WWW service on a system, you have to have some -way to further specify which service you want. In the Internet Protocol suite, -this is done with @dfn{port numbers}, which represent the services, much -like an extension number used with a phone number. - -Port numbers are 16-bit integers. Unix and Unix-like systems reserve ports -below 1024 for ``well known'' services, such as SMTP, FTP, and HTTP. -Numbers above 1024 may be used by any application, although there is no -promise made that a particular port number is always available. - -@node Making Connections, , The TCP/IP Protocols, Introduction -@section Making TCP/IP Connections (And Some Terminology) - -Two terms come up repeatedly when discussing networking: -@dfn{client} and @dfn{server}. For now, we'll discuss these terms -at the @dfn{connection level}, when first establishing connections -between two processes on different systems over a network. -(Once the connection is established, the higher level, or -@dfn{application level} protocols, -such as HTTP or FTP, determine who is the client and who is the -server. Often, it turns out that the client and server are the -same in both roles.) - -@cindex server -The @dfn{server} is the system providing the service, such as the -web server or email server. It is the @dfn{host} (system) which -is @emph{connected to} in a transaction. -For this to work though, the server must be expecting connections. -Much as there has to be someone at the office building to answer -the phone@footnote{In the days before voice mail systems!}, the -server process (usually) has to be started first and waiting -for a connection. - -@cindex client -The @dfn{client} is the system requesting the service. -It is the system @emph{initiating the connection} in a transaction. -(Just as when you pick up the phone to call an office or store.) - -In the TCP/IP framework, each end of a connection is represented by a pair -of (@var{address}, @var{port}) pairs. For the duration of the connection, -the ports in use at each end are unique, and cannot be used simultaneously -by other processes on the same system. (Only after closing a connection -can a new one be built up on the same port. This is contrary to the usual -behavior of fully developed web servers which have to avoid situations -in which they are not reachable. We have to pay this price in order to -enjoy the benefits of a simple communication paradigm in @command{gawk}.) - -@cindex blocking -@cindex synchronous communications -Furthermore, once the connection is established, communications -are @dfn{synchronous}. I.e., each end waits on the other to finish -transmitting, before replying. This is much like two people in a phone -conversation. While both could talk simultaneously, doing so usually -doesn't work too well. - -In the case of TCP, the synchronicity is enforced by the protocol when -sending data. Data writes @dfn{block} until the data have been received on the -other end. For both TCP and UDP, data reads block until there is incoming -data waiting to be read. This is summarized in the following table, -where an ``X'' indicates that the given action blocks. - -@ifnottex -@multitable {Protocol} {Reads} {Writes} -@item TCP @tab X @tab X -@item UDP @tab X @tab -@item RAW @tab X @tab -@end multitable -@end ifnottex -@tex -\centerline{ -\vbox{\bigskip % space above the table (about 1 linespace) -% Because we have vertical rules, we can't let TeX insert interline space -% in its usual way. -\offinterlineskip -\halign{\hfil\strut# &\vrule #& \hfil#\hfil& \hfil#\hfil\cr -Protocol&&\quad Reads\quad &Writes\cr -\noalign{\hrule} -\omit&height 2pt\cr -\noalign{\hrule height0pt}% without this the rule does not extend; why? -TCP&&X&X\cr -UDP&&X&\cr -RAW&&X&\cr -}}} -@end tex - -@node Using Networking, Some Applications and Techniques, Introduction, Top -@comment node-name, next, previous, up -@chapter Networking With @command{gawk} - -@cindex network -The @command{awk} programming language was originally developed as a -pattern-matching language for writing short programs to perform -data manipulation tasks. -@command{awk}'s strength is the manipulation of textual data -that is stored in files. -It was never meant to be used for networking purposes. -To exploit its features in a -networking context, it's necessary to use an access mode for network connections -that resembles the access of files as closely as possible. - -@cindex Perl -@cindex Python -@cindex Tcl/Tk -@command{awk} is also meant to be a prototyping language. It is used -to demonstrate feasibility and to play with features and user interfaces. -This can be done with file-like handling of network -connections. -@command{gawk} trades the lack -of many of the advanced features of the TCP/IP family of protocols -for the convenience of simple connection handling. -The advanced -features are available when programming in C or Perl. In fact, the -network programming -in this @value{CHAPTER} -is very similar to what is described in books like -@cite{Internet Programming with Python}, -@cite{Advanced Perl Programming}, -or -@cite{Web Client Programming with Perl}. -But it's done here without first having to learn object-oriented ideology, underlying -languages such as Tcl/Tk, Perl, Python, or all of the libraries necessary to -extend these languages before they are ready for the Internet. - -This @value{CHAPTER} demonstrates how to use the TCP protocol. The -other protocols are much less important for most users (UDP) or even -untractable (RAW). - -@menu -* Gawk Special Files:: How to do @command{gawk} networking. -* TCP Connecting:: Making a TCP connection. -* Troubleshooting:: Troubleshooting TCP/IP connections. -* Interacting:: Interacting with a service. -* Setting Up:: Setting up a service. -* Email:: Reading email. -* Web page:: Reading a Web page. -* Primitive Service:: A primitive Web service. -* Interacting Service:: A Web service with interaction. -* Simple Server:: A simple Web server. -* Caveats:: Network programming caveats. -* Challenges:: Where to go from here. -@end menu - -@node Gawk Special Files, TCP Connecting, Using Networking, Using Networking -@comment node-name, next, previous, up -@section @command{gawk} Networking Mechanisms -@cindex network - -The @samp{|&} operator introduced in @command{gawk} 3.1 for use in -communicating with a @dfn{co-process} is described in -@ref{Two-way I/O, ,Two-way Communications With Another Process, gawk, GAWK: Effective AWK Programming}. -It shows how to do two-way I/O to a -separate process, sending it data with @code{print} or @code{printf} and -reading data with @code{getline}. If you haven't read it already, you should -detour there to do so. - -@command{gawk} transparently extends the two-way I/O mechanism to simple networking through -the use of special @value{FN}s. When a ``co-process'' is started that matches -the special files we are about to describe, @command{gawk} creates the appropriate network -connection, and then two-way I/O proceeds as usual. - -At the C, C++ (and basic Perl) level, networking is accomplished -via @dfn{sockets}, an Application Programming Interface (API) originally -developed at the University of California at Berkeley that is now used -almost universally for TCP/IP networking. -Socket level programming, while fairly straightforward, requires paying -attention to a number of details, as well as using binary data. It is not -well-suited for use from a high-level language like @command{awk}. -The special files provided in @command{gawk} hide the details from -the programmer, making things much simpler and easier to use. -@c Who sez we can't toot our own horn occasionally? - -The special @value{FN} for network access is made up of several fields, all -of them mandatory, none of them optional: - -@example -/inet/@var{protocol}/@var{localport}/@var{hostname}/@var{remoteport} -@end example - -The @file{/inet/} field is, of course, constant when accessing the network. -The @var{localport} and @var{remoteport} fields do not have a meaning -when used with @file{/inet/raw} because ``ports'' only apply to -TCP and UDP. So, when using @file{/inet/raw}, the port fields always have -to be @samp{0}. - -@menu -* Special File Fields:: The fields in the special file name. -* Comparing Protocols:: Differences between the protocols. -@end menu - -@node Special File Fields, Comparing Protocols, Gawk Special Files, Gawk Special Files -@subsection The Fields of the Special @value{FFN} -This @value{SECTION} explains the meaning of all the other fields, -as well as the range of values and the defaults. -All of the fields are mandatory. To let the system pick a value, -or if the field doesn't apply to the protocol, specify it as @samp{0}. - -@table @var -@item protocol -Determines which member of the TCP/IP -family of protocols is selected to transport the data across the -network. There are three possible values (always written in lowercase): -@samp{tcp}, @samp{udp}, and @samp{raw}. The exact meaning of each is -explained later in this @value{SECTION}. - -@item localport -Determines which port on the local -machine is used to communicate across the network. It has no meaning -with @file{/inet/raw} and must therefore be @samp{0}. Application level clients -usually use @samp{0} to indicate they do not care which local port is -used---instead they specify a remote port to connect to. It is vital for -application level servers to use a number different from @samp{0} here -because their service has to be available at a specific publicly-known -port number. It is possible to use a name from @file{/etc/services} here. - -@item hostname -Determines which remote host is to -be at the other end of the connection. Application level servers must fill -this field with a @samp{0} to indicate their being open for all other hosts -to connect to them and enforce connection level server behavior this way. -It is not possible for an application level server to restrict its -availability to one remote host by entering a host name here. -Application level clients must enter a name different from @samp{0}. -The name can be either symbolic -(e.g., @samp{jpl-devvax.jpl.nasa.gov}) or numeric (e.g., @samp{128.149.1.143}). - -@item remoteport -Determines which port on the remote -machine is used to communicate across the network. It has no meaning -with @file{/inet/raw} and must therefore be 0. -For @file{/inet/tcp} and @file{/inet/udp}, -application level clients @emph{must} use a number -other than @samp{0} to indicate which port on the remote machine -they want to connect to. Application level servers must not fill this field with -a @samp{0}. Instead they specify a local port for clients to connect to. -It is possible to use a name from @file{/etc/services} here. -@end table - -Experts in network programming will notice that the usual -client/server asymmetry found at the level of the socket API is not visible -here. This is for the sake of simplicity of the high-level concept. If this -asymmetry is necessary for your application, -use another language. -For @command{gawk}, it is -more important to enable users to write a client program with a minimum -of code. What happens when first accessing a network connection is seen -in the following pseudo-code: - -@smallexample -if ((name of remote host given) && (other side accepts connection)) @{ - rendez-vous successful; transmit with getline or print -@} else @{ - if ((other side did not accept) && (localport == 0)) - exit unsuccessful - if (TCP) @{ - set up a server accepting connections - this means waiting for the client on the other side to connect - @} else - ready -@} -@end smallexample - -The exact behavior of this algorithm depends on the values of the -fields of the special @value{FN}. When in doubt, the following table -gives you the combinations of values and their meaning. If this -table is too complicated, focus on the three lines printed in -@strong{bold}. All the examples in -@ref{Using Networking, ,Networking With @command{gawk}}, -use only the -patterns printed in bold letters. - -@multitable {12345678901234} {123456} {123456} {1234567} {1234567890123456789012345} -@item @sc{protocol} @tab @sc{local port} @tab @sc{host name} -@tab @sc{remote port} @tab @sc{Resulting connection level behavior} -@item @strong{tcp} @tab @strong{0} @tab @strong{x} @tab @strong{x} @tab - @strong{Dedicated client, fails if immediately connecting to a - server on the other side fails} -@item udp @tab 0 @tab x @tab x @tab Dedicated client -@item raw @tab 0 @tab x @tab 0 @tab Dedicated client, works only as @code{root} -@item @strong{tcp, udp} @tab @strong{x} @tab @strong{x} @tab @strong{x} @tab - @strong{Client, switches to dedicated server if necessary} -@item @strong{tcp, udp} @tab @strong{x} @tab @strong{0} @tab @strong{0} @tab - @strong{Dedicated server} -@item raw @tab 0 @tab 0 @tab 0 @tab Dedicated server, works only as @code{root} -@item tcp, udp, raw @tab x @tab x @tab 0 @tab Invalid -@item tcp, udp, raw @tab 0 @tab 0 @tab x @tab Invalid -@item tcp, udp, raw @tab x @tab 0 @tab x @tab Invalid -@item tcp, udp @tab 0 @tab 0 @tab 0 @tab Invalid -@item tcp, udp @tab 0 @tab x @tab 0 @tab Invalid -@item raw @tab x @tab 0 @tab 0 @tab Invalid -@item raw @tab 0 @tab x @tab x @tab Invalid -@item raw @tab x @tab x @tab x @tab Invalid -@end multitable - -In general, TCP is the preferred mechanism to use. It is the simplest -protocol to understand and to use. Use the others only if circumstances -demand low-overhead. - -@node Comparing Protocols, , Special File Fields, Gawk Special Files -@subsection Comparing Protocols - -This @value{SECTION} develops a pair of programs (sender and receiver) -that do nothing but send a timestamp from one machine to another. The -sender and the receiver are implemented with each of the three protocols -available and demonstrate the differences between them. - -@menu -* File /inet/tcp:: The TCP special file. -* File /inet/udp:: The UDB special file. -* File /inet/raw:: The RAW special file. -@end menu - -@node File /inet/tcp, File /inet/udp, Comparing Protocols, Comparing Protocols -@subsubsection @file{/inet/tcp} -@cindex @file{/inet/tcp} special files -@cindex TCP -Once again, always use TCP. -(Use UDP when low-overhead is a necessity, and use RAW for -network experimentation.) -The first example is the sender -program: - -@example -# Server -BEGIN @{ - print strftime() |& "/inet/tcp/8888/0/0" - close("/inet/tcp/8888/0/0") -@} -@end example - -The receiver is very simple: - -@example -# Client -BEGIN @{ - "/inet/tcp/0/localhost/8888" |& getline - print $0 - close("/inet/tcp/0/localhost/8888") -@} -@end example - -TCP guarantees that the bytes arrive at the receiving end in exactly -the same order that they were sent. No byte is lost -(except for broken connections), doubled, or out of order. Some -overhead is necessary to accomplish this, but this is the price to pay for -a reliable service. -It does matter which side starts first. The sender/server has to be started -first, and it waits for the receiver to read a line. - -@node File /inet/udp, File /inet/raw, File /inet/tcp, Comparing Protocols -@subsubsection @file{/inet/udp} -@cindex @file{/inet/udp} special files -@cindex UDP -The server and client programs that use UDP are almost identical to their TCP counterparts; -only the @var{protocol} has changed. As before, it does matter which side -starts first. The receiving side blocks and waits for the sender. -In this case, the receiver/client has to be started first: - -@page -@example -# Server -BEGIN @{ - print strftime() |& "/inet/udp/8888/0/0" - close("/inet/udp/8888/0/0") -@} -@end example - -The receiver is almost identical to the TCP receiver: - -@example -# Client -BEGIN @{ - "/inet/udp/0/localhost/8888" |& getline - print $0 - close("/inet/udp/0/localhost/8888") -@} -@end example - -UDP cannot guarantee that the datagrams at the receiving end will arrive in exactly -the same order they were sent. Some datagrams could be -lost, some doubled, and some out of order. But no overhead is necessary to -accomplish this. This unreliable behavior is good enough for tasks -such as data acquisition, logging, and even stateless services like NFS. - -@node File /inet/raw, , File /inet/udp, Comparing Protocols -@subsubsection @file{/inet/raw} -@cindex @file{/inet/raw} special files -@cindex RAW - -This is an IP-level protocol. Only @code{root} is allowed to access this -special file. It is meant to be the basis for implementing -and experimenting with transport level protocols.@footnote{This special file -is reserved, but not otherwise currently implemented.} -In the most general case, -the sender has to supply the encapsulating header bytes in front of the -packet and the receiver has to strip the additional bytes from the message. - -@cindex dark corner -RAW receivers cannot receive packets sent with TCP or UDP because the -operating system does not deliver the packets to a RAW receiver. The -operating system knows about some of the protocols on top of IP -and decides on its own which packet to deliver to which process. -@value{DARKCORNER} -Therefore, the UDP receiver must be used for receiving UDP -datagrams sent with the RAW sender. This is a dark corner, not only of -@command{gawk}, but also of TCP/IP. - -@cindex SPAK utility -For extended experimentation with protocols, look into -the approach implemented in a tool called SPAK. -This tool reflects the hierarchical layering of protocols (encapsulation) -in the way data streams are piped out of one program into the next one. -It shows which protocol is based on which other (lower-level) protocol -by looking at the command-line ordering of the program calls. -Cleverly thought out, SPAK is much better than @command{gawk}'s -@file{/inet} for learning the meaning of each and every bit in the -protocol headers. - -The next example uses the RAW protocol to emulate -the behavior of UDP. The sender program is the same as above, but with some -additional bytes that fill the places of the UDP fields: - -@example -@group -BEGIN @{ - Message = "Hello world\n" - SourcePort = 0 - DestinationPort = 8888 - MessageLength = length(Message)+8 - RawService = "/inet/raw/0/localhost/0" - printf("%c%c%c%c%c%c%c%c%s", - SourcePort/256, SourcePort%256, - DestinationPort/256, DestinationPort%256, - MessageLength/256, MessageLength%256, - 0, 0, Message) |& RawService - fflush(RawService) - close(RawService) -@} -@end group -@end example - -Since this program tries -to emulate the behavior of UDP, it checks if -the RAW sender is understood by the UDP receiver but not if the RAW receiver -can understand the UDP sender. In a real network, the -RAW receiver is hardly -of any use because it gets every IP packet that -comes across the network. There are usually so many packets that -@command{gawk} would be too slow for processing them. -Only on a network with little -traffic can the IP-level receiver program be tested. Programs for analyzing -IP traffic on modem or ISDN channels should be possible. - -Port numbers do not have a meaning when using @file{/inet/raw}. Their fields -have to be @samp{0}. Only TCP and UDP use ports. Receiving data from -@file{/inet/raw} is difficult, not only because of processing speed but also -because data is usually binary and not restricted to ASCII. This -implies that line separation with @code{RS} does not work as usual. - -@node TCP Connecting, Troubleshooting, Gawk Special Files, Using Networking -@section Establishing a TCP Connection - -Let's observe a network connection at work. Type in the following program -and watch the output. Within a second, it connects via TCP (@file{/inet/tcp}) -to the machine it is running on (@samp{localhost}), and asks the service -@samp{daytime} on the machine what time it is: - -@cindex @code{|&} I/O operator -@cindex @code{getline} built-in function -@example -BEGIN @{ - "/inet/tcp/0/localhost/daytime" |& getline - print $0 - close("/inet/tcp/0/localhost/daytime") -@} -@end example - -Even experienced @command{awk} users will find the second line strange in two -respects: - -@itemize @bullet -@item -A special file is used as a shell command that pipes its output -into @code{getline}. One would rather expect to see the special file -being read like any other file (@samp{getline < -"/inet/tcp/0/localhost/daytime")}. - -@item -The operator @samp{|&} has not been part of any @command{awk} -implementation (until now). -It is actually the only extension of the @command{awk} -language needed (apart from the special files) to introduce network access. -@end itemize - -The @samp{|&} operator was introduced in @command{gawk} 3.1 in order to -overcome the crucial restriction that access to files and pipes in -@command{awk} is always unidirectional. It was formerly impossible to use -both access modes on the same file or pipe. Instead of changing the whole -concept of file access, the @samp{|&} operator -behaves exactly like the usual pipe operator except for two additions: - -@itemize @bullet -@item -Normal shell commands connected to their @command{gawk} program with a @samp{|&} -pipe can be accessed bidirectionally. The @samp{|&} turns out to be a quite -general, useful, and natural extension of @command{awk}. - -@item -Pipes that consist of a special @value{FN} for network connections are not -executed as shell commands. Instead, they can be read and written to, just -like a full-duplex network connection. -@end itemize - -In the earlier example, the @samp{|&} operator tells @code{getline} -to read a line from the special file @file{/inet/tcp/0/localhost/daytime}. -We could also have printed a line into the special file. But instead we just -read a line with the time, printed it, and closed the connection. -(While we could just let @command{gawk} close the connection by finishing -the program, in this @value{DOCUMENT} -we are pedantic, and always explicitly close the connections.) - -@node Troubleshooting, Interacting, TCP Connecting, Using Networking -@section Troubleshooting Connection Problems -It may well be that for some reason the above program does not run on your -machine. When looking at possible reasons for this, you will learn much -about typical problems that arise in network programming. First of all, -your implementation of @command{gawk} may not support network access -because it is -a pre-3.1 version or you do not have a network interface in your machine. -Perhaps your machine uses some other protocol -like DECnet or Novell's IPX. For the rest of this @value{CHAPTER}, -we will assume -you work on a Unix machine that supports TCP/IP. If the above program does -not run on such a machine, it may help to replace the name -@samp{localhost} with the name of your machine or its IP address. If it -does, you could replace @samp{localhost} with the name of another machine -in your vicinity. This way, the program connects to another machine. -Now you should see the date and time being printed by the program. -Otherwise your machine may not support the @samp{daytime} service. -Try changing the service to @samp{chargen} or @samp{ftp}. This way, the program -connects to other services that should give you some response. If you are -curious, you should have a look at your file @file{/etc/services}. It could -look like this: - -@ignore -@multitable {1234567890123} {1234567890123} {123456789012345678901234567890123456789012} -@item Service @strong{name} @tab Service @strong{number} -@item echo @tab 7/tcp @tab echo sends back each line it receivces -@item echo @tab 7/udp @tab echo is good for testing purposes -@item discard @tab 9/tcp @tab discard behaves like @file{/dev/null} -@item discard @tab 9/udp @tab discard just throws away each line -@item daytime @tab 13/tcp @tab daytime sends date & time once per connection -@item daytime @tab 13/udp -@item chargen @tab 19/tcp @tab chargen infinitely produces character sets -@item chargen @tab 19/udp @tab chargen is good for testing purposes -@item ftp @tab 21/tcp @tab ftp is the usual file transfer protocol -@item telnet @tab 23/tcp @tab telnet is the usual login facility -@item smtp @tab 25/tcp @tab smtp is the Simple Mail Transfer Protocol -@item finger @tab 79/tcp @tab finger tells you who is logged in -@item www @tab 80/tcp @tab www is the HyperText Transfer Protocol -@item pop2 @tab 109/tcp @tab pop2 is an older version of pop3 -@item pop2 @tab 109/udp -@item pop3 @tab 110/tcp @tab pop3 is the Post Office Protocol -@item pop3 @tab 110/udp @tab pop3 is used for receiving email -@item nntp @tab 119/tcp @tab nntp is the USENET News Transfer Protocol -@item irc @tab 194/tcp @tab irc is the Internet Relay Chat -@item irc @tab 194/udp -@end multitable -@end ignore - -@smallexample -# /etc/services: -# -# Network services, Internet style -# -# Name Number/Protcol Alternate name # Comments - -echo 7/tcp -echo 7/udp -discard 9/tcp sink null -discard 9/udp sink null -daytime 13/tcp -daytime 13/udp -chargen 19/tcp ttytst source -chargen 19/udp ttytst source -ftp 21/tcp -telnet 23/tcp -smtp 25/tcp mail -finger 79/tcp -www 80/tcp http # WorldWideWeb HTTP -www 80/udp # HyperText Transfer Protocol -pop-2 109/tcp postoffice # POP version 2 -pop-2 109/udp -pop-3 110/tcp # POP version 3 -pop-3 110/udp -nntp 119/tcp readnews untp # USENET News -irc 194/tcp # Internet Relay Chat -irc 194/udp -@dots{} -@end smallexample - -@cindex Linux -@cindex GNU/Linux -@cindex Microsoft Windows -Here, you find a list of services that traditional Unix machines usually -support. If your GNU/Linux machine does not do so, it may be that these -services are switched off in some startup script. Systems running some -flavor of Microsoft Windows usually do @emph{not} support such services. -Nevertheless, it @emph{is} possible to do networking with @command{gawk} on -Microsoft -Windows.@footnote{Microsoft prefered to ignore the TCP/IP -family of protocols until 1995. Then came the rise of the Netscape browser -as a landmark ``killer application.'' Microsoft added TCP/IP support and -their own browser to Microsoft Windows 95 at the last minute. They even back-ported -their TCP/IP implementation to Microsoft Windows for Workgroups 3.11, but it was -a rather rudimentary and half-hearted implementation. Nevertheless, -the equivalent of @file{/etc/services} resides under -@file{c:\windows\services} on Microsoft Windows.} -The first column of the file gives the name of the service, -the second a unique number, and the protocol that one can use to connect to -this service. -The rest of the line is treated as a comment. -You see that some services (@samp{echo}) support TCP as -well as UDP. - -@node Interacting, Setting Up, Troubleshooting, Using Networking -@section Interacting with a Network Service - -The next program makes use of the possibility to really interact with a -network service by printing something into the special file. It asks the -so-called @command{finger} service if a user of the machine is logged in. When -testing this program, try to change @samp{localhost} to -some other machine name in your local network: - -@c system if test ! -d eg ; then mkdir eg ; fi -@c system if test ! -d eg/network ; then mkdir eg/network ; fi -@example -@c file eg/network/fingerclient.awk -BEGIN @{ - NetService = "/inet/tcp/0/localhost/finger" - print "@var{name}" |& NetService - while ((NetService |& getline) > 0) - print $0 - close(NetService) -@} -@c endfile -@end example - -After telling the service on the machine which user to look for, -the program repeatedly reads lines that come as a reply. When no more -lines are coming (because the service has closed the connection), the -program also closes the connection. Try replacing @code{"@var{name}"} with your -login name (or the name of someone else logged in). For a list -of all users currently logged in, replace @var{name} with an empty string -@code{""}. - -@cindex Linux -@cindex GNU/Linux -The final @code{close} command could be safely deleted from -the above script, because the operating system closes any open connection -by default when a script reaches the end of execution. In order to avoid -portability problems, it is best to always close connections explicitly. -With the Linux kernel, -for example, proper closing results in flushing of buffers. Letting -the close happen by default may result in discarding buffers. - -@ignore -@c Chuck comments that this seems out of place. He's right. I dunno -@c where to put it though. -@cindex @command{finger} utility -@cindex RFC 1288 -In the early days of the Internet (up until about 1992), you could use -such a program to check if some user in another country was logged in on -a specific machine. -RFC 1288@footnote{@uref{http://www.cis.ohio-state.edu/htbin/rfc/rfc1288.html}} -provides the exact definition of the @command{finger} protocol. -Every contemporary Unix system also has a command named @command{finger}, -which functions as a client for the protocol of the same name. -Still today, some people maintain simple information systems -with this ancient protocol. For example, by typing -@samp{finger quake@@seismo.unr.edu} -you get the latest @dfn{Earthquake Bulletin} for the state of Nevada. - -@cindex Earthquake Bulletin -@smallexample -$ finger quake@@seismo.unr.edu - -[@dots{}] - -DATE-(UTC)-TIME LAT LON DEP MAG COMMENTS -yy/mm/dd hh:mm:ss deg. deg. km - -98/12/14 21:09:22 37.47N 116.30W 0.0 2.3Md 76.4 km S of WARM SPRINGS, NEVA -98/12/14 22:05:09 39.69N 120.41W 11.9 2.1Md 53.8 km WNW of RENO, NEVADA -98/12/15 14:14:19 38.04N 118.60W 2.0 2.3Md 51.0 km S of HAWTHORNE, NEVADA -98/12/17 01:49:02 36.06N 117.58W 13.9 3.0Md 74.9 km SE of LONE PINE, CALIFOR -98/12/17 05:39:26 39.95N 120.87W 6.2 2.6Md 101.6 km WNW of RENO, NEVADA -98/12/22 06:07:42 38.68N 119.82W 5.2 2.3Md 50.7 km S of CARSON CITY, NEVAD -@end smallexample - -@noindent -This output from @command{finger} contains the time, location, depth, -magnitude, and a short comment about -the earthquakes registered in that region during the last 10 days. -In many places today the use of such services is restricted -because most networks have firewalls and proxy servers between them -and the Internet. Most firewalls are programmed to not let -@command{finger} requests go beyond the local network. - -@cindex Coke machine -Another (ab)use of the @command{finger} protocol are several Coke machines -that are connected to the Internet. There is a short list of such -Coke machines.@footnote{@uref{http://ca.yahoo.com/Computers_and_Internet/Internet/Devices_Connected_to_the_Internet/Soda_Machines/}} -You can access them either from the command-line or with a simple -@command{gawk} script. They usually tell you about the different -flavors of Coke and beer available there. If you have an account there, -you can even order some drink this way. -@end ignore - -When looking at @file{/etc/services} you may have noticed that the -@samp{daytime} service is also available with @samp{udp}. In the earlier -example, change @samp{tcp} to @samp{udp}, -and change @samp{finger} to @samp{daytime}. -After starting the modified program, you see the expected day and time message. -The program then hangs, because it waits for more lines coming from the -service. However, they never come. This behavior is a consequence of the -differences between TCP and UDP. When using UDP, neither party is -automatically informed about the other closing the connection. -Continuing to experiment this way reveals many other subtle -differences between TCP and UDP. To avoid such trouble, one should always -remember the advice Douglas E.@: Comer and David Stevens give in -Volume III of their series @cite{Internetworking With TCP} -(page 14): - -@cindex TCP -@cindex UDP -@quotation -When designing client-server applications, beginners are strongly -advised to use TCP because it provides reliable, connection-oriented -communication. Programs only use UDP if the application protocol handles -reliability, the application requires hardware broadcast or multicast, -or the application cannot tolerate virtual circuit overhead. -@end quotation - -@node Setting Up, Email, Interacting, Using Networking -@section Setting Up a Service -The preceding programs behaved as clients that connect to a server somewhere -on the Internet and request a particular service. Now we set up such a -service to mimic the behavior of the @samp{daytime} service. -Such a server does not know in advance who is going to connect to it over -the network. Therefore we cannot insert a name for the host to connect to -in our special @value{FN}. - -Start the following program in one window. Notice that the service does -not have the name @samp{daytime}, but the number @samp{8888}. -From looking at @file{/etc/services}, you know that names like @samp{daytime} -are just mnemonics for predetermined 16-bit integers. -Only the system administrator (@code{root}) could enter -our new service into @file{/etc/services} with an appropriate name. -Also notice that the service name has to be entered into a different field -of the special @value{FN} because we are setting up a server, not a client: - -@cindex @command{finger} utility -@cindex server -@example -BEGIN @{ - print strftime() |& "/inet/tcp/8888/0/0" - close("/inet/tcp/8888/0/0") -@} -@end example - -Now open another window on the same machine. -Copy the client program given as the first example -(@pxref{TCP Connecting, ,Establishing a TCP Connection}) -to a new file and edit it, changing the name @samp{daytime} to -@samp{8888}. Then start the modified client. You should get a reply -like this: - -@example -Sat Sep 27 19:08:16 CEST 1997 -@end example - -@noindent -Both programs explicitly close the connection. - -@cindex Microsoft Windows -@cindex reserved ports -Now we will intentionally make a mistake to see what happens when the name -@samp{8888} (the so-called port) is already used by another service. -Start the server -program in both windows. The first one works, but the second one -complains that it could not open the connection. Each port on a single -machine can only be used by one server program at a time. Now terminate the -server program and change the name @samp{8888} to @samp{echo}. After restarting it, -the server program does not run any more and you know why: there already is -an @samp{echo} service running on your machine. But even if this isn't true, -you would not get -your own @samp{echo} server running on a Unix machine, -because the ports with numbers smaller -than 1024 (@samp{echo} is at port 7) are reserved for @code{root}. -On machines running some flavor of Microsoft Windows, there is no restriction -that reserves ports 1 to 1024 for a privileged user; hence you can start -an @samp{echo} server there. - -Turning this short server program into something really useful is simple. -Imagine a server that first reads a @value{FN} from the client through the -network connection, then does something with the file and -sends a result back to the client. The server-side processing -could be: - -@example -BEGIN @{ - NetService = "/inet/tcp/8888/0/0" - NetService |& getline - CatPipe = ("cat " $1) # sets $0 and the fields - while ((CatPipe | getline) > 0) - print $0 |& NetService - close(NetService) -@} -@end example - -@noindent -and we would -have a remote copying facility. Such a server reads the name of a file -from any client that connects to it and transmits the contents of the -named file across the net. The server-side processing could also be -the execution of a command that is transmitted across the network. From this -example, you can see how simple it is to open up a security hole on your -machine. If you allow clients to connect to your machine and -execute arbitrary commands, anyone would be free to do @samp{rm -rf *}. - -@node Email, Web page, Setting Up, Using Networking -@section Reading Email -@cindex POP -@cindex SMTP -@cindex RFC 1939 -@cindex RFC 821 -The distribution of email is usually done by dedicated email servers that -communicate with your machine using special protocols. To receive email, we -will use the Post Office Protocol (POP). Sending can be done with the much -older Simple Mail Transfer Protocol (SMTP). -@ignore -@footnote{RFC 1939 defines POP. -RFC 821 defines SMTP. See -@uref{http://rfc.fh-koeln.de/doc/rfc/html/rfc.html, RFCs in HTML}.} -@end ignore - -When you type in the following program, replace the @var{emailhost} by the -name of your local email server. Ask your administrator if the server has a -POP service, and then use its name or number in the program below. -Now the program is ready to connect to your email server, but it will not -succeed in retrieving your mail because it does not yet know your login -name or password. Replace them in the program and it -shows you the first email the server has in store: - -@example -BEGIN @{ - POPService = "/inet/tcp/0/@var{emailhost}/pop3" - RS = ORS = "\r\n" - print "user @var{name}" |& POPService - POPService |& getline - print "pass @var{password}" |& POPService - POPService |& getline - print "retr 1" |& POPService - POPService |& getline - if ($1 != "+OK") exit - print "quit" |& POPService - RS = "\r\n\\.\r\n" - POPService |& getline - print $0 - close(POPService) -@} -@end example - -@cindex RFC 1939 -The record separators @code{RS} and @code{ORS} are redefined because the -protocol (POP) requires CR-LF to separate lines. After identifying -yourself to the email service, the command @samp{retr 1} instructs the -service to send the first of all your email messages in line. If the service -replies with something other than @samp{+OK}, the program exits; maybe there -is no email. Otherwise, the program first announces that it intends to finish -reading email, and then redefines @code{RS} in order to read the entire -email as multiline input in one record. From the POP RFC, we know that the body -of the email always ends with a single line containing a single dot. -The program looks for this using @samp{RS = "\r\n\\.\r\n"}. -When it finds this sequence in the mail message, it quits. -You can invoke this program as often as you like; it does not delete the -message it reads, but instead leaves it on the server. - -@node Web page, Primitive Service, Email, Using Networking -@section Reading a Web Page -@cindex HTTP -@cindex RFC 2068 -@cindex RFC 2616 - -Retrieving a web page from a web server is as simple as -retrieving email from an email server. We only have to use a -similar, but not identical, protocol and a different port. The name of the -protocol is HyperText Transfer Protocol (HTTP) and the port number is usually -80. As in the preceding @value{SECTION}, ask your administrator about the -name of your local web server or proxy web server and its port number -for HTTP requests. - -@ignore -@c Chuck says this stuff isn't necessary -More detailed information about HTTP can be found at -the home of the web protocols,@footnote{@uref{http://www.w3.org/pub/WWW/Protocols}} -including the specification of HTTP in RFC 2068. The protocol specification -in RFC 2068 is concise and you can get it for free. If you need more -explanation and you are willing to pay for a book, you might be -interested in one of these books: - -@enumerate - -@item -When we started writing web clients and servers with @command{gawk}, -the only book available with details about HTTP was the one by Paul Hethmon -called -@cite{Illustrated Guide to HTTP}.@footnote{@uref{http://www.browsebooks.com/Hethmon/?882}} -Hethmon not only describes HTTP, -he also implements a simple web server in C++. - -@item -Since July 2000, O'Reilly offers the book by Clinton Wong called -@cite{HTTP Pocket Reference}.@footnote{@uref{http://www.oreilly.com/catalog/httppr}} -It only has 75 pages but its -focus definitely is HTTP. This pocket reference is not a replacement -for the RFC, but I wish I had had it back in 1997 when I started writing -scripts to handle HTTP. - -@item -Another small booklet about HTTP is the one by Toexcell Incorporated Staff, -ISBN 1-58348-270-9, called -@cite{Hypertext Transfer Protocol Http 1.0 Specifications} - -@end enumerate -@end ignore - -The following program employs a rather crude approach toward retrieving a -web page. It uses the prehistoric syntax of HTTP 0.9, which almost all -web servers still support. The most noticeable thing about it is that the -program directs the request to the local proxy server whose name you insert -in the special @value{FN} (which in turn calls @samp{www.yahoo.com}): - -@example -BEGIN @{ - RS = ORS = "\r\n" - HttpService = "/inet/tcp/0/@var{proxy}/80" - print "GET http://www.yahoo.com" |& HttpService - while ((HttpService |& getline) > 0) - print $0 - close(HttpService) -@} -@end example - -@cindex RFC 1945 -@cindex HTML -@cindex Yahoo! -Again, lines are separated by a redefined @code{RS} and @code{ORS}. -The @code{GET} request that we send to the server is the only kind of -HTTP request that existed when the web was created in the early 1990s. -HTTP calls this @code{GET} request a ``method,'' which tells the -service to transmit a web page (here the home page of the Yahoo! search -engine). Version 1.0 added the request methods @code{HEAD} and -@code{POST}. The current version of HTTP is 1.1,@footnote{Version 1.0 of -HTTP was defined in RFC 1945. HTTP 1.1 was initially specified in RFC -2068. In June 1999, RFC 2068 was made obsolete by RFC 2616. It is an update -without any substantial changes.} and knows the additional request -methods @code{OPTIONS}, @code{PUT}, @code{DELETE}, and @code{TRACE}. -You can fill in any valid web address, and the program prints the -HTML code of that page to your screen. - -Notice the similarity between the responses of the POP and HTTP -services. First, you get a header that is terminated by an empty line, and -then you get the body of the page in HTML. The lines of the headers also -have the same form as in POP. There is the name of a parameter, -then a colon, and finally the value of that parameter. - -@cindex CGI -@cindex @file{gif} image format -@cindex @file{png} image format -Images (@file{.png} or @file{.gif} files) can also be retrieved this way, -but then you -get binary data that should be redirected into a file. Another -application is calling a CGI (Common Gateway Interface) script on some -server. CGI scripts are used when the contents of a web page are not -constant, but generated instantly at the moment you send a request -for the page. For example, to get a detailed report about the current -quotes of Motorola stock shares, call a CGI script at Yahoo! with -the following: - -@example -get = "GET http://quote.yahoo.com/q?s=MOT&d=t" -print get |& HttpService -@end example - -You can also request weather reports this way. -@ignore -@cindex Boutell, Thomas -A good book to go on with is -the -@cite{HTML Source Book}.@footnote{@uref{http://www.utoronto.ca/webdocs/HTMLdocs/NewHTML/book.html}} -There are also some books on CGI programming -like @cite{CGI Programming in C & Perl}, -by Thomas Boutell@footnote{@uref{http://cseng.aw.com/bookdetail.qry?ISBN=0-201-42219-0&ptype=0}}, -and @cite{The CGI Book}.@footnote{@uref{http://www.cgibook.com}} -Another good source is @cite{The CGI Resource Index}}.@footnote{@uref{http://www.cgi-resources.com}} -@end ignore - -@node Primitive Service, Interacting Service, Web page, Using Networking -@section A Primitive Web Service -Now we know enough about HTTP to set up a primitive web service that just -says @code{"Hello, world"} when someone connects to it with a browser. -Compared -to the situation in the preceding @value{SECTION}, our program changes the role. It -tries to behave just like the server we have observed. Since we are setting -up a server here, we have to insert the port number in the @samp{localport} -field of the special @value{FN}. The other two fields (@var{hostname} and -@var{remoteport}) have to contain a @samp{0} because we do not know in -advance which host will connect to our service. - -In the early 1990s, all a server had to do was send an HTML document and -close the connection. Here, we adhere to the modern syntax of HTTP. -The steps are as follows: - -@enumerate 1 -@item -Send a status line telling the web browser that everything -is OK. - -@item -Send a line to tell the browser how many bytes follow in the -body of the message. This was not necessary earlier because both -parties knew that the document ended when the connection closed. Nowadays -it is possible to stay connected after the transmission of one web page. -This is to avoid the network traffic necessary for repeatedly establishing -TCP connections for requesting several images. Thus, there is the need to tell -the receiving party how many bytes will be sent. The header is terminated -as usual with an empty line. - -@item -Send the @code{"Hello, world"} body -in HTML. -The useless @code{while} loop swallows the request of the browser. -We could actually omit the loop, and on most machines the program would still -work. -First, start the following program: -@end enumerate - -@example -@c file eg/network/hello-serv.awk -BEGIN @{ - RS = ORS = "\r\n" - HttpService = "/inet/tcp/8080/0/0" - Hello = "<HTML><HEAD>" \ - "<TITLE>A Famous Greeting</TITLE></HEAD>" \ - "<BODY><H1>Hello, world</H1></BODY></HTML>" - Len = length(Hello) + length(ORS) - print "HTTP/1.0 200 OK" |& HttpService - print "Content-Length: " Len ORS |& HttpService - print Hello |& HttpService - while ((HttpService |& getline) > 0) - continue; - close(HttpService) -@} -@c endfile -@end example - -Now, on the same machine, start your favorite browser and let it point to -@uref{http://localhost:8080} (the browser needs to know on which port -our server is listening for requests). If this does not work, the browser -probably tries to connect to a proxy server that does not know your machine. -If so, change the browser's configuration so that the browser does not try to -use a proxy to connect to your machine. - -@node Interacting Service, Simple Server, Primitive Service, Using Networking -@section A Web Service with Interaction -@cindex GUI -@ifinfo -This node shows how to set up a simple web server. -The subnode is a library file that we will use with all the examples in -@ref{Some Applications and Techniques}. -@end ifinfo - -@menu -* CGI Lib:: A simple CGI library. -@end menu - -Setting up a web service that allows user interaction is more difficult and -shows us the limits of network access in @command{gawk}. In this @value{SECTION}, -we develop a main program (a @code{BEGIN} pattern and its action) -that will become the core of event-driven execution controlled by a -graphical user interface (GUI). -Each HTTP event that the user triggers by some action within the browser -is received in this central procedure. Parameters and menu choices are -extracted from this request and an appropriate measure is taken according to -the user's choice. -For example: - -@cindex HTTP server, core logic -@example -BEGIN @{ - if (MyHost == "") @{ - "uname -n" | getline MyHost - close("uname -n") - @} - if (MyPort == 0) MyPort = 8080 - HttpService = "/inet/tcp/" MyPort "/0/0" - MyPrefix = "http://" MyHost ":" MyPort - SetUpServer() - while ("awk" != "complex") @{ - # header lines are terminated this way - RS = ORS = "\r\n" - Status = 200 # this means OK - Reason = "OK" - Header = TopHeader - Document = TopDoc - Footer = TopFooter - if (GETARG["Method"] == "GET") @{ - HandleGET() - @} else if (GETARG["Method"] == "HEAD") @{ - # not yet implemented - @} else if (GETARG["Method"] != "") @{ - print "bad method", GETARG["Method"] - @} - Prompt = Header Document Footer - print "HTTP/1.0", Status, Reason |& HttpService - print "Connection: Close" |& HttpService - print "Pragma: no-cache" |& HttpService - len = length(Prompt) + length(ORS) - print "Content-length:", len |& HttpService - print ORS Prompt |& HttpService - # ignore all the header lines - while ((HttpService |& getline) > 0) - ; - # stop talking to this client - close(HttpService) - # wait for new client request - HttpService |& getline - # do some logging - print systime(), strftime(), $0 - # read request parameters - CGI_setup($1, $2, $3) - @} -@} -@end example - -This web server presents menu choices in the form of HTML links. -Therefore, it has to tell the browser the name of the host it is -residing on. When starting the server, the user may supply the name -of the host from the command line with @samp{gawk -v MyHost="Rumpelstilzchen"}. -If the user does not do this, the server looks up the name of the host it is -running on for later use as a web address in HTML documents. The same -applies to the port number. These values are inserted later into the -HTML content of the web pages to refer to the home system. - -Each server that is built around this core has to initialize some -application-dependent variables (such as the default home page) in a procedure -@code{SetUpServer}, which is called immediately before entering the -infinite loop of the server. For now, we will write an instance that -initiates a trivial interaction. With this home page, the client user -can click on two possible choices, and receive the current date either -in human-readable format or in seconds since 1970: - -@example -function SetUpServer() @{ - TopHeader = "<HTML><HEAD>" - TopHeader = TopHeader \ - "<title>My name is GAWK, GNU AWK</title></HEAD>" - TopDoc = "<BODY><h2>\ - Do you prefer your date <A HREF=" MyPrefix \ - "/human>human</A> or \ - <A HREF=" MyPrefix "/POSIX>POSIXed</A>?</h2>" ORS ORS - TopFooter = "</BODY></HTML>" -@} -@end example - -On the first run through the main loop, the default line terminators are -set and the default home page is copied to the actual home page. Since this -is the first run, @code{GETARG["Method"]} is not initialized yet, hence the -case selection over the method does nothing. Now that the home page is -initialized, the server can start communicating to a client browser. - -@cindex RFC 2068 -@cindex CGI -It does so by printing the HTTP header into the network connection -(@samp{print @dots{} |& HttpService}). This command blocks execution of -the server script until a client connects. If this server -script is compared with the primitive one we wrote before, you will notice -two additional lines in the header. The first instructs the browser -to close the connection after each request. The second tells the -browser that it should never try to @emph{remember} earlier requests -that had identical web addresses (no caching). Otherwise, it could happen -that the browser retrieves the time of day in the previous example just once, -and later it takes the web page from the cache, always displaying the same -time of day although time advances each second. - -Having supplied the initial home page to the browser with a valid document -stored in the parameter @code{Prompt}, it closes the connection and waits -for the next request. When the request comes, a log line is printed that -allows us to see which request the server receives. The final step in the -loop is to call the function @code{CGI_setup}, which reads all the lines -of the request (coming from the browser), processes them, and stores the -transmitted parameters in the array @code{PARAM}. The complete -text of these application-independent functions can be found in -@ref{CGI Lib, ,A Simple CGI Library}. -For now, we use a simplified version of @code{CGI_setup}: - -@example -function CGI_setup( method, uri, version, i) @{ - delete GETARG; delete MENU; delete PARAM - GETARG["Method"] = $1 - GETARG["URI"] = $2 - GETARG["Version"] = $3 - i = index($2, "?") - # is there a "?" indicating a CGI request? -@group - if (i > 0) @{ - split(substr($2, 1, i-1), MENU, "[/:]") - split(substr($2, i+1), PARAM, "&") - for (i in PARAM) @{ - j = index(PARAM[i], "=") - GETARG[substr(PARAM[i], 1, j-1)] = \ - substr(PARAM[i], j+1) - @} - @} else @{ # there is no "?", no need for splitting PARAMs - split($2, MENU, "[/:]") - @} -@end group -@} -@end example - -At first, the function clears all variables used for -global storage of request parameters. The rest of the function serves -the purpose of filling the global parameters with the extracted new values. -To accomplish this, the name of the requested resource is split into -parts and stored for later evaluation. If the request contains a @samp{?}, -then the request has CGI variables seamlessly appended to the web address. -Everything in front of the @samp{?} is split up into menu items, and -everything behind the @samp{?} is a list of @samp{@var{variable}=@var{value}} pairs -(separated by @samp{&}) that also need splitting. This way, CGI variables are -isolated and stored. This procedure lacks recognition of special characters -that are transmitted in coded form@footnote{As defined in RFC 2068.}. Here, any -optional request header and body parts are ignored. We do not need -header parameters and the request body. However, when refining our approach or -working with the @code{POST} and @code{PUT} methods, reading the header -and body -becomes inevitable. Header parameters should then be stored in a global -array as well as the body. - -On each subsequent run through the main loop, one request from a browser is -received, evaluated, and answered according to the user's choice. This can be -done by letting the value of the HTTP method guide the main loop into -execution of the procedure @code{HandleGET}, which evaluates the user's -choice. In this case, we have only one hierarchical level of menus, -but in the general case, -menus are nested. -The menu choices at each level are -separated by @samp{/}, just as in @value{FN}s. Notice how simple it is to -construct menus of arbitrary depth: - -@example -function HandleGET() @{ - if ( MENU[2] == "human") @{ - Footer = strftime() TopFooter - @} else if (MENU[2] == "POSIX") @{ - Footer = systime() TopFooter - @} -@} -@end example - -@cindex CGI -The disadvantage of this approach is that our server is slow and can -handle only one request at a time. Its main advantage, however, is that -the server -consists of just one @command{gawk} program. No need for installing an -@command{httpd}, and no need for static separate HTML files, CGI scripts, or -@code{root} privileges. This is rapid prototyping. -This program can be started on the same host that runs your browser. -Then let your browser point to @uref{http://localhost:8080}. - -@cindex @file{xbm} image format -@cindex image format -@cindex GNUPlot utility -It is also possible to include images into the HTML pages. -Most browsers support the not very well-known -@file{.xbm} format, -which may contain only -monochrome pictures but is an ASCII format. Binary images are possible but -not so easy to handle. Another way of including images is to generate them -with a tool such as GNUPlot, -by calling the tool with the @code{system} function or through a pipe. - -@node CGI Lib, , Interacting Service, Interacting Service -@subsection A Simple CGI Library -@quotation -@i{HTTP is like being married: you have to be able to handle whatever -you're given, while being very careful what you send back.}@* -Phil Smith III,@* -@uref{http://www.netfunny.com/rhf/jokes/99/Mar/http.html} -@end quotation - -In @ref{Interacting Service, ,A Web Service with Interaction}, -we saw the function @code{CGI_setup} as part of the web server -``core logic'' framework. The code presented there handles almost -everything necessary for CGI requests. -One thing it doesn't do is handle encoded characters in the requests. -For example, an @samp{&} is encoded as a percent sign followed by -the hexadecimal value---@samp{%26}. These encoded values should be -decoded. -Following is a simple library to perform these tasks. -This code is used for all web server examples -used throughout the rest of this @value{DOCUMENT}. -If you want to use it for your own web server, store the source code -into a file named @file{inetlib.awk}. Then you can include -these functions into your code by placing the following statement -into your program: - -@example -@@include inetlib.awk -@end example - -@noindent -on the first line of your script. But beware, this mechanism is -only possible if you invoke your web server script with @command{igawk} -instead of the usual @command{awk} or @command{gawk}. -Here is the code: - -@example -@c file eg/network/coreserv.awk -# CGI Library and core of a web server -@c endfile -@ignore -@c file eg/network/coreserv.awk -# -# Juergen Kahrs, Juergen.Kahrs@@vr-web.de -# with Arnold Robbins, arnold@@gnu.org -# September 2000 - -@c endfile -@end ignore -@c file eg/network/coreserv.awk -# Global arrays -# GETARG --- arguments to CGI GET command -# MENU --- menu items (path names) -# PARAM --- parameters of form x=y - -# Optional variable MyHost contains host address -# Optional variable MyPort contains port number -# Needs TopHeader, TopDoc, TopFooter -# Sets MyPrefix, HttpService, Status, Reason - -BEGIN @{ - if (MyHost == "") @{ - "uname -n" | getline MyHost - close("uname -n") - @} - if (MyPort == 0) MyPort = 8080 - HttpService = "/inet/tcp/" MyPort "/0/0" - MyPrefix = "http://" MyHost ":" MyPort - SetUpServer() - while ("awk" != "complex") @{ - # header lines are terminated this way - RS = ORS = "\r\n" - Status = 200 # this means OK - Reason = "OK" - Header = TopHeader - Document = TopDoc - Footer = TopFooter - if (GETARG["Method"] == "GET") @{ - HandleGET() - @} else if (GETARG["Method"] == "HEAD") @{ - # not yet implemented - @} else if (GETARG["Method"] != "") @{ - print "bad method", GETARG["Method"] - @} - Prompt = Header Document Footer - print "HTTP/1.0", Status, Reason |& HttpService - print "Connection: Close" |& HttpService - print "Pragma: no-cache" |& HttpService - len = length(Prompt) + length(ORS) - print "Content-length:", len |& HttpService - print ORS Prompt |& HttpService - # ignore all the header lines - while ((HttpService |& getline) > 0) - continue - # stop talking to this client - close(HttpService) - # wait for new client request - HttpService |& getline - # do some logging - print systime(), strftime(), $0 - CGI_setup($1, $2, $3) - @} -@} - -function CGI_setup( method, uri, version, i) -@{ - delete GETARG - delete MENU - delete PARAM - GETARG["Method"] = method - GETARG["URI"] = uri - GETARG["Version"] = version - - i = index(uri, "?") - if (i > 0) @{ # is there a "?" indicating a CGI request? - split(substr(uri, 1, i-1), MENU, "[/:]") - split(substr(uri, i+1), PARAM, "&") - for (i in PARAM) @{ - PARAM[i] = _CGI_decode(PARAM[i]) - j = index(PARAM[i], "=") - GETARG[substr(PARAM[i], 1, j-1)] = \ - substr(PARAM[i], j+1) - @} - @} else @{ # there is no "?", no need for splitting PARAMs - split(uri, MENU, "[/:]") - @} - for (i in MENU) # decode characters in path - if (i > 4) # but not those in host name - MENU[i] = _CGI_decode(MENU[i]) -@} -@c endfile -@end example - -This isolates details in a single function, @code{CGI_setup}. -Decoding of encoded characters is pushed off to a helper function, -@code{_CGI_decode}. The use of the leading underscore (@samp{_}) in -the function name is intended to indicate that it is an ``internal'' -function, although there is nothing to enforce this: - -@example -@c file eg/network/coreserv.awk -function _CGI_decode(str, hexdigs, i, pre, code1, code2, - val, result) -@{ - hexdigs = "123456789abcdef" - - i = index(str, "%") - if (i == 0) # no work to do - return str - - do @{ - pre = substr(str, 1, i-1) # part before %xx - code1 = substr(str, i+1, 1) # first hex digit - code2 = substr(str, i+2, 1) # second hex digit - str = substr(str, i+3) # rest of string - - code1 = tolower(code1) - code2 = tolower(code2) - val = index(hexdigs, code1) * 16 \ - + index(hexdigs, code2) - - result = result pre sprintf("%c", val) - i = index(str, "%") - @} while (i != 0) - if (length(str) > 0) - result = result str - return result -@} -@c endfile -@end example - -This works by splitting the string apart around an encoded character. -The two digits are converted to lowercase and looked up in a string -of hex digits. Note that @code{0} is not in the string on purpose; -@code{index} returns zero when it's not found, automatically giving -the correct value! Once the hexadecimal value is converted from -characters in a string into a numerical value, @code{sprintf} -converts the value back into a real character. -The following is a simple test harness for the above functions: - -@example -@c file eg/network/testserv.awk -BEGIN @{ - CGI_setup("GET", - "http://www.gnu.org/cgi-bin/foo?p1=stuff&p2=stuff%26junk" \ - "&percent=a %25 sign", - "1.0") - for (i in MENU) - printf "MENU[\"%s\"] = %s\n", i, MENU[i] - for (i in PARAM) - printf "PARAM[\"%s\"] = %s\n", i, PARAM[i] - for (i in GETARG) - printf "GETARG[\"%s\"] = %s\n", i, GETARG[i] -@} -@c endfile -@end example - -And this is the result when we run it: - -@c artificial line wrap in last output line -@example -$ gawk -f testserv.awk -@print{} MENU["4"] = www.gnu.org -@print{} MENU["5"] = cgi-bin -@print{} MENU["6"] = foo -@print{} MENU["1"] = http -@print{} MENU["2"] = -@print{} MENU["3"] = -@print{} PARAM["1"] = p1=stuff -@print{} PARAM["2"] = p2=stuff&junk -@print{} PARAM["3"] = percent=a % sign -@print{} GETARG["p1"] = stuff -@print{} GETARG["percent"] = a % sign -@print{} GETARG["p2"] = stuff&junk -@print{} GETARG["Method"] = GET -@print{} GETARG["Version"] = 1.0 -@print{} GETARG["URI"] = http://www.gnu.org/cgi-bin/foo?p1=stuff& -p2=stuff%26junk&percent=a %25 sign -@end example - -@node Simple Server, Caveats, Interacting Service, Using Networking -@section A Simple Web Server -@cindex GUI -In the preceding @value{SECTION}, we built the core logic for event driven GUIs. -In this @value{SECTION}, we finally extend the core to a real application. -No one would actually write a commercial web server in @command{gawk}, but -it is instructive to see that it is feasible in principle. - -@iftex -@image{uf002331,4in} -@end iftex - -@cindex ELIZA program -@cindex Weizenbaum, Joseph -The application is ELIZA, the famous program by Joseph Weizenbaum that -mimics the behavior of a professional psychotherapist when talking to you. -Weizenbaum would certainly object to this description, but this is part of -the legend around ELIZA. -Take the site-independent core logic and append the following code: - -@example -@c file eg/network/eliza.awk -function SetUpServer() @{ - SetUpEliza() - TopHeader = \ - "<HTML><title>An HTTP-based System with GAWK</title>\ - <HEAD><META HTTP-EQUIV=\"Content-Type\"\ - CONTENT=\"text/html; charset=iso-8859-1\"></HEAD>\ - <BODY BGCOLOR=\"#ffffff\" TEXT=\"#000000\"\ - LINK=\"#0000ff\" VLINK=\"#0000ff\"\ - ALINK=\"#0000ff\"> <A NAME=\"top\">" - TopDoc = "\ - <h2>Please choose one of the following actions:</h2>\ - <UL>\ - <LI>\ - <A HREF=" MyPrefix "/AboutServer>About this server</A>\ - </LI><LI>\ - <A HREF=" MyPrefix "/AboutELIZA>About Eliza</A></LI>\ - <LI>\ - <A HREF=" MyPrefix \ - "/StartELIZA>Start talking to Eliza</A></LI></UL>" - TopFooter = "</BODY></HTML>" -@} -@c endfile -@end example - -@code{SetUpServer} is similar to the previous example, -except for calling another function, @code{SetUpEliza}. -This approach can be used to implement other kinds of servers. -The only changes needed to do so are hidden in the functions -@code{SetUpServer} and @code{HandleGET}. Perhaps it might be necessary to -implement other HTTP methods. -The @command{igawk} program that comes with @command{gawk} -may be useful for this process. - -When extending this example to a complete application, the first -thing to do is to implement the function @code{SetUpServer} to -initialize the HTML pages and some variables. These initializations -determine the way your HTML pages look (colors, titles, menu -items, etc.). - -@cindex GUI -The function @code{HandleGET} is a nested case selection that decides -which page the user wants to see next. Each nesting level refers to a menu -level of the GUI. Each case implements a certain action of the menu. On the -deepest level of case selection, the handler essentially knows what the -user wants and stores the answer into the variable that holds the HTML -page contents: - -@smallexample -@c file eg/network/eliza.awk -function HandleGET() @{ - # A real HTTP server would treat some parts of the URI as a file name. - # We take parts of the URI as menu choices and go on accordingly. - if(MENU[2] == "AboutServer") @{ - Document = "This is not a CGI script.\ - This is an httpd, an HTML file, and a CGI script all \ - in one GAWK script. It needs no separate www-server, \ - no installation, and no root privileges.\ - <p>To run it, do this:</p><ul>\ - <li> start this script with \"gawk -f httpserver.awk\",</li>\ - <li> and on the same host let your www browser open location\ - \"http://localhost:8080\"</li>\ - </ul>\<p>\ Details of HTTP come from:</p><ul>\ - <li>Hethmon: Illustrated Guide to HTTP</p>\ - <li>RFC 2068</li></ul><p>JK 14.9.1997</p>" - @} else if (MENU[2] == "AboutELIZA") @{ - Document = "This is an implementation of the famous ELIZA\ - program by Joseph Weizenbaum. It is written in GAWK and\ -/bin/sh: expad: command not found - @} else if (MENU[2] == "StartELIZA") @{ - gsub(/\+/, " ", GETARG["YouSay"]) - # Here we also have to substitute coded special characters - Document = "<form method=GET>" \ - "<h3>" ElizaSays(GETARG["YouSay"]) "</h3>\ - <p><input type=text name=YouSay value=\"\" size=60>\ - <br><input type=submit value=\"Tell her about it\"></p></form>" - @} -@} -@c endfile -@end smallexample - -Now we are down to the heart of ELIZA, so you can see how it works. -Initially the user does not say anything; then ELIZA resets its money -counter and asks the user to tell what comes to mind open heartedly. -The subsequent answers are converted to uppercase and stored for -later comparison. ELIZA presents the bill when being confronted with -a sentence that contains the phrase ``shut up.'' Otherwise, it looks for -keywords in the sentence, conjugates the rest of the sentence, remembers -the keyword for later use, and finally selects an answer from the set of -possible answers: - -@smallexample -@c file eg/network/eliza.awk -function ElizaSays(YouSay) @{ - if (YouSay == "") @{ - cost = 0 - answer = "HI, IM ELIZA, TELL ME YOUR PROBLEM" - @} else @{ - q = toupper(YouSay) - gsub("'", "", q) - if(q == qold) @{ - answer = "PLEASE DONT REPEAT YOURSELF !" - @} else @{ - if (index(q, "SHUT UP") > 0) @{ - answer = "WELL, PLEASE PAY YOUR BILL. ITS EXACTLY ... $"\ - int(100*rand()+30+cost/100) - @} else @{ - qold = q - w = "-" # no keyword recognized yet - for (i in k) @{ # search for keywords - if (index(q, i) > 0) @{ - w = i - break - @} - @} - if (w == "-") @{ # no keyword, take old subject - w = wold - subj = subjold - @} else @{ # find subject - subj = substr(q, index(q, w) + length(w)+1) - wold = w - subjold = subj # remember keyword and subject - @} - for (i in conj) - gsub(i, conj[i], q) # conjugation - # from all answers to this keyword, select one randomly - answer = r[indices[int(split(k[w], indices) * rand()) + 1]] - # insert subject into answer - gsub("_", subj, answer) - @} - @} - @} - cost += length(answer) # for later payment : 1 cent per character - return answer -@} -@c endfile -@end smallexample - -In the long but simple function @code{SetUpEliza}, you can see tables -for conjugation, keywords, and answers.@footnote{The version shown -here is abbreviated. The full version comes with the @command{gawk} -distribution.} The associative array @code{k} -contains indices into the array of answers @code{r}. To choose an -answer, ELIZA just picks an index randomly: - -@example -@c file eg/network/eliza.awk -function SetUpEliza() @{ - srand() - wold = "-" - subjold = " " - - # table for conjugation - conj[" ARE " ] = " AM " - conj["WERE " ] = "WAS " - conj[" YOU " ] = " I " - conj["YOUR " ] = "MY " - conj[" IVE " ] =\ - conj[" I HAVE " ] = " YOU HAVE " - conj[" YOUVE " ] =\ - conj[" YOU HAVE "] = " I HAVE " - conj[" IM " ] =\ - conj[" I AM " ] = " YOU ARE " - conj[" YOURE " ] =\ - conj[" YOU ARE " ] = " I AM " - - # table of all answers - r[1] = "DONT YOU BELIEVE THAT I CAN _" - r[2] = "PERHAPS YOU WOULD LIKE TO BE ABLE TO _ ?" -@c endfile - @dots{} -@end example -@ignore -@c file eg/network/eliza.awk - r[3] = "YOU WANT ME TO BE ABLE TO _ ?" - r[4] = "PERHAPS YOU DONT WANT TO _ " - r[5] = "DO YOU WANT TO BE ABLE TO _ ?" - r[6] = "WHAT MAKES YOU THINK I AM _ ?" - r[7] = "DOES IT PLEASE YOU TO BELIEVE I AM _ ?" - r[8] = "PERHAPS YOU WOULD LIKE TO BE _ ?" - r[9] = "DO YOU SOMETIMES WISH YOU WERE _ ?" - r[10] = "DONT YOU REALLY _ ?" - r[11] = "WHY DONT YOU _ ?" - r[12] = "DO YOU WISH TO BE ABLE TO _ ?" - r[13] = "DOES THAT TROUBLE YOU ?" - r[14] = "TELL ME MORE ABOUT SUCH FEELINGS" - r[15] = "DO YOU OFTEN FEEL _ ?" - r[16] = "DO YOU ENJOY FEELING _ ?" - r[17] = "DO YOU REALLY BELIEVE I DONT _ ?" - r[18] = "PERHAPS IN GOOD TIME I WILL _ " - r[19] = "DO YOU WANT ME TO _ ?" - r[20] = "DO YOU THINK YOU SHOULD BE ABLE TO _ ?" - r[21] = "WHY CANT YOU _ ?" - r[22] = "WHY ARE YOU INTERESTED IN WHETHER OR NOT I AM _ ?" - r[23] = "WOULD YOU PREFER IF I WERE NOT _ ?" - r[24] = "PERHAPS IN YOUR FANTASIES I AM _ " - r[25] = "HOW DO YOU KNOW YOU CANT _ ?" - r[26] = "HAVE YOU TRIED ?" - r[27] = "PERHAPS YOU CAN NOW _ " - r[28] = "DID YOU COME TO ME BECAUSE YOU ARE _ ?" - r[29] = "HOW LONG HAVE YOU BEEN _ ?" - r[30] = "DO YOU BELIEVE ITS NORMAL TO BE _ ?" - r[31] = "DO YOU ENJOY BEING _ ?" - r[32] = "WE WERE DISCUSSING YOU -- NOT ME" - r[33] = "Oh, I _" - r[34] = "YOU'RE NOT REALLY TALKING ABOUT ME, ARE YOU ?" - r[35] = "WHAT WOULD IT MEAN TO YOU, IF YOU GOT _ ?" - r[36] = "WHY DO YOU WANT _ ?" - r[37] = "SUPPOSE YOU SOON GOT _" - r[38] = "WHAT IF YOU NEVER GOT _ ?" - r[39] = "I SOMETIMES ALSO WANT _" - r[40] = "WHY DO YOU ASK ?" - r[41] = "DOES THAT QUESTION INTEREST YOU ?" - r[42] = "WHAT ANSWER WOULD PLEASE YOU THE MOST ?" - r[43] = "WHAT DO YOU THINK ?" - r[44] = "ARE SUCH QUESTIONS IN YOUR MIND OFTEN ?" - r[45] = "WHAT IS IT THAT YOU REALLY WANT TO KNOW ?" - r[46] = "HAVE YOU ASKED ANYONE ELSE ?" - r[47] = "HAVE YOU ASKED SUCH QUESTIONS BEFORE ?" - r[48] = "WHAT ELSE COMES TO MIND WHEN YOU ASK THAT ?" - r[49] = "NAMES DON'T INTEREST ME" - r[50] = "I DONT CARE ABOUT NAMES -- PLEASE GO ON" - r[51] = "IS THAT THE REAL REASON ?" - r[52] = "DONT ANY OTHER REASONS COME TO MIND ?" - r[53] = "DOES THAT REASON EXPLAIN ANYTHING ELSE ?" - r[54] = "WHAT OTHER REASONS MIGHT THERE BE ?" - r[55] = "PLEASE DON'T APOLOGIZE !" - r[56] = "APOLOGIES ARE NOT NECESSARY" - r[57] = "WHAT FEELINGS DO YOU HAVE WHEN YOU APOLOGIZE ?" - r[58] = "DON'T BE SO DEFENSIVE" - r[59] = "WHAT DOES THAT DREAM SUGGEST TO YOU ?" - r[60] = "DO YOU DREAM OFTEN ?" - r[61] = "WHAT PERSONS APPEAR IN YOUR DREAMS ?" - r[62] = "ARE YOU DISTURBED BY YOUR DREAMS ?" - r[63] = "HOW DO YOU DO ... PLEASE STATE YOUR PROBLEM" - r[64] = "YOU DON'T SEEM QUITE CERTAIN" - r[65] = "WHY THE UNCERTAIN TONE ?" - r[66] = "CAN'T YOU BE MORE POSITIVE ?" - r[67] = "YOU AREN'T SURE ?" - r[68] = "DON'T YOU KNOW ?" - r[69] = "WHY NO _ ?" - r[70] = "DON'T SAY NO, IT'S ALWAYS SO NEGATIVE" - r[71] = "WHY NOT ?" - r[72] = "ARE YOU SURE ?" - r[73] = "WHY NO ?" - r[74] = "WHY ARE YOU CONCERNED ABOUT MY _ ?" - r[75] = "WHAT ABOUT YOUR OWN _ ?" - r[76] = "CAN'T YOU THINK ABOUT A SPECIFIC EXAMPLE ?" - r[77] = "WHEN ?" - r[78] = "WHAT ARE YOU THINKING OF ?" - r[79] = "REALLY, ALWAYS ?" - r[80] = "DO YOU REALLY THINK SO ?" - r[81] = "BUT YOU ARE NOT SURE YOU _ " - r[82] = "DO YOU DOUBT YOU _ ?" - r[83] = "IN WHAT WAY ?" - r[84] = "WHAT RESEMBLANCE DO YOU SEE ?" - r[85] = "WHAT DOES THE SIMILARITY SUGGEST TO YOU ?" - r[86] = "WHAT OTHER CONNECTION DO YOU SEE ?" - r[87] = "COULD THERE REALLY BE SOME CONNECTIONS ?" - r[88] = "HOW ?" - r[89] = "YOU SEEM QUITE POSITIVE" - r[90] = "ARE YOU SURE ?" - r[91] = "I SEE" - r[92] = "I UNDERSTAND" - r[93] = "WHY DO YOU BRING UP THE TOPIC OF FRIENDS ?" - r[94] = "DO YOUR FRIENDS WORRY YOU ?" - r[95] = "DO YOUR FRIENDS PICK ON YOU ?" - r[96] = "ARE YOU SURE YOU HAVE ANY FRIENDS ?" - r[97] = "DO YOU IMPOSE ON YOUR FRIENDS ?" - r[98] = "PERHAPS YOUR LOVE FOR FRIENDS WORRIES YOU" - r[99] = "DO COMPUTERS WORRY YOU ?" - r[100] = "ARE YOU TALKING ABOUT ME IN PARTICULAR ?" - r[101] = "ARE YOU FRIGHTENED BY MACHINES ?" - r[102] = "WHY DO YOU MENTION COMPUTERS ?" - r[103] = "WHAT DO YOU THINK MACHINES HAVE TO DO WITH YOUR PROBLEMS ?" - r[104] = "DON'T YOU THINK COMPUTERS CAN HELP PEOPLE ?" - r[105] = "WHAT IS IT ABOUT MACHINES THAT WORRIES YOU ?" - r[106] = "SAY, DO YOU HAVE ANY PSYCHOLOGICAL PROBLEMS ?" - r[107] = "WHAT DOES THAT SUGGEST TO YOU ?" - r[108] = "I SEE" - r[109] = "IM NOT SURE I UNDERSTAND YOU FULLY" - r[110] = "COME COME ELUCIDATE YOUR THOUGHTS" - r[111] = "CAN YOU ELABORATE ON THAT ?" - r[112] = "THAT IS QUITE INTERESTING" - r[113] = "WHY DO YOU HAVE PROBLEMS WITH MONEY ?" - r[114] = "DO YOU THINK MONEY IS EVERYTHING ?" - r[115] = "ARE YOU SURE THAT MONEY IS THE PROBLEM ?" - r[116] = "I THINK WE WANT TO TALK ABOUT YOU, NOT ABOUT ME" - r[117] = "WHAT'S ABOUT ME ?" - r[118] = "WHY DO YOU ALWAYS BRING UP MY NAME ?" -@c endfile -@end ignore - -@example -@c file eg/network/eliza.awk - # table for looking up answers that - # fit to a certain keyword - k["CAN YOU"] = "1 2 3" - k["CAN I"] = "4 5" - k["YOU ARE"] =\ - k["YOURE"] = "6 7 8 9" -@c endfile - @dots{} -@end example -@ignore -@c file eg/network/eliza.awk - k["I DONT"] = "10 11 12 13" - k["I FEEL"] = "14 15 16" - k["WHY DONT YOU"] = "17 18 19" - k["WHY CANT I"] = "20 21" - k["ARE YOU"] = "22 23 24" - k["I CANT"] = "25 26 27" - k["I AM"] =\ - k["IM "] = "28 29 30 31" - k["YOU "] = "32 33 34" - k["I WANT"] = "35 36 37 38 39" - k["WHAT"] =\ - k["HOW"] =\ - k["WHO"] =\ - k["WHERE"] =\ - k["WHEN"] =\ - k["WHY"] = "40 41 42 43 44 45 46 47 48" - k["NAME"] = "49 50" - k["CAUSE"] = "51 52 53 54" - k["SORRY"] = "55 56 57 58" - k["DREAM"] = "59 60 61 62" - k["HELLO"] =\ - k["HI "] = "63" - k["MAYBE"] = "64 65 66 67 68" - k[" NO "] = "69 70 71 72 73" - k["YOUR"] = "74 75" - k["ALWAYS"] = "76 77 78 79" - k["THINK"] = "80 81 82" - k["LIKE"] = "83 84 85 86 87 88 89" - k["YES"] = "90 91 92" - k["FRIEND"] = "93 94 95 96 97 98" - k["COMPUTER"] = "99 100 101 102 103 104 105" - k["-"] = "106 107 108 109 110 111 112" - k["MONEY"] = "113 114 115" - k["ELIZA"] = "116 117 118" -@c endfile -@end ignore -@example -@c file eg/network/eliza.awk -@} -@c endfile -@end example - -@cindex Humphrys, Mark -@cindex ELIZA program -@cindex Yahoo! -Some interesting remarks and details (including the original source code -of ELIZA) are found on Mark Humphrys' home page. Yahoo! also has a -page with a collection of ELIZA-like programs. Many of them are written -in Java, some of them disclosing the Java source code, and a few even -explain how to modify the Java source code. - -@node Caveats, Challenges, Simple Server, Using Networking -@section Network Programming Caveats - -By now it should be clear -that debugging a networked application is more -complicated than debugging a single-process single-hosted application. -The behavior of a networked application sometimes looks non-causal because -it is not reproducible in a strong sense. Whether a network application -works or not sometimes depends on the following: - -@itemize @bullet -@item -How crowded the underlying network is. - -@item -If the party at the other end is running or not. - -@item -The state of the party at the other end. -@end itemize - -@cindex network -The most difficult problems for a beginner arise from the hidden states of the -underlying network. After closing a TCP connection, it's often necessary to wait -a short while before reopening the connection. Even more difficult is the -establishment of a connection that previously ended with a ``broken pipe.'' -Those connections have to ``time out'' for a minute or so -before they can reopen. -Check this with the command @samp{netstat -a}, which -provides a list of still ``active'' connections. - -@node Challenges, , Caveats, Using Networking -@section Where To Go From Here - -@cindex Loebner, Hugh -@cindex Contest -Now, you have learned enough to build your own application. You could, -for example, take part in the -Loebner Contest -to win $100,000. The Loebner Prize is the first formal instantiation of -a Turing Test. Hugh Loebner agreed with The Cambridge Center for Behavioral -Studies to underwrite a contest designed to implement the Turing Test. -Dr.@: Loebner pledged a Grand Prize of $100,000 for the first computer whose -responses were indistinguishable from a human's. Each year an annual prize -of $2000 and a bronze medal is awarded to the @emph{most} human computer. -The winner of the annual contest is the best entry relative to other entries -that year, irrespective of how good it is in an absolute sense. Here is -an example of a conversation with the winning program of 1997: - -@smallexample -(c)1997 Cambridge Center For Behavioral Studies all rights reserved [CONVERSE] -[Intelligent Research Ltd.] Start at: [1997/04/29 08:59:40] -PROGRAM[10:39:20]Hi, my name is Catherine, what's yours? -JUDGE02[10:39:36]My name is Nathan. -PROGRAM[10:39:37]Well, it's nice to meet you, Nathan. -PROGRAM[10:39:45]Did you see that story on CNN last night about the lesbian -PROGRAM[10:40:03]couple who came out at a White House party on Sunday? -JUDGE02[10:40:45]No. I'm sorry I missed that. How did the Prez take it? -PROGRAM[10:40:47]Ellen Degeneres was one of them - she was kissing her lover -PROGRAM[10:41:08]in front of Bill Clinton. Do you think he was very -PROGRAM[10:41:21]embarrassed? -JUDGE02[10:41:37]You tell me. I didn't see it. -PROGRAM[10:41:38]Are you still a Friend of Bill Clinton? -JUDGE02[10:42:22]Yes. I would like to talk about something else. Do you know anything about sculpture. -PROGRAM[10:42:24]Well, I suppose he's doing his best like most of us, but he -PROGRAM[10:42:35]does seem to attract an UNUSUAL amount of bad publicity, -PROGRAM[10:42:51]wouldnt you say--can it all be bad luck? -JUDGE02[10:43:20]I asked you if you knew anything about sculpture. -PROGRAM[10:43:21]OK, but what happens now--does it all drift on, with -PROGRAM[10:43:36]everyone waiting to see how much his friends have said about -PROGRAM[10:43:47]him under threat of lots and lots of jail-or will something -PROGRAM[10:44:01]dramatic actually happen? -JUDGE02[10:44:19]Do you know what sculpture is? -PROGRAM[10:44:20]Well, youre more a fan of his than I am I guess---though I -PROGRAM[10:44:34]cant SWEAR what Id have done in that locked hotel room, I -PROGRAM[10:44:51]must say. -JUDGE02[10:45:26]Have you ever been to the Salmagundi Club? -PROGRAM[10:45:29]Never. to go back to the original discussion Time to get -PROGRAM[10:45:43]off this one I think and onto something more down to earth! -JUDGE02[10:46:20]What is more down to earth than art? -PROGRAM[10:46:21]I don't know what -- to be honest. - -PROGRAM[10:46:30]I suppose it depends on where you're coming from, but as the -PROGRAM[10:46:46]song goes in My Fair Lady, "Why can't a woman be more like a -PROGRAM[10:47:03]man? " -JUDGE02[10:47:52]I liked that song but I prefer "Why can't children be more like we are" -JUDGE02[10:48:10]from another play the name of which I cannot remember. -PROGRAM[10:48:12]What, if anything, do you enjoy about being a man? -JUDGE02[10:49:01]I would enjoy all aspects of life no matter what my gender. -PROGRAM[10:49:02]Do you think of yourself as being attractive? -@end smallexample - -@cindex Clinton, Bill -This program insists on always speaking about the same story around Bill -Clinton. You see, even a program with a rather narrow mind can behave so -much like a human being that it can win this prize. It is quite common to -let these programs talk to each other via network connections. But during the -competition itself, the program and its computer have to be present at the -place the competition is held. We all would love to see a @command{gawk} -program win in such an event. Maybe it is up to you to accomplish this? - -Some other ideas for useful networked applications: -@itemize @bullet -@item -Read the file @file{doc/awkforai.txt} in the @command{gawk} distribution. -It was written by Ronald P.@: Loui (Associate Professor of -Computer Science, at Washington University in St. Louis, -@email{loui@@ai.wustl.edu}) and summarizes why -he teaches @command{gawk} to students of Artificial Intelligence. Here are -some passages from the text: - -@cindex AI -@cindex PROLOG -@cindex Loui, Ronald P. -@cindex agent -@quotation -The GAWK manual can -be consumed in a single lab session and the language can be mastered by -the next morning by the average student. GAWK's automatic -initialization, implicit coercion, I/O support and lack of pointers -forgive many of the mistakes that young programmers are likely to make. -Those who have seen C but not mastered it are happy to see that GAWK -retains some of the same sensibilities while adding what must be -regarded as spoonsful of syntactic sugar.@* -@dots{}@* -@cindex robot -There are further simple answers. Probably the best is the fact that -increasingly, undergraduate AI programming is involving the Web. Oren -Etzioni (University of Washington, Seattle) has for a while been arguing -that the ``softbot'' is replacing the mechanical engineers' robot as the -most glamorous AI testbed. If the artifact whose behavior needs to be -controlled in an intelligent way is the software agent, then a language -that is well-suited to controlling the software environment is the -appropriate language. That would imply a scripting language. If the -robot is KAREL, then the right language is ``turn left; turn right.'' If -the robot is Netscape, then the right language is something that can -generate @samp{netscape -remote 'openURL(http://cs.wustl.edu/~loui)'} with -elan.@* -@dots{}@* -AI programming requires high-level thinking. There have always been a few -gifted programmers who can write high-level programs in assembly language. -Most however need the ambient abstraction to have a higher floor.@* -@dots{}@* -Second, inference is merely the expansion of notation. No matter whether -the logic that underlies an AI program is fuzzy, probabilistic, deontic, -defeasible, or deductive, the logic merely defines how strings can be -transformed into other strings. A language that provides the best -support for string processing in the end provides the best support for -logic, for the exploration of various logics, and for most forms of -symbolic processing that AI might choose to call ``reasoning'' instead of -``logic.'' The implication is that PROLOG, which saves the AI programmer -from having to write a unifier, saves perhaps two dozen lines of GAWK -code at the expense of strongly biasing the logic and representational -expressiveness of any approach. -@end quotation - -Now that @command{gawk} itself can connect to the Internet, it should be obvious -that it is suitable for writing intelligent web agents. - -@item -@command{awk} is strong at pattern recognition and string processing. -So, it is well suited to the classic problem of language translation. -A first try could be a program that knows the 100 most frequent English -words and their counterparts in German or French. The service could be -implemented by regularly reading email with the program above, replacing -each word by its translation and sending the translation back via SMTP. -Users would send English email to their translation service and get -back a translated email message in return. As soon as this works, -more effort can be spent on a real translation program. - -@item -Another dialogue-oriented application (on the verge -of ridicule) is the email ``support service.'' Troubled customers write an -email to an automatic @command{gawk} service that reads the email. It looks -for keywords in the mail and assembles a reply email accordingly. By carefully -investigating the email header, and repeating these keywords through the -reply email, it is rather simple to give the customer a feeling that -someone cares. Ideally, such a service would search a database of previous -cases for solutions. If none exists, the database could, for example, consist -of all the newsgroups, mailing lists and FAQs on the Internet. -@end itemize - -@node Some Applications and Techniques, Links, Using Networking, Top -@comment node-name, next, previous, up - -@chapter Some Applications and Techniques -In this @value{CHAPTER}, we look at a number of self-contained -scripts, with an emphasis on concise networking. Along the way, we -work towards creating building blocks that encapsulate often needed -functions of the networking world, show new techniques that -broaden the scope of problems that can be solved with @command{gawk}, and -explore leading edge technology that may shape the future of networking. - -We often refer to the site-independent core of the server that -we built in -@ref{Simple Server, ,A Simple Web Server}. -When building new and non-trivial servers, we -always copy this building block and append new instances of the two -functions @code{SetUpServer} and @code{HandleGET}. - -This makes a lot of sense, since -this scheme of event-driven -execution provides @command{gawk} with an interface to the most widely -accepted standard for GUIs: the web browser. Now, @command{gawk} can even rival -Tcl/Tk. - -@cindex Tcl/Tk -@cindex JavaScript -Tcl and @command{gawk} have much in common. Both are simple scripting languages -that allow us to quickly solve problems with short programs. But Tcl has Tk -on top of it and @command{gawk} had nothing comparable up to now. While Tcl -needs a large and ever changing library (Tk, which was bound to the X Window -System until recently), @command{gawk} needs just the networking interface -and some kind of browser on the client's side. Besides better portability, -the most important advantage of this approach (embracing well-established -standards such HTTP and HTML) is that @emph{we do not need to change the -language}. We let others do the work of fighting over protocols and standards. -We can use HTML, JavaScript, VRML, or whatever else comes along to do our work. - -@menu -* PANIC:: An Emergency Web Server. -* GETURL:: Retrieving Web Pages. -* REMCONF:: Remote Configuration Of Embedded Systems. -* URLCHK:: Look For Changed Web Pages. -* WEBGRAB:: Extract Links From A Page. -* STATIST:: Graphing A Statistical Distribution. -* MAZE:: Walking Through A Maze In Virtual Reality. -* MOBAGWHO:: A Simple Mobile Agent. -* STOXPRED:: Stock Market Prediction As A Service. -* PROTBASE:: Searching Through A Protein Database. -@end menu - -@node PANIC, GETURL, Some Applications and Techniques, Some Applications and Techniques -@section PANIC: an Emergency Web Server -@cindex PANIC program -At first glance, the @code{"Hello, world"} example in -@ref{Primitive Service, ,A Primitive Web Service}, -seems useless. By adding just a few lines, we can turn it into something useful. - -The PANIC program tells everyone who connects that the local -site is not working. When a web server breaks down, it makes a difference -if customers get a strange ``network unreachable'' message, or a short message -telling them that the server has a problem. In such an emergency, -the hard disk and everything on it (including the regular web service) may -be unavailable. Rebooting the web server off a diskette makes sense in this -setting. - -To use the PANIC program as an emergency web server, all you need are the -@command{gawk} executable and the program below on a diskette. By default, -it connects to port 8080. A different value may be supplied on the -command line: - -@example -@c file eg/network/panic.awk -BEGIN @{ - RS = ORS = "\r\n" - if (MyPort == 0) MyPort = 8080 - HttpService = "/inet/tcp/" MyPort "/0/0" - Hello = "<HTML><HEAD><TITLE>Out Of Service</TITLE>" \ - "</HEAD><BODY><H1>" \ - "This site is temporarily out of service." \ - "</H1></BODY></HTML>" - Len = length(Hello) + length(ORS) - while ("awk" != "complex") @{ - print "HTTP/1.0 200 OK" |& HttpService - print "Content-Length: " Len ORS |& HttpService - print Hello |& HttpService - while ((HttpService |& getline) > 0) - continue; - close(HttpService) - @} -@} -@c endfile -@end example - -@node GETURL, REMCONF, PANIC, Some Applications and Techniques -@section GETURL: Retrieving Web Pages -@cindex GETURL program -@cindex robot -GETURL is a versatile building block for shell scripts that need to retrieve -files from the Internet. It takes a web address as a command-line parameter and -tries to retrieve the contents of this address. The contents are printed -to standard output, while the header is printed to @file{/dev/stderr}. -A surrounding shell script -could analyze the contents and extract the text or the links. An ASCII -browser could be written around GETURL. But more interestingly, web robots are -straightforward to write on top of GETURL. On the Internet, you can find -several programs of the same name that do the same job. They are usually -much more complex internally and at least 10 times longer. - -At first, GETURL checks if it was called with exactly one web address. -Then, it checks if the user chose to use a special proxy server whose name -is handed over in a variable. By default, it is assumed that the local -machine serves as proxy. GETURL uses the @code{GET} method by default -to access the web page. By handing over the name of a different method -(such as @code{HEAD}), it is possible to choose a different behavior. With -the @code{HEAD} method, the user does not receive the body of the page -content, but does receive the header: - -@example -@c file eg/network/geturl.awk -BEGIN @{ - if (ARGC != 2) @{ - print "GETURL - retrieve Web page via HTTP 1.0" - print "IN:\n the URL as a command-line parameter" - print "PARAM(S):\n -v Proxy=MyProxy" - print "OUT:\n the page content on stdout" - print " the page header on stderr" - print "JK 16.05.1997" - print "ADR 13.08.2000" - exit - @} - URL = ARGV[1]; ARGV[1] = "" - if (Proxy == "") Proxy = "127.0.0.1" - if (ProxyPort == 0) ProxyPort = 80 - if (Method == "") Method = "GET" - HttpService = "/inet/tcp/0/" Proxy "/" ProxyPort - ORS = RS = "\r\n\r\n" - print Method " " URL " HTTP/1.0" |& HttpService - HttpService |& getline Header - print Header > "/dev/stderr" - while ((HttpService |& getline) > 0) - printf "%s", $0 - close(HttpService) -@} -@c endfile -@end example - -This program can be changed as needed, but be careful with the last lines. -Make sure transmission of binary data is not corrupted by additional line -breaks. Even as it is now, the byte sequence @code{"\r\n\r\n"} would -disappear if it were contained in binary data. Don't get caught in a -trap when trying a quick fix on this one. - -@node REMCONF, URLCHK, GETURL, Some Applications and Techniques -@section REMCONF: Remote Configuration of Embedded Systems -@cindex REMCONF program -@cindex Linux -@cindex GNU/Linux -@cindex Yahoo! -Today, you often find powerful processors in embedded systems. Dedicated -network routers and controllers for all kinds of machinery are examples -of embedded systems. Processors like the Intel 80x86 or the AMD Elan are -able to run multitasking operating systems, such as XINU or GNU/Linux -in embedded PCs. These systems are small and usually do not have -a keyboard or a display. Therefore it is difficult to set up their -configuration. There are several widespread ways to set them up: - -@itemize @bullet -@item -DIP switches - -@item -Read Only Memories such as EPROMs - -@item -Serial lines or some kind of keyboard - -@item -Network connections via @command{telnet} or SNMP - -@item -HTTP connections with HTML GUIs -@end itemize - -In this @value{SECTION}, we look at a solution that uses HTTP connections -to control variables of an embedded system that are stored in a file. -Since embedded systems have tight limits on resources like memory, -it is difficult to employ advanced techniques such as SNMP and HTTP -servers. @command{gawk} fits in quite nicely with its single executable -which needs just a short script to start working. -The following program stores the variables in a file, and a concurrent -process in the embedded system may read the file. The program uses the -site-independent part of the simple web server that we developed in -@ref{Interacting Service, ,A Web Service with Interaction}. -As mentioned there, all we have to do is to write two new procedures -@code{SetUpServer} and @code{HandleGET}: - -@smallexample -@c file eg/network/remconf.awk -function SetUpServer() @{ - TopHeader = "<HTML><title>Remote Configuration</title>" - TopDoc = "<BODY>\ - <h2>Please choose one of the following actions:</h2>\ - <UL>\ - <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\ - <LI><A HREF=" MyPrefix "/ReadConfig>Read Configuration</A></LI>\ - <LI><A HREF=" MyPrefix "/CheckConfig>Check Configuration</A></LI>\ - <LI><A HREF=" MyPrefix "/ChangeConfig>Change Configuration</A></LI>\ - <LI><A HREF=" MyPrefix "/SaveConfig>Save Configuration</A></LI>\ - </UL>" - TopFooter = "</BODY></HTML>" - if (ConfigFile == "") ConfigFile = "config.asc" -@} -@c endfile -@end smallexample - -The function @code{SetUpServer} initializes the top level HTML texts -as usual. It also initializes the name of the file that contains the -configuration parameters and their values. In case the user supplies -a name from the command line, that name is used. The file is expected to -contain one parameter per line, with the name of the parameter in -column one and the value in column two. - -The function @code{HandleGET} reflects the structure of the menu -tree as usual. The first menu choice tells the user what this is all -about. The second choice reads the configuration file line by line -and stores the parameters and their values. Notice that the record -separator for this file is @code{"\n"}, in contrast to the record separator -for HTTP. The third menu choice builds an HTML table to show -the contents of the configuration file just read. The fourth choice -does the real work of changing parameters, and the last one just saves -the configuration into a file: - -@smallexample -@c file eg/network/remconf.awk -function HandleGET() @{ - if(MENU[2] == "AboutServer") @{ - Document = "This is a GUI for remote configuration of an\ - embedded system. It is is implemented as one GAWK script." - @} else if (MENU[2] == "ReadConfig") @{ - RS = "\n" - while ((getline < ConfigFile) > 0) - config[$1] = $2; - close(ConfigFile) - RS = "\r\n" - Document = "Configuration has been read." - @} else if (MENU[2] == "CheckConfig") @{ - Document = "<TABLE BORDER=1 CELLPADDING=5>" - for (i in config) - Document = Document "<TR><TD>" i "</TD>" \ - "<TD>" config[i] "</TD></TR>" - Document = Document "</TABLE>" - @} else if (MENU[2] == "ChangeConfig") @{ - if ("Param" in GETARG) @{ # any parameter to set? - if (GETARG["Param"] in config) @{ # is parameter valid? - config[GETARG["Param"]] = GETARG["Value"] - Document = (GETARG["Param"] " = " GETARG["Value"] ".") - @} else @{ - Document = "Parameter <b>" GETARG["Param"] "</b> is invalid." - @} - @} else @{ - Document = "<FORM method=GET><h4>Change one parameter</h4>\ - <TABLE BORDER CELLPADDING=5>\ - <TR><TD>Parameter</TD><TD>Value</TD></TR>\ - <TR><TD><input type=text name=Param value=\"\" size=20></TD>\ - <TD><input type=text name=Value value=\"\" size=40></TD>\ - </TR></TABLE><input type=submit value=\"Set\"></FORM>" - @} - @} else if (MENU[2] == "SaveConfig") @{ - for (i in config) - printf("%s %s\n", i, config[i]) > ConfigFile - close(ConfigFile) - Document = "Configuration has been saved." - @} -@} -@c endfile -@end smallexample - -@cindex MiniSQL -We could also view the configuration file as a database. From this -point of view, the previous program acts like a primitive database server. -Real SQL database systems also make a service available by providing -a TCP port that clients can connect to. But the application level protocols -they use are usually proprietary and also change from time to time. -This is also true for the protocol that -MiniSQL uses. - -@node URLCHK, WEBGRAB, REMCONF, Some Applications and Techniques -@section URLCHK: Look for Changed Web Pages -@cindex URLCHK program -Most people who make heavy use of Internet resources have a large -bookmark file with pointers to interesting web sites. It is impossible -to regularly check by hand if any of these sites have changed. A program -is needed to automatically look at the headers of web pages and tell -which ones have changed. URLCHK does the comparison after using GETURL -with the @code{HEAD} method to retrieve the header. - -Like GETURL, this program first checks that it is called with exactly -one command-line parameter. URLCHK also takes the same command-line variables -@code{Proxy} and @code{ProxyPort} as GETURL, -because these variables are handed over to GETURL for each URL -that gets checked. The one and only parameter is the name of a file that -contains one line for each URL. In the first column, we find the URL, and -the second and third columns hold the length of the URL's body when checked -for the two last times. Now, we follow this plan: - -@enumerate -@item -Read the URLs from the file and remember their most recent lengths - -@item -Delete the contents of the file - -@item -For each URL, check its new length and write it into the file - -@item -If the most recent and the new length differ, tell the user -@end enumerate - -It may seem a bit peculiar to read the URLs from a file together -with their two most recent lengths, but this approach has several -advantages. You can call the program again and again with the same -file. After running the program, you can regenerate the changed URLs -by extracting those lines that differ in their second and third columns: - -@c inspired by URLCHK in iX 5/97 166. -@smallexample -@c file eg/network/urlchk.awk -BEGIN @{ - if (ARGC != 2) @{ - print "URLCHK - check if URLs have changed" - print "IN:\n the file with URLs as a command-line parameter" - print " file contains URL, old length, new length" - print "PARAMS:\n -v Proxy=MyProxy -v ProxyPort=8080" - print "OUT:\n same as file with URLs" - print "JK 02.03.1998" - exit - @} - URLfile = ARGV[1]; ARGV[1] = "" - if (Proxy != "") Proxy = " -v Proxy=" Proxy - if (ProxyPort != "") ProxyPort = " -v ProxyPort=" ProxyPort - while ((getline < URLfile) > 0) - Length[$1] = $3 + 0 - close(URLfile) # now, URLfile is read in and can be updated - GetHeader = "gawk " Proxy ProxyPort " -v Method=\"HEAD\" -f geturl.awk " - for (i in Length) @{ - GetThisHeader = GetHeader i " 2>&1" - while ((GetThisHeader | getline) > 0) - if (toupper($0) ~ /CONTENT-LENGTH/) NewLength = $2 + 0 - close(GetThisHeader) - print i, Length[i], NewLength > URLfile - if (Length[i] != NewLength) # report only changed URLs - print i, Length[i], NewLength - @} - close(URLfile) -@} -@c endfile -@end smallexample - -Another thing that may look strange is the way GETURL is called. -Before calling GETURL, we have to check if the proxy variables need -to be passed on. If so, we prepare strings that will become part -of the command line later. In @code{GetHeader}, we store these strings -together with the longest part of the command line. Later, in the loop -over the URLs, @code{GetHeader} is appended with the URL and a redirection -operator to form the command that reads the URL's header over the Internet. -GETURL always produces the headers over @file{/dev/stderr}. That is -the reason why we need the redirection operator to have the header -piped in. - -This program is not perfect because it assumes that changing URLs -results in changed lengths, which is not necessarily true. A more -advanced approach is to look at some other header line that -holds time information. But, as always when things get a bit more -complicated, this is left as an exercise to the reader. - -@node WEBGRAB, STATIST, URLCHK, Some Applications and Techniques -@section WEBGRAB: Extract Links from a Page -@cindex WEBGRAB program -@c Inspired by iX 1/98 157. -@cindex robot -Sometimes it is necessary to extract links from web pages. -Browsers do it, web robots do it, and sometimes even humans do it. -Since we have a tool like GETURL at hand, we can solve this problem with -some help from the Bourne shell: - -@example -@c file eg/network/webgrab.awk -BEGIN @{ RS = "http://[#%&\\+\\-\\./0-9\\:;\\?A-Z_a-z\\~]*" @} -RT != "" @{ - command = ("gawk -v Proxy=MyProxy -f geturl.awk " RT \ - " > doc" NR ".html") - print command -@} -@c endfile -@end example - -Notice that the regular expression for URLs is rather crude. A precise -regular expression is much more complex. But this one works -rather well. One problem is that it is unable to find internal links of -an HTML document. Another problem is that -@samp{ftp}, @samp{telnet}, @samp{news}, @samp{mailto}, and other kinds -of links are missing in the regular expression. -However, it is straightforward to add them, if doing so is necessary for other tasks. - -This program reads an HTML file and prints all the HTTP links that it finds. -It relies on @command{gawk}'s ability to use regular expressions as record -separators. With @code{RS} set to a regular expression that matches links, -the second action is executed each time a non-empty link is found. -We can find the matching link itself in @code{RT}. - -The action could use the @code{system} function to let another GETURL -retrieve the page, but here we use a different approach. -This simple program prints shell commands that can be piped into @command{sh} -for execution. This way it is possible to first extract -the links, wrap shell commands around them, and pipe all the shell commands -into a file. After editing the file, execution of the file retrieves -exactly those files that we really need. In case we do not want to edit, -we can retrieve all the pages like this: - -@smallexample -gawk -f geturl.awk http://www.suse.de | gawk -f webgrab.awk | sh -@end smallexample - -@cindex Microsoft Windows -After this, you will find the contents of all referenced documents in -files named @file{doc*.html} even if they do not contain HTML code. -The most annoying thing is that we always have to pass the proxy to -GETURL. If you do not like to see the headers of the web pages -appear on the screen, you can redirect them to @file{/dev/null}. -Watching the headers appear can be quite interesting, because -it reveals -interesting details such as which web server the companies use. -Now, it is clear how the clever marketing people -use web robots to determine the -market shares -of Microsoft and Netscape in the web server market. - -Port 80 of any web server is like a small hole in a repellent firewall. -After attaching a browser to port 80, we usually catch a glimpse -of the bright side of the server (its home page). With a tool like GETURL -at hand, we are able to discover some of the more concealed -or even ``indecent'' services (i.e., lacking conformity to standards of quality). -It can be exciting to see the fancy CGI scripts that lie -there, revealing the inner workings of the server, ready to be called: - -@itemize @bullet -@item -With a command such as: - -@example -gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/ -@end example - -some servers give you a directory listing of the CGI files. -Knowing the names, you can try to call some of them and watch -for useful results. Sometimes there are executables in such directories -(such as Perl interpreters) that you may call remotely. If there are -subdirectories with configuration data of the web server, this can also -be quite interesting to read. - -@item -@cindex apache -The well-known Apache web server usually has its CGI files in the -directory @file{/cgi-bin}. There you can often find the scripts -@file{test-cgi} and @file{printenv}. Both tell you some things -about the current connection and the installation of the web server. -Just call: - -@smallexample -gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/test-cgi -gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/printenv -@end smallexample - -@item -Sometimes it is even possible to retrieve system files like the web -server's log file---possibly containing customer data---or even the file -@file{/etc/passwd}. -(We don't recommend this!) -@end itemize - -@strong{Caution:} -Although this may sound funny or simply irrelevant, we are talking about -severe security holes. Try to explore your own system this way and make -sure that none of the above reveals too much information about your system. - -@node STATIST, MAZE, WEBGRAB, Some Applications and Techniques -@section STATIST: Graphing a Statistical Distribution -@cindex STATIST program - -@cindex GNUPlot utility -@cindex image format -@cindex @file{gif} image format -@cindex @file{png} image format -@cindex @file{ps} image format -@cindex Boutell, Thomas -@iftex -@image{statist,3in} -@end iftex -In the HTTP server examples we've shown thus far, we never present an image -to the browser and its user. Presenting images is one task. Generating -images that reflect some user input and presenting these dynamically -generated images is another. In this @value{SECTION}, we use GNUPlot -for generating @file{.png}, @file{.ps}, or @file{.gif} -files.@footnote{Due to licensing problems, the default -installation of GNUPlot disables the generation of @file{.gif} files. -If your installed version does not accept @samp{set term gif}, -just download and install the most recent version of GNUPlot and the -@uref{http://www.boutell.com/gd/, GD library} -by Thomas Boutell. -Otherwise you still have the chance to generate some -ASCII-art style images with GNUPlot by using @samp{set term dumb}. -(We tried it and it worked.)} - -The program we develop takes the statistical parameters of two samples -and computes the t-test statistics. As a result, we get the probabilities -that the means and the variances of both samples are the same. In order to -let the user check plausibility, the program presents an image of the -distributions. The statistical computation follows -@cite{Numerical Recipes in C: The Art of Scientific Computing} -by William H.@: Press, Saul A.@: Teukolsky, William T.@: Vetterling, and Brian P. Flannery. -Since @command{gawk} does not have a built-in function -for the computation of the beta function, we use the @code{ibeta} function -of GNUPlot. As a side effect, we learn how to use GNUPlot as a -sophisticated calculator. The comparison of means is done as in @code{tutest}, -paragraph 14.2, page 613, and the comparison of variances is done as in @code{ftest}, -page 611 in @cite{Numerical Recipes}. -@cindex Numerical Recipes - -As usual, we take the site-independent code for servers and append -our own functions @code{SetUpServer} and @code{HandleGET}: - -@smallexample -@c file eg/network/statist.awk -function SetUpServer() @{ - TopHeader = "<HTML><title>Statistics with GAWK</title>" - TopDoc = "<BODY>\ - <h2>Please choose one of the following actions:</h2>\ - <UL>\ - <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\ - <LI><A HREF=" MyPrefix "/EnterParameters>Enter Parameters</A></LI>\ - </UL>" - TopFooter = "</BODY></HTML>" - GnuPlot = "gnuplot 2>&1" - m1=m2=0; v1=v2=1; n1=n2=10 -@} -@c endfile -@end smallexample - -Here, you see the menu structure that the user sees. Later, we -will see how the program structure of the @code{HandleGET} function -reflects the menu structure. What is missing here is the link for the -image we generate. In an event-driven environment, request, -generation, and delivery of images are separated. - -Notice the way we initialize the @code{GnuPlot} command string for -the pipe. By default, -GNUPlot outputs the generated image via standard output, as well as -the results of @code{print}(ed) calculations via standard error. -The redirection causes standard error to be mixed into standard -output, enabling us to read results of calculations with @code{getline}. -By initializing the statistical parameters with some meaningful -defaults, we make sure the user gets an image the first time -he uses the program. - -@cindex JavaScript -Following is the rather long function @code{HandleGET}, which -implements the contents of this service by reacting to the different -kinds of requests from the browser. Before you start playing with -this script, make sure that your browser supports JavaScript and that it also -has this option switched on. The script uses a short snippet of -JavaScript code for delayed opening of a window with an image. -A more detailed explanation follows: - -@smallexample -@c file eg/network/statist.awk -function HandleGET() @{ - if(MENU[2] == "AboutServer") @{ - Document = "This is a GUI for a statistical computation.\ - It compares means and variances of two distributions.\ - It is implemented as one GAWK script and uses GNUPLOT." - @} else if (MENU[2] == "EnterParameters") @{ - Document = "" - if ("m1" in GETARG) @{ # are there parameters to compare? - Document = Document "<SCRIPT LANGUAGE=\"JavaScript\">\ - setTimeout(\"window.open(\\\"" MyPrefix "/Image" systime()\ - "\\\",\\\"dist\\\", \\\"status=no\\\");\", 1000); </SCRIPT>" - m1 = GETARG["m1"]; v1 = GETARG["v1"]; n1 = GETARG["n1"] - m2 = GETARG["m2"]; v2 = GETARG["v2"]; n2 = GETARG["n2"] - t = (m1-m2)/sqrt(v1/n1+v2/n2) - df = (v1/n1+v2/n2)*(v1/n1+v2/n2)/((v1/n1)*(v1/n1)/(n1-1) \ - + (v2/n2)*(v2/n2) /(n2-1)) - if (v1>v2) @{ - f = v1/v2 - df1 = n1 - 1 - df2 = n2 - 1 - @} else @{ - f = v2/v1 - df1 = n2 - 1 - df2 = n1 - 1 - @} - print "pt=ibeta(" df/2 ",0.5," df/(df+t*t) ")" |& GnuPlot - print "pF=2.0*ibeta(" df2/2 "," df1/2 "," \ - df2/(df2+df1*f) ")" |& GnuPlot - print "print pt, pF" |& GnuPlot - RS="\n"; GnuPlot |& getline; RS="\r\n" # $1 is pt, $2 is pF - print "invsqrt2pi=1.0/sqrt(2.0*pi)" |& GnuPlot - print "nd(x)=invsqrt2pi/sd*exp(-0.5*((x-mu)/sd)**2)" |& GnuPlot - print "set term png small color" |& GnuPlot - #print "set term postscript color" |& GnuPlot - #print "set term gif medium size 320,240" |& GnuPlot - print "set yrange[-0.3:]" |& GnuPlot - print "set label 'p(m1=m2) =" $1 "' at 0,-0.1 left" |& GnuPlot - print "set label 'p(v1=v2) =" $2 "' at 0,-0.2 left" |& GnuPlot - print "plot mu=" m1 ",sd=" sqrt(v1) ", nd(x) title 'sample 1',\ - mu=" m2 ",sd=" sqrt(v2) ", nd(x) title 'sample 2'" |& GnuPlot - print "quit" |& GnuPlot - GnuPlot |& getline Image - while ((GnuPlot |& getline) > 0) - Image = Image RS $0 - close(GnuPlot) - @} - Document = Document "\ - <h3>Do these samples have the same Gaussian distribution?</h3>\ - <FORM METHOD=GET> <TABLE BORDER CELLPADDING=5>\ - <TR>\ - <TD>1. Mean </TD> - <TD><input type=text name=m1 value=" m1 " size=8></TD>\ - <TD>1. Variance</TD> - <TD><input type=text name=v1 value=" v1 " size=8></TD>\ - <TD>1. Count </TD> - <TD><input type=text name=n1 value=" n1 " size=8></TD>\ - </TR><TR>\ - <TD>2. Mean </TD> - <TD><input type=text name=m2 value=" m2 " size=8></TD>\ - <TD>2. Variance</TD> - <TD><input type=text name=v2 value=" v2 " size=8></TD>\ - <TD>2. Count </TD> - <TD><input type=text name=n2 value=" n2 " size=8></TD>\ - </TR> <input type=submit value=\"Compute\">\ - </TABLE></FORM><BR>" - @} else if (MENU[2] ~ "Image") @{ - Reason = "OK" ORS "Content-type: image/png" - #Reason = "OK" ORS "Content-type: application/x-postscript" - #Reason = "OK" ORS "Content-type: image/gif" - Header = Footer = "" - Document = Image - @} -@} -@c endfile -@end smallexample - -@cindex PostScript -As usual, we give a short description of the service in the first -menu choice. The third menu choice shows us that generation and -presentation of an image are two separate actions. While the latter -takes place quite instantly in the third menu choice, the former -takes place in the much longer second choice. Image data passes from the -generating action to the presenting action via the variable @code{Image} -that contains a complete @file{.png} image, which is otherwise stored -in a file. If you prefer @file{.ps} or @file{.gif} images over the -default @file{.png} images, you may select these options by uncommenting -the appropriate lines. But remember to do so in two places: when -telling GNUPlot which kind of images to generate, and when transmitting the -image at the end of the program. - -Looking at the end of the program, -the way we pass the @samp{Content-type} to the browser is a bit unusual. -It is appended to the @samp{OK} of the first header line -to make sure the type information becomes part of the header. -The other variables that get transmitted across the network are -made empty, because in this case we do not have an HTML document to -transmit, but rather raw image data to contain in the body. - -Most of the work is done in the second menu choice. It starts with a -strange JavaScript code snippet. When first implementing this server, -we used a short @code{@w{"<IMG SRC="} MyPrefix "/Image>"} here. But then -browsers got smarter and tried to improve on speed by requesting the -image and the HTML code at the same time. When doing this, the browser -tries to build up a connection for the image request while the request for -the HTML text is not yet completed. The browser tries to connect -to the @command{gawk} server on port 8080 while port 8080 is still in use for -transmission of the HTML text. The connection for the image cannot be -built up, so the image appears as ``broken'' in the browser window. -We solved this problem by telling the browser to open a separate window -for the image, but only after a delay of 1000 milliseconds. -By this time, the server should be ready for serving the next request. - -But there is one more subtlety in the JavaScript code. -Each time the JavaScript code opens a window for the image, the -name of the image is appended with a timestamp (@code{systime}). -Why this constant change of name for the image? Initially, we always named -the image @code{Image}, but then the Netscape browser noticed the name -had @emph{not} changed since the previous request and displayed the -previous image (caching behavior). The server core -is implemented so that browsers are told @emph{not} to cache anything. -Obviously HTTP requests do not always work as expected. One way to -circumvent the cache of such overly smart browsers is to change the -name of the image with each request. These three lines of JavaScript -caused us a lot of trouble. - -The rest can be broken -down into two phases. At first, we check if there are statistical -parameters. When the program is first started, there usually are no -parameters because it enters the page coming from the top menu. -Then, we only have to present the user a form that he can use to change -statistical parameters and submit them. Subsequently, the submission of -the form causes the execution of the first phase because @emph{now} -there @emph{are} parameters to handle. - -Now that we have parameters, we know there will be an image available. -Therefore we insert the JavaScript code here to initiate the opening -of the image in a separate window. Then, -we prepare some variables that will be passed to GNUPlot for calculation -of the probabilities. Prior to reading the results, we must temporarily -change @code{RS} because GNUPlot separates lines with newlines. -After instructing GNUPlot to generate a @file{.png} (or @file{.ps} or -@file{.gif}) image, we initiate the insertion of some text, -explaining the resulting probabilities. The final @samp{plot} command -actually generates the image data. This raw binary has to be read in carefully -without adding, changing, or deleting a single byte. Hence the unusual -initialization of @code{Image} and completion with a @code{while} loop. - -When using this server, it soon becomes clear that it is far from being -perfect. It mixes source code of six scripting languages or protocols: - -@itemize @bullet -@item GNU @command{awk} implements a server for the protocol: -@item HTTP which transmits: -@item HTML text which contains a short piece of: -@item JavaScript code opening a separate window. -@item A Bourne shell script is used for piping commands into: -@item GNUPlot to generate the image to be opened. -@end itemize - -After all this work, the GNUPlot image opens in the JavaScript window -where it can be viewed by the user. - -It is probably better not to mix up so many different languages. -The result is not very readable. Furthermore, the -statistical part of the server does not take care of invalid input. -Among others, using negative variances will cause invalid results. - -@node MAZE, MOBAGWHO, STATIST, Some Applications and Techniques -@section MAZE: Walking Through a Maze In Virtual Reality -@cindex MAZE -@cindex VRML -@c VRML in iX 11/96 134. -@quotation -@cindex Perlis, Alan -@i{In the long run, every program becomes rococo, and then rubble.}@* -Alan Perlis -@end quotation - -By now, we know how to present arbitrary @samp{Content-type}s to a browser. -In this @value{SECTION}, our server will present a 3D world to our browser. -The 3D world is described in a scene description language (VRML, -Virtual Reality Modeling Language) that allows us to travel through a -perspective view of a 2D maze with our browser. Browsers with a -VRML plugin enable exploration of this technology. We could do -one of those boring @samp{Hello world} examples here, that are usually -presented when introducing novices to -VRML. If you have never written -any VRML code, have a look at -the VRML FAQ. -Presenting a static VRML scene is a bit trivial; in order to expose -@command{gawk}'s new capabilities, we will present a dynamically generated -VRML scene. The function @code{SetUpServer} is very simple because it -only sets the default HTML page and initializes the random number -generator. As usual, the surrounding server lets you browse the maze. - -@smallexample -@c file eg/network/maze.awk -function SetUpServer() @{ - TopHeader = "<HTML><title>Walk through a maze</title>" - TopDoc = "\ - <h2>Please choose one of the following actions:</h2>\ - <UL>\ - <LI><A HREF=" MyPrefix "/AboutServer>About this server</A>\ - <LI><A HREF=" MyPrefix "/VRMLtest>Watch a simple VRML scene</A>\ - </UL>" - TopFooter = "</HTML>" - srand() -@} -@c endfile -@end smallexample - -The function @code{HandleGET} is a bit longer because it first computes -the maze and afterwards generates the VRML code that is sent across -the network. As shown in the STATIST example -(@pxref{STATIST}), -we set the type of the -content to VRML and then store the VRML representation of the maze as the -page content. We assume that the maze is stored in a 2D array. Initially, -the maze consists of walls only. Then, we add an entry and an exit to the -maze and let the rest of the work be done by the function @code{MakeMaze}. -Now, only the wall fields are left in the maze. By iterating over the these -fields, we generate one line of VRML code for each wall field. - -@smallexample -@c file eg/network/maze.awk -function HandleGET() @{ - if (MENU[2] == "AboutServer") @{ - Document = "If your browser has a VRML 2 plugin,\ - this server shows you a simple VRML scene." - @} else if (MENU[2] == "VRMLtest") @{ - XSIZE = YSIZE = 11 # initially, everything is wall - for (y = 0; y < YSIZE; y++) - for (x = 0; x < XSIZE; x++) - Maze[x, y] = "#" - delete Maze[0, 1] # entry is not wall - delete Maze[XSIZE-1, YSIZE-2] # exit is not wall - MakeMaze(1, 1) - Document = "\ -#VRML V2.0 utf8\n\ -Group @{\n\ - children [\n\ - PointLight @{\n\ - ambientIntensity 0.2\n\ - color 0.7 0.7 0.7\n\ - location 0.0 8.0 10.0\n\ - @}\n\ - DEF B1 Background @{\n\ - skyColor [0 0 0, 1.0 1.0 1.0 ]\n\ - skyAngle 1.6\n\ - groundColor [1 1 1, 0.8 0.8 0.8, 0.2 0.2 0.2 ]\n\ - groundAngle [ 1.2 1.57 ]\n\ - @}\n\ - DEF Wall Shape @{\n\ - geometry Box @{size 1 1 1@}\n\ - appearance Appearance @{ material Material @{ diffuseColor 0 0 1 @} @}\n\ - @}\n\ - DEF Entry Viewpoint @{\n\ - position 0.5 1.0 5.0\n\ - orientation 0.0 0.0 -1.0 0.52\n\ - @}\n" - for (i in Maze) @{ - split(i, t, SUBSEP) - Document = Document " Transform @{ translation " - Document = Document t[1] " 0 -" t[2] " children USE Wall @}\n" - @} - Document = Document " ] # end of group for world\n@}" - Reason = "OK" ORS "Content-type: model/vrml" - Header = Footer = "" - @} -@} -@c endfile -@end smallexample - -Finally, we have a look at @code{MakeMaze}, the function that generates -the @code{Maze} array. When entered, this function assumes that the array -has been initialized so that each element represents a wall element and -the maze is initially full of wall elements. Only the entrance and the exit -of the maze should have been left free. The parameters of the function tell -us which element must be marked as not being a wall. After this, we take -a look at the four neighbouring elements and remember which we have already -treated. Of all the neighbouring elements, we take one at random and -walk in that direction. Therefore, the wall element in that direction has -to be removed and then, we call the function recursively for that element. -The maze is only completed if we iterate the above procedure for -@emph{all} neighbouring elements (in random order) and for our present -element by recursively calling the function for the present element. This -last iteration could have been done in a loop, -but it is done much simpler recursively. - -Notice that elements with coordinates that are both odd are assumed to be -on our way through the maze and the generating process cannot terminate -as long as there is such an element not being @code{delete}d. All other -elements are potentially part of the wall. - -@smallexample -@c file eg/network/maze.awk -function MakeMaze(x, y) @{ - delete Maze[x, y] # here we are, we have no wall here - p = 0 # count unvisited fields in all directions - if (x-2 SUBSEP y in Maze) d[p++] = "-x" - if (x SUBSEP y-2 in Maze) d[p++] = "-y" - if (x+2 SUBSEP y in Maze) d[p++] = "+x" - if (x SUBSEP y+2 in Maze) d[p++] = "+y" - if (p>0) @{ # if there are univisited fields, go there - p = int(p*rand()) # choose one unvisited field at random - if (d[p] == "-x") @{ delete Maze[x - 1, y]; MakeMaze(x - 2, y) - @} else if (d[p] == "-y") @{ delete Maze[x, y - 1]; MakeMaze(x, y - 2) - @} else if (d[p] == "+x") @{ delete Maze[x + 1, y]; MakeMaze(x + 2, y) - @} else if (d[p] == "+y") @{ delete Maze[x, y + 1]; MakeMaze(x, y + 2) - @} # we are back from recursion - MakeMaze(x, y); # try again while there are unvisited fields - @} -@} -@c endfile -@end smallexample - -@node MOBAGWHO, STOXPRED, MAZE, Some Applications and Techniques -@section MOBAGWHO: a Simple Mobile Agent -@cindex MOBAGWHO program -@cindex agent -@quotation -@cindex Hoare, C.A.R. -@i{There are two ways of constructing a software design: One way is to -make it so simple that there are obviously no deficiencies, and the -other way is to make it so complicated that there are no obvious -deficiencies.} @* -C. A. R. Hoare -@end quotation - -A @dfn{mobile agent} is a program that can be dispatched from a computer and -transported to a remote server for execution. This is called @dfn{migration}, -which means that a process on another system is started that is independent -from its originator. Ideally, it wanders through -a network while working for its creator or owner. In places like -the UMBC Agent Web, -people are quite confident that (mobile) agents are a software engineering -paradigm that enables us to significantly increase the efficiency -of our work. Mobile agents could become the mediators between users and -the networking world. For an unbiased view at this technology, -see the remarkable paper @cite{Mobile Agents: Are they a good -idea?}.@footnote{@uref{http://www.research.ibm.com/massive/mobag.ps}} - -@ignore -@c Chuck says to take all of this out. -@cindex Tcl/Tk -A good instance of this paradigm is -@cite{Agent Tcl},@footnote{@uref{http://agent.cs.dartmouth.edu/software/agent2.0/}} -an extension of the Tcl language. After introducing a typical -development environment, the aforementioned paper shows a nice little -example application that we will try to rebuild in @command{gawk}. The -@command{who} agent takes a list of servers and wanders from one server -to the next one, always looking to see who is logged in. -Having reached the last -one, it sends back a message with a list of all users it found on each -machine. - -But before implementing something that might or might not be a mobile -agent, let us clarify the concept and some important terms. The agent -paradigm in general is such a young scientific discipline that it has -not yet developed a widely-accepted terminology. Some authors try to -give precise definitions, but their scope is often not wide enough -to be generally accepted. Franklin and Graesser ask -@cite{Is it an Agent or just a Program: A Taxonomy for Autonomous -Agents}@footnote{@uref{http://www.msci.memphis.edu/~franklin/AgentProg.html}} -and give even better answers than Caglayan and Harrison in their -@cite{Agent Sourcebook}.@footnote{@uref{http://www.aminda.com/mazzu/sourcebook/}} - -@itemize @minus -@item -@i{An autonomous agent is a system situated within and a part of -an environment that senses that environment and acts on it, over time, in -pursuit of its own agenda and so as to effect what it senses in the future.} -(Quoted from Franklin and Graesser.) -@item -A mobile agent is able to transport itself from one machine to another. -@item -The term @dfn{migration} often denotes this process of moving. -But neither of the two sources above even mentions this term, while others -use it regularly. -@end itemize - -Before delving into the (rather demanding) details of -implementation, let us give just one more quotation as a final -motivation. Steven Farley published an excellent paper called -@cite{Mobile Agent System Architecture},@footnote{This often -cited text originally appeared as a conference paper here: -@uref{http://www.sigs.com/publications/docs/java/9705/farley.html} -Many bibliographies on the Internet point to this dead link. Meanwhile, -the paper appeared as a contribution to a book called More Java Gems here: -@uref{http://uk.cambridge.org/computerscience/object/catalogue/0521774772/default.htm}} -in which he asks ``Why use an agent architecture?'' - -@quotation -If client-server systems are the currently established norm and distributed -object systems such as CORBA are defining the future standards, why bother -with agents? Agent architectures have certain advantages over these other -types. Three of the most important advantages are: -@cindex CORBA - -@enumerate -@item -An agent performs much processing at the server where local bandwidth -is high, thus reducing the amount of network bandwidth consumed and increasing -overall performance. In contrast, a CORBA client object with the equivalent -functionality of a given agent must make repeated remote method calls to -the server object because CORBA objects cannot move across the network -at runtime. - -@item -An agent operates independently of the application from which the -agent was invoked. The agent operates asynchronously, meaning that the -client application does not need to wait for the results. This is especially -important for mobile users who are not always connected to the network. - -@item -The use of agents allows for the injection of new functionality into -a system at run time. An agent system essentially contains its own automatic -software distribution mechanism. Since CORBA has no built-in support for -mobile code, new functionality generally has to be installed manually. - -@end enumerate - -Of course a non-agent system can exhibit these same features with some -work. But the mobile code paradigm supports the transfer of executable -code to a remote location for asynchronous execution from the start. An -agent architecture should be considered for systems where the above features -are primary requirements. -@end quotation -@end ignore - -When trying to migrate a process from one system to another, -a server process is needed on the receiving side. Depending on the kind -of server process, several ways of implementation come to mind. -How the process is implemented depends upon the kind of server process: - -@itemize @bullet -@item -HTTP can be used as the protocol for delivery of the migrating -process. In this case, we use a common web -server as the receiving server process. A universal CGI script -mediates between migrating process and web server. -Each server willing to accept migrating agents makes this universal -service available. HTTP supplies the @code{POST} method to transfer -some data to a file on the web server. When a CGI script is called -remotely with the @code{POST} method instead of the usual @code{GET} method, -data is transmitted from the client process to the standard input -of the server's CGI script. So, to implement a mobile agent, -we must not only write the agent program to start on the client -side, but also the CGI script to receive the agent on the server side. - -@cindex CGI -@cindex apache -@item -The @code{PUT} method can also be used for migration. HTTP does not -require a CGI script for migration via @code{PUT}. However, with common web -servers there is no advantage to this solution, because web servers such as -Apache -require explicit activation of a special @code{PUT} script. - -@item -@cite{Agent Tcl} pursues a different course; it relies on a dedicated server -process with a dedicated protocol specialized for receiving mobile agents. -@end itemize - -Our agent example abuses a common web server as a migration tool. So, it needs a -universal CGI script on the receiving side (the web server). The receiving script is -activated with a @code{POST} request when placed into a location like -@file{/httpd/cgi-bin/PostAgent.sh}. Make sure that the server system uses a -version of @command{gawk} that supports network access (Version 3.1 or later; -verify with @samp{gawk --version}). - -@example -@c file eg/network/PostAgent.sh -#!/bin/sh -MobAg=/tmp/MobileAgent.$$ -# direct script to mobile agent file -cat > $MobAg -# execute agent concurrently -gawk -f $MobAg $MobAg > /dev/null & -# HTTP header, terminator and body -gawk 'BEGIN @{ print "\r\nAgent started" @}' -rm $MobAg # delete script file of agent -@c endfile -@end example - -By making its process id (@code{$$}) part of the unique @value{FN}, the -script avoids conflicts between concurrent instances of the script. -First, all lines -from standard input (the mobile agent's source code) are copied into -this unique file. Then, the agent is started as a concurrent process -and a short message reporting this fact is sent to the submitting client. -Finally, the script file of the mobile agent is removed because it is -no longer needed. Although it is a short script, there are several noteworthy -points: - -@table @asis -@item Security -@emph{There is none}. In fact, the CGI script should never -be made available on a server that is part of the Internet because everyone -would be allowed to execute arbitrary commands with it. This behavior is -acceptable only when performing rapid prototyping. - -@item Self-Reference -Each migrating instance of an agent is started -in a way that enables it to read its own source code from standard input -and use the code for subsequent -migrations. This is necessary because it needs to treat the agent's code -as data to transmit. @command{gawk} is not the ideal language for such -a job. Lisp and Tcl are more suitable because they do not make a distinction -between program code and data. - -@item Independence -After migration, the agent is not linked to its -former home in any way. By reporting @samp{Agent started}, it waves -``Goodbye'' to its origin. The originator may choose to terminate or not. -@end table - -@cindex Lisp -The originating agent itself is started just like any other command-line -script, and reports the results on standard output. By letting the name -of the original host migrate with the agent, the agent that migrates -to a host far away from its origin can report the result back home. -Having arrived at the end of the journey, the agent establishes -a connection and reports the results. This is the reason for -determining the name of the host with @samp{uname -n} and storing it -in @code{MyOrigin} for later use. We may also set variables with the -@option{-v} option from the command line. This interactivity is only -of importance in the context of starting a mobile agent; therefore this -@code{BEGIN} pattern and its action do not take part in migration: - -@smallexample -@c file eg/network/mobag.awk -BEGIN @{ - if (ARGC != 2) @{ - print "MOBAG - a simple mobile agent" - print "CALL:\n gawk -f mobag.awk mobag.awk" - print "IN:\n the name of this script as a command-line parameter" - print "PARAM:\n -v MyOrigin=myhost.com" - print "OUT:\n the result on stdout" - print "JK 29.03.1998 01.04.1998" - exit - @} - if (MyOrigin == "") @{ - "uname -n" | getline MyOrigin - close("uname -n") - @} -@} -@c endfile -@end smallexample - -Since @command{gawk} cannot manipulate and transmit parts of the program -directly, the source code is read and stored in strings. -Therefore, the program scans itself for -the beginning and the ending of functions. -Each line in between is appended to the code string until the end of -the function has been reached. A special case is this part of the program -itself. It is not a function. -Placing a similar framework around it causes it to be treated -like a function. Notice that this mechanism works for all the -functions of the source code, but it cannot guarantee that the order -of the functions is preserved during migration: - -@smallexample -@c file eg/network/mobag.awk -#ReadMySelf -/^function / @{ FUNC = $2 @} -/^END/ || /^#ReadMySelf/ @{ FUNC = $1 @} -FUNC != "" @{ MOBFUN[FUNC] = MOBFUN[FUNC] RS $0 @} -(FUNC != "") && (/^@}/ || /^#EndOfMySelf/) \ - @{ FUNC = "" @} -#EndOfMySelf -@c endfile -@end smallexample - -The web server code in -@ref{Interacting Service, ,A Web Service with Interaction}, -was first developed as a site-independent core. Likewise, the -@command{gawk}-based mobile agent -starts with an agent-independent core, to which can be appended -application-dependent functions. What follows is the only -application-independent function needed for the mobile agent: - -@smallexample -@c file eg/network/mobag.awk -function migrate(Destination, MobCode, Label) @{ - MOBVAR["Label"] = Label - MOBVAR["Destination"] = Destination - RS = ORS = "\r\n" - HttpService = "/inet/tcp/0/" Destination - for (i in MOBFUN) - MobCode = (MobCode "\n" MOBFUN[i]) - MobCode = MobCode "\n\nBEGIN @{" - for (i in MOBVAR) - MobCode = (MobCode "\n MOBVAR[\"" i "\"] = \"" MOBVAR[i] "\"") - MobCode = MobCode "\n@}\n" - print "POST /cgi-bin/PostAgent.sh HTTP/1.0" |& HttpService - print "Content-length:", length(MobCode) ORS |& HttpService - printf "%s", MobCode |& HttpService - while ((HttpService |& getline) > 0) - print $0 - close(HttpService) -@} -@c endfile -@end smallexample - -The @code{migrate} function prepares the -aforementioned strings containing the program code and transmits them to a -server. A consequence of this modular approach is that the @code{migrate} -function takes some parameters that aren't needed in this application, -but that will be in future ones. Its mandatory parameter @code{Destination} holds the -name (or IP address) of the server that the agent wants as a host for its -code. The optional parameter @code{MobCode} may contain some @command{gawk} -code that is inserted during migration in front of all other code. -The optional parameter @code{Label} may contain -a string that tells the agent what to do in program execution after -arrival at its new home site. One of the serious obstacles in implementing -a framework for mobile agents is that it does not suffice to migrate the -code. It is also necessary to migrate the state of execution of the agent. In -contrast to @cite{Agent Tcl}, this program does not try to migrate the complete set -of variables. The following conventions are used: - -@itemize @bullet -@item -Each variable in an agent program is local to the current host and does -@emph{not} migrate. - -@item -The array @code{MOBFUN} shown above is an exception. It is handled -by the function @code{migrate} and does migrate with the application. - -@item -The other exception is the array @code{MOBVAR}. Each variable that -takes part in migration has to be an element of this array. -@code{migrate} also takes care of this. -@end itemize - -Now it's clear what happens to the @code{Label} parameter of the -function @code{migrate}. It is copied into @code{MOBVAR["Label"]} and -travels alongside the other data. Since travelling takes place via HTTP, -records must be separated with @code{"\r\n"} in @code{RS} and -@code{ORS} as usual. The code assembly for migration takes place in -three steps: - -@itemize @bullet -@item -Iterate over @code{MOBFUN} to collect all functions verbatim. - -@item -Prepare a @code{BEGIN} pattern and put assignments to mobile -variables into the action part. - -@item -Transmission itself resembles GETURL: the header with the request -and the @code{Content-length} is followed by the body. In case there is -any reply over the network, it is read completely and echoed to -standard output to avoid irritating the server. -@end itemize - -The application-independent framework is now almost complete. What follows -is the @code{END} pattern that is executed when the mobile agent has -finished reading its own code. First, it checks whether it is already -running on a remote host or not. In case initialization has not yet taken -place, it starts @code{MyInit}. Otherwise (later, on a remote host), it -starts @code{MyJob}: - -@smallexample -@c file eg/network/mobag.awk -END @{ - if (ARGC != 2) exit # stop when called with wrong parameters - if (MyOrigin != "") # is this the originating host? - MyInit() # if so, initialize the application - else # we are on a host with migrated data - MyJob() # so we do our job -@} -@c endfile -@end smallexample - -All that's left to extend the framework into a complete application -is to write two application-specific functions: @code{MyInit} and -@code{MyJob}. Keep in mind that the former is executed once on the -originating host, while the latter is executed after each migration: - -@smallexample -@c file eg/network/mobag.awk -function MyInit() @{ - MOBVAR["MyOrigin"] = MyOrigin - MOBVAR["Machines"] = "localhost/80 max/80 moritz/80 castor/80" - split(MOBVAR["Machines"], Machines) # which host is the first? - migrate(Machines[1], "", "") # go to the first host - while (("/inet/tcp/8080/0/0" |& getline) > 0) # wait for result - print $0 # print result - close("/inet/tcp/8080/0/0") -@} -@c endfile -@end smallexample - -As mentioned earlier, this agent takes the name of its origin -(@code{MyOrigin}) with it. Then, it takes the name of its first -destination and goes there for further work. Notice that this name has -the port number of the web server appended to the name of the server, -because the function @code{migrate} needs it this way to create -the @code{HttpService} variable. Finally, it waits for the result to arrive. -The @code{MyJob} function runs on the remote host: - -@smallexample -@c file eg/network/mobag.awk -function MyJob() @{ - # forget this host - sub(MOBVAR["Destination"], "", MOBVAR["Machines"]) - MOBVAR["Result"]=MOBVAR["Result"] SUBSEP SUBSEP MOBVAR["Destination"] ":" - while (("who" | getline) > 0) # who is logged in? - MOBVAR["Result"] = MOBVAR["Result"] SUBSEP $0 - close("who") - if (index(MOBVAR["Machines"], "/") > 0) @{ # any more machines to visit? - split(MOBVAR["Machines"], Machines) # which host is next? - migrate(Machines[1], "", "") # go there - @} else @{ # no more machines - gsub(SUBSEP, "\n", MOBVAR["Result"]) # send result to origin - print MOBVAR["Result"] |& "/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080" - close("/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080") - @} -@} -@c endfile -@end smallexample - -After migrating, the first thing to do in @code{MyJob} is to delete -the name of the current host from the list of hosts to visit. Now, it -is time to start the real work by appending the host's name to the -result string, and reading line by line who is logged in on this host. -A very annoying circumstance is the fact that the elements of -@code{MOBVAR} cannot hold the newline character (@code{"\n"}). If they -did, migration of this string did not work because the string didn't -obey the syntax rule for a string in @command{gawk}. -@code{SUBSEP} is used as a temporary replacement. -If the list of hosts to visit holds -at least one more entry, the agent migrates to that place to go on -working there. Otherwise, we replace the @code{SUBSEP}s -with a newline character in the resulting string, and report it to -the originating host, whose name is stored in @code{MOBVAR["MyOrigin"]}. - -@node STOXPRED, PROTBASE, MOBAGWHO, Some Applications and Techniques -@section STOXPRED: Stock Market Prediction As A Service -@cindex STOXPRED program -@cindex Yahoo -@quotation -@i{Far out in the uncharted backwaters of the unfashionable end of -the Western Spiral arm of the Galaxy lies a small unregarded yellow sun.} - -@i{Orbiting this at a distance of roughly ninety-two million miles is an -utterly insignificant little blue-green planet whose ape-descendent life -forms are so amazingly primitive that they still think digital watches are -a pretty neat idea.} - -@i{This planet has --- or rather had --- a problem, which was this: -most of the people living on it were unhappy for pretty much of the time. -Many solutions were suggested for this problem, but most of these were -largely concerned with the movements of small green pieces of paper, -which is odd because it wasn't the small green pieces of paper that -were unhappy.} @* -Douglas Adams, @cite{The Hitch Hiker's Guide to the Galaxy} -@end quotation - -@cindex @command{cron} -Valuable services on the Internet are usually @emph{not} implemented -as mobile agents. There are much simpler ways of implementing services. -All Unix systems provide, for example, the @command{cron} service. -Unix system users can write a list of tasks to be done each day, each -week, twice a day, or just once. The list is entered into a file named -@file{crontab}. For example, to distribute a newsletter on a daily -basis this way, use @command{cron} for calling a script each day early -in the morning. - -@example -# run at 8 am on weekdays, distribute the newsletter -0 8 * * 1-5 $HOME/bin/daily.job >> $HOME/log/newsletter 2>&1 -@end example - -The script first looks for interesting information on the Internet, -assembles it in a nice form and sends the results via email to -the customers. - -The following is an example of a primitive -newsletter on stock market prediction. It is a report which first -tries to predict the change of each share in the Dow Jones Industrial -Index for the particular day. Then it mentions some especially -promising shares as well as some shares which look remarkably bad -on that day. The report ends with the usual disclaimer which tells -every child @emph{not} to try this at home and hurt anybody. -@cindex Dow Jones Industrial Index - -@smallexample -Good morning Uncle Scrooge, - -This is your daily stock market report for Monday, October 16, 2000. -Here are the predictions for today: - - AA neutral - GE up - JNJ down - MSFT neutral - @dots{} - UTX up - DD down - IBM up - MO down - WMT up - DIS up - INTC up - MRK down - XOM down - EK down - IP down - -The most promising shares for today are these: - - INTC http://biz.yahoo.com/n/i/intc.html - -The stock shares to avoid today are these: - - EK http://biz.yahoo.com/n/e/ek.html - IP http://biz.yahoo.com/n/i/ip.html - DD http://biz.yahoo.com/n/d/dd.html - @dots{} -@end smallexample - -@ignore -@c Chuck suggests removing this paragraph -If you are not into stock market prediction but want to earn money -with a more humane service, you might prefer to send out horoscopes -to your customers. Or, once every refrigerator in every household on this side -of the Chinese Wall is connected to the Internet, such a service could -inspect the contents of your customer's refrigerators each day and -advise them on nutrition. Big Brother is watching them. -@end ignore - -The script as a whole is rather long. In order to ease the pain of -studying other people's source code, we have broken the script -up into meaningful parts which are invoked one after the other. -The basic structure of the script is as follows: - -@example -@c file eg/network/stoxpred.awk -BEGIN @{ - Init() - ReadQuotes() - CleanUp() - Prediction() - Report() - SendMail() -@} -@c endfile -@end example - -The earlier parts store data into variables and arrays which are -subsequently used by later parts of the script. The @code{Init} function -first checks if the script is invoked correctly (without any parameters). -If not, it informs the user of the correct usage. What follows are preparations -for the retrieval of the historical quote data. The names of the 30 stock -shares are stored in an array @code{name} along with the current date -in @code{day}, @code{month}, and @code{year}. - -All users who are separated -from the Internet by a firewall and have to direct their Internet accesses -to a proxy must supply the name of the proxy to this script with the -@samp{-v Proxy=@var{name}} option. For most users, the default proxy and -port number should suffice. - -@example -@c file eg/network/stoxpred.awk -function Init() @{ - if (ARGC != 1) @{ - print "STOXPRED - daily stock share prediction" - print "IN:\n no parameters, nothing on stdin" - print "PARAM:\n -v Proxy=MyProxy -v ProxyPort=80" - print "OUT:\n commented predictions as email" - print "JK 09.10.2000" - exit - @} - # Remember ticker symbols from Dow Jones Industrial Index - StockCount = split("AA GE JNJ MSFT AXP GM JPM PG BA HD KO \ - SBC C HON MCD T CAT HWP MMM UTX DD IBM MO WMT DIS INTC \ - MRK XOM EK IP", name); - # Remember the current date as the end of the time series - day = strftime("%d") - month = strftime("%m") - year = strftime("%Y") - if (Proxy == "") Proxy = "chart.yahoo.com" - if (ProxyPort == 0) ProxyPort = 80 - YahooData = "/inet/tcp/0/" Proxy "/" ProxyPort -@} -@c endfile -@end example - -@cindex CSV format -There are two really interesting parts in the script. One is the -function which reads the historical stock quotes from an Internet -server. The other is the one that does the actual prediction. In -the following function we see how the quotes are read from the -Yahoo server. The data which comes from the server is in -CSV format (comma-separated values): - -@example -@c file eg/network/stoxdata.txt -Date,Open,High,Low,Close,Volume -9-Oct-00,22.75,22.75,21.375,22.375,7888500 -6-Oct-00,23.8125,24.9375,21.5625,22,10701100 -5-Oct-00,24.4375,24.625,23.125,23.50,5810300 -@c endfile -@end example - -Lines contain values of the same time instant, whereas columns are -separated by commas and contain the kind of data that is described -in the header (first) line. At first, @command{gawk} is instructed to -separate columns by commas (@samp{FS = ","}). In the loop that follows, -a connection to the Yahoo server is first opened, then a download takes -place, and finally the connection is closed. All this happens once for -each ticker symbol. In the body of this loop, an Internet address is -built up as a string according to the rules of the Yahoo server. The -starting and ending date are chosen to be exactly the same, but one year -apart in the past. All the action is initiated within the @code{printf} -command which transmits the request for data to the Yahoo server. - -In the inner loop, the server's data is first read and then scanned -line by line. Only lines which have six columns and the name of a month -in the first column contain relevant data. This data is stored -in the two-dimensional array @code{quote}; one dimension -being time, the other being the ticker symbol. During retrieval of the -first stock's data, the calendar names of the time instances are stored -in the array @code{day} because we need them later. - -@smallexample -@c file eg/network/stoxpred.awk -function ReadQuotes() @{ - # Retrieve historical data for each ticker symbol - FS = "," - for (stock = 1; stock <= StockCount; stock++) @{ - URL = "http://chart.yahoo.com/table.csv?s=" name[stock] \ - "&a=" month "&b=" day "&c=" year-1 \ - "&d=" month "&e=" day "&f=" year \ - "g=d&q=q&y=0&z=" name[stock] "&x=.csv" - printf("GET " URL " HTTP/1.0\r\n\r\n") |& YahooData - while ((YahooData |& getline) > 0) @{ - if (NF == 6 && $1 ~ /Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec/) @{ - if (stock == 1) - days[++daycount] = $1; - quote[$1, stock] = $5 - @} - @} - close(YahooData) - @} - FS = " " -@} -@c endfile -@end smallexample - -Now that we @emph{have} the data, it can be checked once again to make sure -that no individual stock is missing or invalid, and that all the stock quotes are -aligned correctly. Furthermore, we renumber the time instances. The -most recent day gets day number 1 and all other days get consecutive -numbers. All quotes are rounded toward the nearest whole number in US Dollars. - -@smallexample -@c file eg/network/stoxpred.awk -function CleanUp() @{ - # clean up time series; eliminate incomplete data sets - for (d = 1; d <= daycount; d++) @{ - for (stock = 1; stock <= StockCount; stock++) - if (! ((days[d], stock) in quote)) - stock = StockCount + 10 - if (stock > StockCount + 1) - continue - datacount++ - for (stock = 1; stock <= StockCount; stock++) - data[datacount, stock] = int(0.5 + quote[days[d], stock]) - @} - delete quote - delete days -@} -@c endfile -@end smallexample - -Now we have arrived at the second really interesting part of the whole affair. -What we present here is a very primitive prediction algorithm: -@emph{If a stock fell yesterday, assume it will also fall today; if -it rose yesterday, assume it will rise today}. (Feel free to replace this -algorithm with a smarter one.) If a stock changed in the same direction -on two consecutive days, this is an indication which should be highlighted. -Two-day advances are stored in @code{hot} and two-day declines in -@code{avoid}. - -The rest of the function is a sanity check. It counts the number of -correct predictions in relation to the total number of predictions -one could have made in the year before. - -@smallexample -@c file eg/network/stoxpred.awk -function Prediction() @{ - # Predict each ticker symbol by prolonging yesterday's trend - for (stock = 1; stock <= StockCount; stock++) @{ - if (data[1, stock] > data[2, stock]) @{ - predict[stock] = "up" - @} else if (data[1, stock] < data[2, stock]) @{ - predict[stock] = "down" - @} else @{ - predict[stock] = "neutral" - @} - if ((data[1, stock] > data[2, stock]) && (data[2, stock] > data[3, stock])) - hot[stock] = 1 - if ((data[1, stock] < data[2, stock]) && (data[2, stock] < data[3, stock])) - avoid[stock] = 1 - @} - # Do a plausibility check: how many predictions proved correct? - for (s = 1; s <= StockCount; s++) @{ - for (d = 1; d <= datacount-2; d++) @{ - if (data[d+1, s] > data[d+2, s]) @{ - UpCount++ - @} else if (data[d+1, s] < data[d+2, s]) @{ - DownCount++ - @} else @{ - NeutralCount++ - @} - if (((data[d, s] > data[d+1, s]) && (data[d+1, s] > data[d+2, s])) || - ((data[d, s] < data[d+1, s]) && (data[d+1, s] < data[d+2, s])) || - ((data[d, s] == data[d+1, s]) && (data[d+1, s] == data[d+2, s]))) - CorrectCount++ - @} - @} -@} -@c endfile -@end smallexample - -At this point the hard work has been done: the array @code{predict} -contains the predictions for all the ticker symbols. It is up to the -function @code{Report} to find some nice words to introduce the -desired information. - -@smallexample -@c file eg/network/stoxpred.awk -function Report() @{ - # Generate report - report = "\nThis is your daily " - report = report "stock market report for "strftime("%A, %B %d, %Y")".\n" - report = report "Here are the predictions for today:\n\n" - for (stock = 1; stock <= StockCount; stock++) - report = report "\t" name[stock] "\t" predict[stock] "\n" - for (stock in hot) @{ - if (HotCount++ == 0) - report = report "\nThe most promising shares for today are these:\n\n" - report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \ - tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n" - @} - for (stock in avoid) @{ - if (AvoidCount++ == 0) - report = report "\nThe stock shares to avoid today are these:\n\n" - report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \ - tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n" - @} - report = report "\nThis sums up to " HotCount+0 " winners and " AvoidCount+0 - report = report " losers. When using this kind\nof prediction scheme for" - report = report " the 12 months which lie behind us,\nwe get " UpCount - report = report " 'ups' and " DownCount " 'downs' and " NeutralCount - report = report " 'neutrals'. Of all\nthese " UpCount+DownCount+NeutralCount - report = report " predictions " CorrectCount " proved correct next day.\n" - report = report "A success rate of "\ - int(100*CorrectCount/(UpCount+DownCount+NeutralCount)) "%.\n" - report = report "Random choice would have produced a 33% success rate.\n" - report = report "Disclaimer: Like every other prediction of the stock\n" - report = report "market, this report is, of course, complete nonsense.\n" - report = report "If you are stupid enough to believe these predictions\n" - report = report "you should visit a doctor who can treat your ailment." -@} -@c endfile -@end smallexample - -The function @code{SendMail} goes through the list of customers and opens -a pipe to the @code{mail} command for each of them. Each one receives an -email message with a proper subject heading and is addressed with his full name. - -@smallexample -@c file eg/network/stoxpred.awk -function SendMail() @{ - # send report to customers - customer["uncle.scrooge@@ducktown.gov"] = "Uncle Scrooge" - customer["more@@utopia.org" ] = "Sir Thomas More" - customer["spinoza@@denhaag.nl" ] = "Baruch de Spinoza" - customer["marx@@highgate.uk" ] = "Karl Marx" - customer["keynes@@the.long.run" ] = "John Maynard Keynes" - customer["bierce@@devil.hell.org" ] = "Ambrose Bierce" - customer["laplace@@paris.fr" ] = "Pierre Simon de Laplace" - for (c in customer) @{ - MailPipe = "mail -s 'Daily Stock Prediction Newsletter'" c - print "Good morning " customer[c] "," | MailPipe - print report "\n.\n" | MailPipe - close(MailPipe) - @} -@} -@c endfile -@end smallexample - -Be patient when running the script by hand. -Retrieving the data for all the ticker symbols and sending the emails -may take several minutes to complete, depending upon network traffic -and the speed of the available Internet link. -The quality of the prediction algorithm is likely to be disappointing. -Try to find a better one. -Should you find one with a success rate of more than 50%, please tell -us about it! It is only for the sake of curiosity, of course. @code{:-)} - -@ignore -@c chuck says to remove this -Let us give you one final indication as to what one can expect from -a prediction of stock data, which is sometimes said to contain much -randomness. One theory says that all relevant information to be taken -into account when estimating the price of a stock is contained in the -stock quotes. Every bit of useful information has influenced the -fair price. Therefore (the theory says) temporary changes (i.e., fluctuations -within a minute) have to be purely random. But what is the cause of -short-term changes in stock prices? - -Stock prices are fixed when supply and demand meet each other. -What people are willing to pay reflects human expectations. -Human expectations are not necessarily random. On the Internet, -you can find an elucidating paper about predictability and human -expectations: -@uref{http://it.ucsd.edu/IT/Newsletter/archives/meir/05meir.html, -@cite{Reflections on ``Universal Prediction of Individual Sequences''}} -The authors (Feder, Merhav, Gutman) introduce the reader to the subject -by telling a thrilling anecdote. -@cindex Shannon, Claude -@quotation -In the early 50's, at Bell Laboratories, David Hagelbarger built a -simple ``mind reading'' machine, whose purpose was to play the ``penny -matching'' game. In this game, a player chooses head or tail, while a -``mind reading'' machine tries to predict and match his choice. -Surprisingly, as Robert Lucky tells in his book ``Silicon Dreams'', -Hagelbarger's simple, 8-state machine, was able to match the ``pennies'' -of its human opponent 5,218 times over the course of 9,795 plays. -Random guessing would lead to such a high success rate with a probability -less than one out of 10 billion! Shannon, who was interested in prediction, -information, and thinking machines, closely followed Hagelbarger's -machine, and eventually built his own stripped-down version of the machine, -having the same states, but one that used a simpler strategy at each state. -As the legend goes, in a duel between the two machines, Shannon's machine -won by a slight margin! No one knows if this was due to a superior algorithm -or just a chance happening associated with the specific sequence at that game. -In any event, the success of both these machines against ``untrained'' human -opponents was explained by the fact that the human opponents cannot draw -completely random -bits. -@end quotation -@end ignore - -@node PROTBASE, , STOXPRED, Some Applications and Techniques -@section PROTBASE: Searching Through A Protein Database -@cindex PROTBASE -@cindex NCBI, National Center for Biotechnology Information -@cindex BLAST, Basic Local Alignment Search Tool -@cindex Hoare, C.A.R. -@quotation -@i{Hoare's Law of Large Problems: Inside every large problem is a small - problem struggling to get out.} -@end quotation - -Yahoo's database of stock market data is just one among the many large -databases on the Internet. Another one is located at NCBI -(National Center for Biotechnology -Information). Established in 1988 as a national resource for molecular -biology information, NCBI creates public databases, conducts research -in computational biology, develops software tools for analyzing genome -data, and disseminates biomedical information. In this section, we -look at one of NCBI's public services, which is called BLAST -(Basic Local Alignment Search Tool). - -You probably know that the information necessary for reproducing living -cells is encoded in the genetic material of the cells. The genetic material -is a very long chain of four base nucleotides. It is the order of -appearance (the sequence) of nucleotides which contains the information -about the substance to be produced. Scientists in biotechnology often -find a specific fragment, determine the nucleotide sequence, and need -to know where the sequence at hand comes from. This is where the large -databases enter the game. At NCBI, databases store the knowledge -about which sequences have ever been found and where they have been found. -When the scientist sends his sequence to the BLAST service, the server -looks for regions of genetic material in its database which -look the most similar to the delivered nucleotide sequence. After a -search time of some seconds or minutes the server sends an answer to -the scientist. In order to make access simple, NCBI chose to offer -their database service through popular Internet protocols. There are -four basic ways to use the so-called BLAST services: - -@itemize @bullet -@item -The easiest way to use BLAST is through the web. Users may simply point -their browsers at the NCBI home page -and link to the BLAST pages. -NCBI provides a stable URL that may be used to perform BLAST searches -without interactive use of a web browser. This is what we will do later -in this section. -A demonstration client -and a @file{README} file demonstrate how to access this URL. - -@item -Currently, -@command{blastcl3} is the standard network BLAST client. -You can download @command{blastcl3} from the -anonymous FTP location. - -@item -BLAST 2.0 can be run locally as a full executable and can be used to run -BLAST searches against private local databases, or downloaded copies of the -NCBI databases. BLAST 2.0 executables may be found on the NCBI -anonymous FTP server. - -@item -The NCBI BLAST Email server is the best option for people without convenient -access to the web. A similarity search can be performed by sending a properly -formatted mail message containing the nucleotide or protein query sequence to -@email{blast@@ncbi.nlm.nih.gov}. The query sequence is compared against the -specified database using the BLAST algorithm and the results are returned in -an email message. For more information on formulating email BLAST searches, -you can send a message consisting of the word ``HELP'' to the same address, -@email{blast@@ncbi.nlm.nih.gov}. -@end itemize - -Our starting point is the demonstration client mentioned in the first option. -The @file{README} file that comes along with the client explains the whole -process in a nutshell. In the rest of this section, we first show -what such requests look like. Then we show how to use @command{gawk} to -implement a client in about 10 lines of code. Finally, we show how to -interpret the result returned from the service. - -Sequences are expected to be represented in the standard -IUB/IUPAC amino acid and nucleic acid codes, -with these exceptions: lower-case letters are accepted and are mapped -into upper-case; a single hyphen or dash can be used to represent a gap -of indeterminate length; and in amino acid sequences, @samp{U} and @samp{*} -are acceptable letters (see below). Before submitting a request, any numerical -digits in the query sequence should either be removed or replaced by -appropriate letter codes (e.g., @samp{N} for unknown nucleic acid residue -or @samp{X} for unknown amino acid residue). -The nucleic acid codes supported are: - -@example -A --> adenosine M --> A C (amino) -C --> cytidine S --> G C (strong) -G --> guanine W --> A T (weak) -T --> thymidine B --> G T C -U --> uridine D --> G A T -R --> G A (purine) H --> A C T -Y --> T C (pyrimidine) V --> G C A -K --> G T (keto) N --> A G C T (any) - - gap of indeterminate length -@end example - -Now you know the alphabet of nucleotide sequences. The last two lines -of the following example query show you such a sequence, which is obviously -made up only of elements of the alphabet just described. Store this example -query into a file named @file{protbase.request}. You are now ready to send -it to the server with the demonstration client. - -@example -@c file eg/network/protbase.request -PROGRAM blastn -DATALIB month -EXPECT 0.75 -BEGIN ->GAWK310 the gawking gene GNU AWK -tgcttggctgaggagccataggacgagagcttcctggtgaagtgtgtttcttgaaatcat -caccaccatggacagcaaa -@c endfile -@end example - -@cindex FASTA/Pearson format -The actual search request begins with the mandatory parameter @samp{PROGRAM} -in the first column followed by the value @samp{blastn} (the name of the -program) for searching nucleic acids. The next line contains the mandatory -search parameter @samp{DATALIB} with the value @samp{month} for the newest -nucleic acid sequences. The third line contains an optional @samp{EXPECT} -parameter and the value desired for it. The fourth line contains the -mandatory @samp{BEGIN} directive, followed by the query sequence in -FASTA/Pearson format. -Each line of information must be less than 80 characters in length. - -The ``month'' database contains all new or revised sequences released in the -last 30 days and is useful for searching against new sequences. -There are five different blast programs, @command{blastn} being the one that -compares a nucleotide query sequence against a nucleotide sequence database. - -The last server directive that must appear in every request is the -@samp{BEGIN} directive. The query sequence should immediately follow the -@samp{BEGIN} directive and must appear in FASTA/Pearson format. -A sequence in -FASTA/Pearson format begins with a single-line description. -The description line, which is required, is distinguished from the lines of -sequence data that follow it by having a greater-than (@samp{>}) symbol -in the first column. For the purposes of the BLAST server, the text of -the description is arbitrary. - -If you prefer to use a client written in @command{gawk}, just store the following -10 lines of code into a file named @file{protbase.awk} and use this client -instead. Invoke it with @samp{gawk -f protbase.awk protbase.request}. -Then wait a minute and watch the result coming in. In order to replicate -the demonstration client's behaviour as closely as possible, this client -does not use a proxy server. We could also have extended the client program -in @ref{GETURL, ,Retrieving Web Pages}, to implement the client request from -@file{protbase.awk} as a special case. - -@smallexample -@c file eg/network/protbase.awk -@{ request = request "\n" $0 @} - -END @{ - BLASTService = "/inet/tcp/0/www.ncbi.nlm.nih.gov/80" - printf "POST /cgi-bin/BLAST/nph-blast_report HTTP/1.0\n" |& BLASTService - printf "Content-Length: " length(request) "\n\n" |& BLASTService - printf request |& BLASTService - while ((BLASTService |& getline) > 0) - print $0 - close(BLASTService) -@} -@c endfile -@end smallexample - -The demonstration client from NCBI is 214 lines long (written in C) and -it is not immediately obvious what it does. Our client is so short that -it @emph{is} obvious what it does. First it loops over all lines of the -query and stores the whole query into a variable. Then the script -establishes an Internet connection to the NCBI server and transmits the -query by framing it with a proper HTTP request. Finally it receives -and prints the complete result coming from the server. - -Now, let us look at the result. It begins with an HTTP header, which you -can ignore. Then there are some comments about the query having been -filtered to avoid spuriously high scores. After this, there is a reference -to the paper that describes the software being used for searching the data -base. After a repitition of the original query's description we find the -list of significant alignments: - -@smallexample -@c file eg/network/protbase.result -Sequences producing significant alignments: (bits) Value - -gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733... 38 0.20 -gb|AC021056.12|AC021056 Homo sapiens chromosome 3 clone RP11-115... 38 0.20 -emb|AL160278.10|AL160278 Homo sapiens chromosome 9 clone RP11-57... 38 0.20 -emb|AL391139.11|AL391139 Homo sapiens chromosome X clone RP11-35... 38 0.20 -emb|AL365192.6|AL365192 Homo sapiens chromosome 6 clone RP3-421H... 38 0.20 -emb|AL138812.9|AL138812 Homo sapiens chromosome 11 clone RP1-276... 38 0.20 -gb|AC073881.3|AC073881 Homo sapiens chromosome 15 clone CTD-2169... 38 0.20 -@c endfile -@end smallexample - -This means that the query sequence was found in seven human chromosomes. -But the value 0.20 (20%) means that the probability of an accidental match -is rather high (20%) in all cases and should be taken into account. -You may wonder what the first column means. It is a key to the specific -database in which this occurence was found. The unique sequence identifiers -reported in the search results can be used as sequence retrieval keys -via the NCBI server. The syntax of sequence header lines used by the NCBI -BLAST server depends on the database from which each sequence was obtained. -The table below lists the identifiers for the databases from which the -sequences were derived. - -@ifinfo -@example -Database Name Identifier Syntax -============================ ======================== -GenBank gb|accession|locus -EMBL Data Library emb|accession|locus -DDBJ, DNA Database of Japan dbj|accession|locus -NBRF PIR pir||entry -Protein Research Foundation prf||name -SWISS-PROT sp|accession|entry name -Brookhaven Protein Data Bank pdb|entry|chain -Kabat's Sequences of Immuno@dots{} gnl|kabat|identifier -Patents pat|country|number -GenInfo Backbone Id bbs|number -@end example -@end ifinfo - -@ifnotinfo -@multitable {Kabat's Sequences of Immuno@dots{}} {@code{@w{sp|accession|entry name}}} -@item GenBank @tab @code{gb|accession|locus} -@item EMBL Data Library @tab @code{emb|accession|locus} -@item DDBJ, DNA Database of Japan @tab @code{dbj|accession|locus} -@item NBRF PIR @tab @code{pir||entry} -@item Protein Research Foundation @tab @code{prf||name} -@item SWISS-PROT @tab @code{@w{sp|accession|entry name}} -@item Brookhaven Protein Data Bank @tab @code{pdb|entry|chain} -@item Kabat's Sequences of Immuno@dots{} @tab @code{gnl|kabat|identifier} -@item Patents @tab @code{pat|country|number} -@item GenInfo Backbone Id @tab @code{bbs|number} -@end multitable -@end ifnotinfo - - -For example, an identifier might be @samp{gb|AC021182.14|AC021182}, where the -@samp{gb} tag indicates that the identifier refers to a GenBank sequence, -@samp{AC021182.14} is its GenBank ACCESSION, and @samp{AC021182} is the GenBank LOCUS. -The identifier contains no spaces, so that a space indicates the end of the -identifier. - -Let us continue in the result listing. Each of the seven alignments mentioned -above is subsequently described in detail. We will have a closer look at -the first of them. - -@smallexample ->gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733N23, WORKING DRAFT SEQUENCE, 4 - unordered pieces - Length = 176383 - - Score = 38.2 bits (19), Expect = 0.20 - Identities = 19/19 (100%) - Strand = Plus / Plus - -Query: 35 tggtgaagtgtgtttcttg 53 - ||||||||||||||||||| -Sbjct: 69786 tggtgaagtgtgtttcttg 69804 -@end smallexample - -This alignment was located on the human chromosome 7. The fragment on which -part of the query was found had a total length of 176383. Only 19 of the -nucleotides matched and the matching sequence ran from character 35 to 53 -in the query sequence and from 69786 to 69804 in the fragment on chromosome 7. -If you are still reading at this point, you are probably interested in finding -out more about Computational Biology and you might appreciate the following -hints. - -@cindex Computational Biology -@cindex Bioinformatics -@enumerate -@item -There is a book called @cite{Introduction to Computational Biology} -by Michael S. Waterman, which is worth reading if you are seriously -interested. You can find a good -book review -on the Internet. - -@item -While Waterman's book can explain to you the algorithms employed internally -in the database search engines, most practicioners prefer to approach -the subject differently. The applied side of Computational Biology is -called Bioinformatics, and emphasizes the tools available for day-to-day -work as well as how to actually @emph{use} them. One of the very few affordable -books on Bioinformatics is -@cite{Developing Bioinformatics Computer Skills}. - -@item -The sequences @emph{gawk} and @emph{gnuawk} are in widespread use in -the genetic material of virtually every earthly living being. Let us -take this as a clear indication that the divine creator has intended -@code{gawk} to prevail over other scripting languages such as @code{perl}, -@code{tcl}, or @code{python} which are not even proper sequences. (:-) -@end enumerate - -@node Links, GNU Free Documentation License, Some Applications and Techniques, Top -@chapter Related Links - -This section lists the URLs for various items discussed in this @value{CHAPTER}. -They are presented in the order in which they appear. - -@table @asis - -@item @cite{Internet Programming with Python} -@uref{http://www.fsbassociates.com/books/python.htm} - -@item @cite{Advanced Perl Programming} -@uref{http://www.oreilly.com/catalog/advperl} - -@item @cite{Web Client Programming with Perl} -@uref{http://www.oreilly.com/catalog/webclient} - -@item Richard Stevens's home page and book -@uref{http://www.kohala.com/~rstevens} - -@item The SPAK home page -@uref{http://www.userfriendly.net/linux/RPM/contrib/libc6/i386/spak-0.6b-1.i386.html} - -@item Volume III of @cite{Internetworking with TCP/IP}, by Comer and Stevens -@uref{http://www.cs.purdue.edu/homes/dec/tcpip3s.cont.html} - -@item XBM Graphics File Format -@uref{http://www.wotsit.org/download.asp?f=xbm} - -@item GNUPlot -@uref{http://www.cs.dartmouth.edu/gnuplot_info.html} - -@item Mark Humphrys' Eliza page -@uref{http://www.compapp.dcu.ie/~humphrys/eliza.html} - -@item Yahoo! Eliza Information -@uref{http://dir.yahoo.com/Recreation/Games/Computer_Games/Internet_Games/Web_Games/Artificial_Intelligence} - -@item Java versions of Eliza -@uref{http://www.tjhsst.edu/Psych/ch1/eliza.html} - -@item Java versions of Eliza with source code -@uref{http://home.adelphia.net/~lifeisgood/eliza/eliza.htm} - -@item Eliza Programs with Explanations -@uref{http://chayden.net/chayden/eliza/Eliza.shtml} - -@item Loebner Contest -@uref{http://acm.org/~loebner/loebner-prize.htmlx} - -@item Tck/Tk Information -@uref{http://www.scriptics.com/} - -@item Intel 80x86 Processors -@uref{http://developer.intel.com/design/platform/embedpc/what_is.htm} - -@item AMD Elan Processors -@uref{http://www.amd.com/products/epd/processors/4.32bitcont/32bitcont/index.html} - -@item XINU -@uref{http://willow.canberra.edu.au/~chrisc/xinu.html } - -@item GNU/Linux -@uref{http://uclinux.lineo.com/} - -@item Embedded PCs -@uref{http://dir.yahoo.com/Business_and_Economy/Business_to_Business/Computers/Hardware/Embedded_Control/} - -@item MiniSQL -@uref{http://www.hughes.com.au/library/} - -@item Market Share Surveys -@uref{http://www.netcraft.com/survey} - -@item @cite{Numerical Recipes in C: The Art of Scientific Computing} -@uref{http://www.nr.com} - -@item VRML -@uref{http://www.vrml.org} - -@item The VRML FAQ -@uref{http://www.vrml.org/technicalinfo/specifications/specifications.htm#FAQ} - -@item The UMBC Agent Web -@uref{http://www.cs.umbc.edu/agents } - -@item Apache Web Server -@uref{http://www.apache.org} - -@item National Center for Biotechnology Information (NCBI) -@uref{http://www.ncbi.nlm.nih.gov} - -@item Basic Local Alignment Search Tool (BLAST) -@uref{http://www.ncbi.nlm.nih.gov/BLAST/blast_overview.html} - -@item NCBI Home Page -@uref{http://www.ncbi.nlm.nih.gov} - -@item BLAST Pages -@uref{http://www.ncbi.nlm.nih.gov/BLAST} - -@item BLAST Demonstration Client -@uref{ftp://ncbi.nlm.nih.gov/blast/blasturl/} - -@item BLAST anonymous FTP location -@uref{ftp://ncbi.nlm.nih.gov/blast/network/netblast/} - -@item BLAST 2.0 Executables -@uref{ftp://ncbi.nlm.nih.gov/blast/executables/} - -@item IUB/IUPAC Amino Acid and Nucleic Acid Codes -@uref{http://www.uthscsa.edu/geninfo/blastmail.html#item6} - -@item FASTA/Pearson Format -@uref{http://www.ncbi.nlm.nih.gov/BLAST/fasta.html} - -@item Fasta/Pearson Sequence in Java -@uref{http://www.kazusa.or.jp/java/codon_table_java/} - -@item Book Review of @cite{Introduction to Computational Biology} -@uref{http://www.acm.org/crossroads/xrds5-1/introcb.html} - -@item @cite{Developing Bioinformatics Computer Skills} -@uref{http://www.oreilly.com/catalog/bioskills/} - -@end table - -@node GNU Free Documentation License, Index, Links, Top -@unnumbered GNU Free Documentation License -@center Version 1.1, March 2000 - -@display -Copyright (C) 2000 Free Software Foundation, Inc. -59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - -Everyone is permitted to copy and distribute verbatim copies -of this license document, but changing it is not allowed. -@end display -@sp 1 -@enumerate 0 -@item -PREAMBLE - -The purpose of this License is to make a manual, textbook, or other -written document ``free'' in the sense of freedom: to assure everyone -the effective freedom to copy and redistribute it, with or without -modifying it, either commercially or noncommercially. Secondarily, -this License preserves for the author and publisher a way to get -credit for their work, while not being considered responsible for -modifications made by others. - -This License is a kind of ``copyleft'', which means that derivative -works of the document must themselves be free in the same sense. It -complements the GNU General Public License, which is a copyleft -license designed for free software. - -We have designed this License in order to use it for manuals for free -software, because free software needs free documentation: a free -program should come with manuals providing the same freedoms that the -software does. But this License is not limited to software manuals; -it can be used for any textual work, regardless of subject matter or -whether it is published as a printed book. We recommend this License -principally for works whose purpose is instruction or reference. - -@sp 1 -@item -APPLICABILITY AND DEFINITIONS - -This License applies to any manual or other work that contains a -notice placed by the copyright holder saying it can be distributed -under the terms of this License. The ``Document'', below, refers to any -such manual or work. Any member of the public is a licensee, and is -addressed as ``you''. - -A ``Modified Version'' of the Document means any work containing the -Document or a portion of it, either copied verbatim, or with -modifications and/or translated into another language. - -A ``Secondary Section'' is a named appendix or a front-matter section of -the Document that deals exclusively with the relationship of the -publishers or authors of the Document to the Document's overall subject -(or to related matters) and contains nothing that could fall directly -within that overall subject. (For example, if the Document is in part a -textbook of mathematics, a Secondary Section may not explain any -mathematics.) The relationship could be a matter of historical -connection with the subject or with related matters, or of legal, -commercial, philosophical, ethical or political position regarding -them. - -The ``Invariant Sections'' are certain Secondary Sections whose titles -are designated, as being those of Invariant Sections, in the notice -that says that the Document is released under this License. - -The ``Cover Texts'' are certain short passages of text that are listed, -as Front-Cover Texts or Back-Cover Texts, in the notice that says that -the Document is released under this License. - -A ``Transparent'' copy of the Document means a machine-readable copy, -represented in a format whose specification is available to the -general public, whose contents can be viewed and edited directly and -straightforwardly with generic text editors or (for images composed of -pixels) generic paint programs or (for drawings) some widely available -drawing editor, and that is suitable for input to text formatters or -for automatic translation to a variety of formats suitable for input -to text formatters. A copy made in an otherwise Transparent file -format whose markup has been designed to thwart or discourage -subsequent modification by readers is not Transparent. A copy that is -not ``Transparent'' is called ``Opaque''. - -Examples of suitable formats for Transparent copies include plain -ASCII without markup, Texinfo input format, LaTeX input format, SGML -or XML using a publicly available DTD, and standard-conforming simple -HTML designed for human modification. Opaque formats include -PostScript, PDF, proprietary formats that can be read and edited only -by proprietary word processors, SGML or XML for which the DTD and/or -processing tools are not generally available, and the -machine-generated HTML produced by some word processors for output -purposes only. - -The ``Title Page'' means, for a printed book, the title page itself, -plus such following pages as are needed to hold, legibly, the material -this License requires to appear in the title page. For works in -formats which do not have any title page as such, ``Title Page'' means -the text near the most prominent appearance of the work's title, -preceding the beginning of the body of the text. -@sp 1 -@item -VERBATIM COPYING - -You may copy and distribute the Document in any medium, either -commercially or noncommercially, provided that this License, the -copyright notices, and the license notice saying this License applies -to the Document are reproduced in all copies, and that you add no other -conditions whatsoever to those of this License. You may not use -technical measures to obstruct or control the reading or further -copying of the copies you make or distribute. However, you may accept -compensation in exchange for copies. If you distribute a large enough -number of copies you must also follow the conditions in section 3. - -You may also lend copies, under the same conditions stated above, and -you may publicly display copies. -@sp 1 -@item -COPYING IN QUANTITY - -If you publish printed copies of the Document numbering more than 100, -and the Document's license notice requires Cover Texts, you must enclose -the copies in covers that carry, clearly and legibly, all these Cover -Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on -the back cover. Both covers must also clearly and legibly identify -you as the publisher of these copies. The front cover must present -the full title with all words of the title equally prominent and -visible. You may add other material on the covers in addition. -Copying with changes limited to the covers, as long as they preserve -the title of the Document and satisfy these conditions, can be treated -as verbatim copying in other respects. - -If the required texts for either cover are too voluminous to fit -legibly, you should put the first ones listed (as many as fit -reasonably) on the actual cover, and continue the rest onto adjacent -pages. - -If you publish or distribute Opaque copies of the Document numbering -more than 100, you must either include a machine-readable Transparent -copy along with each Opaque copy, or state in or with each Opaque copy -a publicly-accessible computer-network location containing a complete -Transparent copy of the Document, free of added material, which the -general network-using public has access to download anonymously at no -charge using public-standard network protocols. If you use the latter -option, you must take reasonably prudent steps, when you begin -distribution of Opaque copies in quantity, to ensure that this -Transparent copy will remain thus accessible at the stated location -until at least one year after the last time you distribute an Opaque -copy (directly or through your agents or retailers) of that edition to -the public. - -It is requested, but not required, that you contact the authors of the -Document well before redistributing any large number of copies, to give -them a chance to provide you with an updated version of the Document. -@sp 1 -@item -MODIFICATIONS - -You may copy and distribute a Modified Version of the Document under -the conditions of sections 2 and 3 above, provided that you release -the Modified Version under precisely this License, with the Modified -Version filling the role of the Document, thus licensing distribution -and modification of the Modified Version to whoever possesses a copy -of it. In addition, you must do these things in the Modified Version: - -@enumerate A -@item -Use in the Title Page (and on the covers, if any) a title distinct -from that of the Document, and from those of previous versions -(which should, if there were any, be listed in the History section -of the Document). You may use the same title as a previous version -if the original publisher of that version gives permission. - -@item -List on the Title Page, as authors, one or more persons or entities -responsible for authorship of the modifications in the Modified -Version, together with at least five of the principal authors of the -Document (all of its principal authors, if it has less than five). - -@item -State on the Title page the name of the publisher of the -Modified Version, as the publisher. - -@item -Preserve all the copyright notices of the Document. - -@item -Add an appropriate copyright notice for your modifications -adjacent to the other copyright notices. - -@item -Include, immediately after the copyright notices, a license notice -giving the public permission to use the Modified Version under the -terms of this License, in the form shown in the Addendum below. - -@item -Preserve in that license notice the full lists of Invariant Sections -and required Cover Texts given in the Document's license notice. - -@item -Include an unaltered copy of this License. - -@item -Preserve the section entitled ``History'', and its title, and add to -it an item stating at least the title, year, new authors, and -publisher of the Modified Version as given on the Title Page. If -there is no section entitled ``History'' in the Document, create one -stating the title, year, authors, and publisher of the Document as -given on its Title Page, then add an item describing the Modified -Version as stated in the previous sentence. - -@item -Preserve the network location, if any, given in the Document for -public access to a Transparent copy of the Document, and likewise -the network locations given in the Document for previous versions -it was based on. These may be placed in the ``History'' section. -You may omit a network location for a work that was published at -least four years before the Document itself, or if the original -publisher of the version it refers to gives permission. - -@item -In any section entitled ``Acknowledgements'' or ``Dedications'', -preserve the section's title, and preserve in the section all the -substance and tone of each of the contributor acknowledgements -and/or dedications given therein. - -@item -Preserve all the Invariant Sections of the Document, -unaltered in their text and in their titles. Section numbers -or the equivalent are not considered part of the section titles. - -@item -Delete any section entitled ``Endorsements''. Such a section -may not be included in the Modified Version. - -@item -Do not retitle any existing section as ``Endorsements'' -or to conflict in title with any Invariant Section. -@end enumerate - -If the Modified Version includes new front-matter sections or -appendices that qualify as Secondary Sections and contain no material -copied from the Document, you may at your option designate some or all -of these sections as invariant. To do this, add their titles to the -list of Invariant Sections in the Modified Version's license notice. -These titles must be distinct from any other section titles. - -You may add a section entitled ``Endorsements'', provided it contains -nothing but endorsements of your Modified Version by various -parties--for example, statements of peer review or that the text has -been approved by an organization as the authoritative definition of a -standard. - -You may add a passage of up to five words as a Front-Cover Text, and a -passage of up to 25 words as a Back-Cover Text, to the end of the list -of Cover Texts in the Modified Version. Only one passage of -Front-Cover Text and one of Back-Cover Text may be added by (or -through arrangements made by) any one entity. If the Document already -includes a cover text for the same cover, previously added by you or -by arrangement made by the same entity you are acting on behalf of, -you may not add another; but you may replace the old one, on explicit -permission from the previous publisher that added the old one. - -The author(s) and publisher(s) of the Document do not by this License -give permission to use their names for publicity for or to assert or -imply endorsement of any Modified Version. -@sp 1 -@item -COMBINING DOCUMENTS - -You may combine the Document with other documents released under this -License, under the terms defined in section 4 above for modified -versions, provided that you include in the combination all of the -Invariant Sections of all of the original documents, unmodified, and -list them all as Invariant Sections of your combined work in its -license notice. - -The combined work need only contain one copy of this License, and -multiple identical Invariant Sections may be replaced with a single -copy. If there are multiple Invariant Sections with the same name but -different contents, make the title of each such section unique by -adding at the end of it, in parentheses, the name of the original -author or publisher of that section if known, or else a unique number. -Make the same adjustment to the section titles in the list of -Invariant Sections in the license notice of the combined work. - -In the combination, you must combine any sections entitled ``History'' -in the various original documents, forming one section entitled -``History''; likewise combine any sections entitled ``Acknowledgements'', -and any sections entitled ``Dedications''. You must delete all sections -entitled ``Endorsements.'' -@sp 1 -@item -COLLECTIONS OF DOCUMENTS - -You may make a collection consisting of the Document and other documents -released under this License, and replace the individual copies of this -License in the various documents with a single copy that is included in -the collection, provided that you follow the rules of this License for -verbatim copying of each of the documents in all other respects. - -You may extract a single document from such a collection, and distribute -it individually under this License, provided you insert a copy of this -License into the extracted document, and follow this License in all -other respects regarding verbatim copying of that document. -@sp 1 -@item -AGGREGATION WITH INDEPENDENT WORKS - -A compilation of the Document or its derivatives with other separate -and independent documents or works, in or on a volume of a storage or -distribution medium, does not as a whole count as a Modified Version -of the Document, provided no compilation copyright is claimed for the -compilation. Such a compilation is called an ``aggregate'', and this -License does not apply to the other self-contained works thus compiled -with the Document, on account of their being thus compiled, if they -are not themselves derivative works of the Document. - -If the Cover Text requirement of section 3 is applicable to these -copies of the Document, then if the Document is less than one quarter -of the entire aggregate, the Document's Cover Texts may be placed on -covers that surround only the Document within the aggregate. -Otherwise they must appear on covers around the whole aggregate. -@sp 1 -@item -TRANSLATION - -Translation is considered a kind of modification, so you may -distribute translations of the Document under the terms of section 4. -Replacing Invariant Sections with translations requires special -permission from their copyright holders, but you may include -translations of some or all Invariant Sections in addition to the -original versions of these Invariant Sections. You may include a -translation of this License provided that you also include the -original English version of this License. In case of a disagreement -between the translation and the original English version of this -License, the original English version will prevail. -@sp 1 -@item -TERMINATION - -You may not copy, modify, sublicense, or distribute the Document except -as expressly provided for under this License. Any other attempt to -copy, modify, sublicense or distribute the Document is void, and will -automatically terminate your rights under this License. However, -parties who have received copies, or rights, from you under this -License will not have their licenses terminated so long as such -parties remain in full compliance. -@sp 1 -@item -FUTURE REVISIONS OF THIS LICENSE - -The Free Software Foundation may publish new, revised versions -of the GNU Free Documentation License from time to time. Such new -versions will be similar in spirit to the present version, but may -differ in detail to address new problems or concerns. See -@uref{http://www.gnu.org/copyleft/}. - -Each version of the License is given a distinguishing version number. -If the Document specifies that a particular numbered version of this -License ``or any later version'' applies to it, you have the option of -following the terms and conditions either of that specified version or -of any later version that has been published (not as a draft) by the -Free Software Foundation. If the Document does not specify a version -number of this License, you may choose any version ever published (not -as a draft) by the Free Software Foundation. - -@end enumerate - -@c fakenode --- for prepinfo -@unnumberedsec ADDENDUM: How to use this License for your documents - -To use this License in a document you have written, include a copy of -the License in the document and put the following copyright and -license notices just after the title page: - -@smallexample -@group - - Copyright (C) @var{year} @var{your name}. - Permission is granted to copy, distribute and/or modify this document - under the terms of the GNU Free Documentation License, Version 1.1 - or any later version published by the Free Software Foundation; - with the Invariant Sections being @var{list their titles}, with the - Front-Cover Texts being @var{list}, and with the Back-Cover Texts being @var{list}. - A copy of the license is included in the section entitled ``GNU - Free Documentation License''. -@end group -@end smallexample -If you have no Invariant Sections, write ``with no Invariant Sections'' -instead of saying which ones are invariant. If you have no -Front-Cover Texts, write ``no Front-Cover Texts'' instead of -``Front-Cover Texts being @var{list}''; likewise for Back-Cover Texts. - -If your document contains nontrivial examples of program code, we -recommend releasing these examples in parallel under your choice of -free software license, such as the GNU General Public License, -to permit their use in free software. - -@node Index, , GNU Free Documentation License, Top -@comment node-name, next, previous, up - -@unnumbered Index -@printindex cp -@bye - -Conventions: -1. Functions, built-in or otherwise, do NOT have () after them. -2. Gawk built-in vars and functions are in @code. Also program vars and - functions. -3. HTTP method names are in @code. -4. Protocols such as echo, ftp, etc are in @samp. -5. URLs are in @url. -6. All RFC's in the index. Put a space between `RFC' and the number. |
