diff options
Diffstat (limited to 'usr.bin/lex/flex.1')
| -rw-r--r-- | usr.bin/lex/flex.1 | 1001 |
1 files changed, 0 insertions, 1001 deletions
diff --git a/usr.bin/lex/flex.1 b/usr.bin/lex/flex.1 deleted file mode 100644 index 6aba4d6..0000000 --- a/usr.bin/lex/flex.1 +++ /dev/null @@ -1,1001 +0,0 @@ -.TH FLEX 1 "November 1993" "Version 2.4" -.SH NAME -flex \- fast lexical analyzer generator -.SH SYNOPSIS -.B flex -.B [\-bcdfhilnpstvwBFILTV78+ \-C[aefFmr] \-Pprefix \-Sskeleton] -.I [filename ...] -.SH DESCRIPTION -.I flex -is a tool for generating -.I scanners: -programs which recognized lexical patterns in text. -.I flex -reads -the given input files, or its standard input if no file names are given, -for a description of a scanner to generate. The description is in -the form of pairs -of regular expressions and C code, called -.I rules. flex -generates as output a C source file, -.B lex.yy.c, -which defines a routine -.B yylex(). -This file is compiled and linked with the -.B \-lfl -library to produce an executable. When the executable is run, -it analyzes its input for occurrences -of the regular expressions. Whenever it finds one, it executes -the corresponding C code. -.PP -For full documentation, see -.B flexdoc(1). -This manual entry is intended for use as a quick reference. -.SH OPTIONS -.I flex -has the following options: -.TP -.B \-b -generate backing-up information to -.I lex.backup. -This is a list of scanner states which require backing up and the input -characters on which they do so. By adding rules one can remove -backing-up states. If all backing-up states are eliminated and -.B \-Cf -or -.B \-CF -is used, the generated scanner will run faster. -.TP -.B \-c -is a do-nothing, deprecated option included for POSIX compliance. -.IP -.B NOTE: -in previous releases of -.I flex -.B \-c -specified table-compression options. This functionality is -now given by the -.B \-C -flag. To ease the the impact of this change, when -.I flex -encounters -.B \-c, -it currently issues a warning message and assumes that -.B \-C -was desired instead. In the future this "promotion" of -.B \-c -to -.B \-C -will go away in the name of full POSIX compliance (unless -the POSIX meaning is removed first). -.TP -.B \-d -makes the generated scanner run in -.I debug -mode. Whenever a pattern is recognized and the global -.B yy_flex_debug -is non-zero (which is the default), the scanner will -write to -.I stderr -a line of the form: -.nf - - --accepting rule at line 53 ("the matched text") - -.fi -The line number refers to the location of the rule in the file -defining the scanner (i.e., the file that was fed to flex). Messages -are also generated when the scanner backs up, accepts the -default rule, reaches the end of its input buffer (or encounters -a NUL; the two look the same as far as the scanner's concerned), -or reaches an end-of-file. -.TP -.B \-f -specifies -.I fast scanner. -No table compression is done and stdio is bypassed. -The result is large but fast. This option is equivalent to -.B \-Cfr -(see below). -.TP -.B \-h -generates a "help" summary of -.I flex's -options to -.I stderr -and then exits. -.TP -.B \-i -instructs -.I flex -to generate a -.I case-insensitive -scanner. The case of letters given in the -.I flex -input patterns will -be ignored, and tokens in the input will be matched regardless of case. The -matched text given in -.I yytext -will have the preserved case (i.e., it will not be folded). -.TP -.B \-l -turns on maximum compatibility with the original AT&T lex implementation, -at a considerable performance cost. This option is incompatible with -.B \-+, \-f, \-F, \-Cf, -or -.B \-CF. -See -.I flexdoc(1) -for details. -.TP -.B \-n -is another do-nothing, deprecated option included only for -POSIX compliance. -.TP -.B \-p -generates a performance report to stderr. The report -consists of comments regarding features of the -.I flex -input file which will cause a loss of performance in the resulting scanner. -If you give the flag twice, you will also get comments regarding -features that lead to minor performance losses. -.TP -.B \-s -causes the -.I default rule -(that unmatched scanner input is echoed to -.I stdout) -to be suppressed. If the scanner encounters input that does not -match any of its rules, it aborts with an error. -.TP -.B \-t -instructs -.I flex -to write the scanner it generates to standard output instead -of -.B lex.yy.c. -.TP -.B \-v -specifies that -.I flex -should write to -.I stderr -a summary of statistics regarding the scanner it generates. -.TP -.B \-w -suppresses warning messages. -.TP -.B \-B -instructs -.I flex -to generate a -.I batch -scanner instead of an -.I interactive -scanner (see -.B \-I -below). See -.I flexdoc(1) -for details. Scanners using -.B \-Cf -or -.B \-CF -compression options automatically specify this option, too. -.TP -.B \-F -specifies that the -.ul -fast -scanner table representation should be used (and stdio bypassed). -This representation is about as fast as the full table representation -.B (-f), -and for some sets of patterns will be considerably smaller (and for -others, larger). It cannot be used with the -.B \-+ -option. See -.B flexdoc(1) -for more details. -.IP -This option is equivalent to -.B \-CFr -(see below). -.TP -.B \-I -instructs -.I flex -to generate an -.I interactive -scanner, that is, a scanner which stops immediately rather than -looking ahead if it knows -that the currently scanned text cannot be part of a longer rule's match. -This is the opposite of -.I batch -scanners (see -.B \-B -above). See -.B flexdoc(1) -for details. -.IP -Note, -.B \-I -cannot be used in conjunction with -.I full -or -.I fast tables, -i.e., the -.B \-f, \-F, \-Cf, -or -.B \-CF -flags. For other table compression options, -.B \-I -is the default. -.TP -.B \-L -instructs -.I flex -not to generate -.B #line -directives in -.B lex.yy.c. -The default is to generate such directives so error -messages in the actions will be correctly -located with respect to the original -.I flex -input file, and not to -the fairly meaningless line numbers of -.B lex.yy.c. -.TP -.B \-T -makes -.I flex -run in -.I trace -mode. It will generate a lot of messages to -.I stderr -concerning -the form of the input and the resultant non-deterministic and deterministic -finite automata. This option is mostly for use in maintaining -.I flex. -.TP -.B \-V -prints the version number to -.I stderr -and exits. -.TP -.B \-7 -instructs -.I flex -to generate a 7-bit scanner, which can save considerable table space, -especially when using -.B \-Cf -or -.B \-CF -(and, at most sites, -.B \-7 -is on by default for these options. To see if this is the case, use the -.B -v -verbose flag and check the flag summary it reports). -.TP -.B \-8 -instructs -.I flex -to generate an 8-bit scanner. This is the default except for the -.B \-Cf -and -.B \-CF -compression options, for which the default is site-dependent, and -can be checked by inspecting the flag summary generated by the -.B \-v -option. -.TP -.B \-+ -specifies that you want flex to generate a C++ -scanner class. See the section on Generating C++ Scanners in -.I flexdoc(1) -for details. -.TP -.B \-C[aefFmr] -controls the degree of table compression and scanner optimization. -.IP -.B \-Ca -trade off larger tables in the generated scanner for faster performance -because the elements of the tables are better aligned for memory access -and computation. This option can double the size of the tables used by -your scanner. -.IP -.B \-Ce -directs -.I flex -to construct -.I equivalence classes, -i.e., sets of characters -which have identical lexical properties. -Equivalence classes usually give -dramatic reductions in the final table/object file sizes (typically -a factor of 2-5) and are pretty cheap performance-wise (one array -look-up per character scanned). -.IP -.B \-Cf -specifies that the -.I full -scanner tables should be generated - -.I flex -should not compress the -tables by taking advantages of similar transition functions for -different states. -.IP -.B \-CF -specifies that the alternate fast scanner representation (described in -.B flexdoc(1)) -should be used. This option cannot be used with -.B \-+. -.IP -.B \-Cm -directs -.I flex -to construct -.I meta-equivalence classes, -which are sets of equivalence classes (or characters, if equivalence -classes are not being used) that are commonly used together. Meta-equivalence -classes are often a big win when using compressed tables, but they -have a moderate performance impact (one or two "if" tests and one -array look-up per character scanned). -.IP -.B \-Cr -causes the generated scanner to -.I bypass -using stdio for input. In general this option results in a minor -performance gain only worthwhile if used in conjunction with -.B \-Cf -or -.B \-CF. -It can cause surprising behavior if you use stdio yourself to -read from -.I yyin -prior to calling the scanner. -.IP -A lone -.B \-C -specifies that the scanner tables should be compressed but neither -equivalence classes nor meta-equivalence classes should be used. -.IP -The options -.B \-Cf -or -.B \-CF -and -.B \-Cm -do not make sense together - there is no opportunity for meta-equivalence -classes if the table is not being compressed. Otherwise the options -may be freely mixed. -.IP -The default setting is -.B \-Cem, -which specifies that -.I flex -should generate equivalence classes -and meta-equivalence classes. This setting provides the highest -degree of table compression. You can trade off -faster-executing scanners at the cost of larger tables with -the following generally being true: -.nf - - slowest & smallest - -Cem - -Cm - -Ce - -C - -C{f,F}e - -C{f,F} - -C{f,F}a - fastest & largest - -.fi -.IP -.B \-C -options are cumulative. -.TP -.B \-Pprefix -changes the default -.I "yy" -prefix used by -.I flex -to be -.I prefix -instead. See -.I flexdoc(1) -for a description of all the global variables and file names that -this affects. -.TP -.B \-Sskeleton_file -overrides the default skeleton file from which -.I flex -constructs its scanners. You'll never need this option unless you are doing -.I flex -maintenance or development. -.SH SUMMARY OF FLEX REGULAR EXPRESSIONS -The patterns in the input are written using an extended set of regular -expressions. These are: -.nf - - x match the character 'x' - . any character except newline - [xyz] a "character class"; in this case, the pattern - matches either an 'x', a 'y', or a 'z' - [abj-oZ] a "character class" with a range in it; matches - an 'a', a 'b', any letter from 'j' through 'o', - or a 'Z' - [^A-Z] a "negated character class", i.e., any character - but those in the class. In this case, any - character EXCEPT an uppercase letter. - [^A-Z\\n] any character EXCEPT an uppercase letter or - a newline - r* zero or more r's, where r is any regular expression - r+ one or more r's - r? zero or one r's (that is, "an optional r") - r{2,5} anywhere from two to five r's - r{2,} two or more r's - r{4} exactly 4 r's - {name} the expansion of the "name" definition - (see above) - "[xyz]\\"foo" - the literal string: [xyz]"foo - \\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v', - then the ANSI-C interpretation of \\x. - Otherwise, a literal 'X' (used to escape - operators such as '*') - \\123 the character with octal value 123 - \\x2a the character with hexadecimal value 2a - (r) match an r; parentheses are used to override - precedence (see below) - - - rs the regular expression r followed by the - regular expression s; called "concatenation" - - - r|s either an r or an s - - - r/s an r but only if it is followed by an s. The - s is not part of the matched text. This type - of pattern is called as "trailing context". - ^r an r, but only at the beginning of a line - r$ an r, but only at the end of a line. Equivalent - to "r/\\n". - - - <s>r an r, but only in start condition s (see - below for discussion of start conditions) - <s1,s2,s3>r - same, but in any of start conditions s1, - s2, or s3 - <*>r an r in any start condition, even an exclusive one. - - - <<EOF>> an end-of-file - <s1,s2><<EOF>> - an end-of-file when in start condition s1 or s2 - -.fi -The regular expressions listed above are grouped according to -precedence, from highest precedence at the top to lowest at the bottom. -Those grouped together have equal precedence. -.PP -Some notes on patterns: -.IP - -Negated character classes -.I match newlines -unless "\\n" (or an equivalent escape sequence) is one of the -characters explicitly present in the negated character class -(e.g., "[^A-Z\\n]"). -.IP - -A rule can have at most one instance of trailing context (the '/' operator -or the '$' operator). The start condition, '^', and "<<EOF>>" patterns -can only occur at the beginning of a pattern, and, as well as with '/' and '$', -cannot be grouped inside parentheses. The following are all illegal: -.nf - - foo/bar$ - foo|(bar$) - foo|^bar - <sc1>foo<sc2>bar - -.fi -.SH SUMMARY OF SPECIAL ACTIONS -In addition to arbitrary C code, the following can appear in actions: -.IP - -.B ECHO -copies yytext to the scanner's output. -.IP - -.B BEGIN -followed by the name of a start condition places the scanner in the -corresponding start condition. -.IP - -.B REJECT -directs the scanner to proceed on to the "second best" rule which matched the -input (or a prefix of the input). -.B yytext -and -.B yyleng -are set up appropriately. Note that -.B REJECT -is a particularly expensive feature in terms scanner performance; -if it is used in -.I any -of the scanner's actions it will slow down -.I all -of the scanner's matching. Furthermore, -.B REJECT -cannot be used with the -.B \-f -or -.B \-F -options. -.IP -Note also that unlike the other special actions, -.B REJECT -is a -.I branch; -code immediately following it in the action will -.I not -be executed. -.IP - -.B yymore() -tells the scanner that the next time it matches a rule, the corresponding -token should be -.I appended -onto the current value of -.B yytext -rather than replacing it. -.IP - -.B yyless(n) -returns all but the first -.I n -characters of the current token back to the input stream, where they -will be rescanned when the scanner looks for the next match. -.B yytext -and -.B yyleng -are adjusted appropriately (e.g., -.B yyleng -will now be equal to -.I n -). -.IP - -.B unput(c) -puts the character -.I c -back onto the input stream. It will be the next character scanned. -.IP - -.B input() -reads the next character from the input stream (this routine is called -.B yyinput() -if the scanner is compiled using -.B C++). -.IP - -.B yyterminate() -can be used in lieu of a return statement in an action. It terminates -the scanner and returns a 0 to the scanner's caller, indicating "all done". -.IP -By default, -.B yyterminate() -is also called when an end-of-file is encountered. It is a macro and -may be redefined. -.IP - -.B YY_NEW_FILE -is an action available only in <<EOF>> rules. It means "Okay, I've -set up a new input file, continue scanning". It is no longer required; -you can just assign -.I yyin -to point to a new file in the <<EOF>> action. -.IP - -.B yy_create_buffer( file, size ) -takes a -.I FILE -pointer and an integer -.I size. -It returns a YY_BUFFER_STATE -handle to a new input buffer large enough to accomodate -.I size -characters and associated with the given file. When in doubt, use -.B YY_BUF_SIZE -for the size. -.IP - -.B yy_switch_to_buffer( new_buffer ) -switches the scanner's processing to scan for tokens from -the given buffer, which must be a YY_BUFFER_STATE. -.IP - -.B yy_delete_buffer( buffer ) -deletes the given buffer. -.SH VALUES AVAILABLE TO THE USER -.IP - -.B char *yytext -holds the text of the current token. It may be modified but not lengthened -(you cannot append characters to the end). Modifying the last character -may affect the activity of rules anchored using '^' during the next scan; -see -.B flexdoc(1) -for details. -.IP -If the special directive -.B %array -appears in the first section of the scanner description, then -.B yytext -is instead declared -.B char yytext[YYLMAX], -where -.B YYLMAX -is a macro definition that you can redefine in the first section -if you don't like the default value (generally 8KB). Using -.B %array -results in somewhat slower scanners, but the value of -.B yytext -becomes immune to calls to -.I input() -and -.I unput(), -which potentially destroy its value when -.B yytext -is a character pointer. The opposite of -.B %array -is -.B %pointer, -which is the default. -.IP -You cannot use -.B %array -when generating C++ scanner classes -(the -.B \-+ -flag). -.IP - -.B int yyleng -holds the length of the current token. -.IP - -.B FILE *yyin -is the file which by default -.I flex -reads from. It may be redefined but doing so only makes sense before -scanning begins or after an EOF has been encountered. Changing it in -the midst of scanning will have unexpected results since -.I flex -buffers its input; use -.B yyrestart() -instead. -Once scanning terminates because an end-of-file -has been seen, -.B -you can assign -.I yyin -at the new input file and then call the scanner again to continue scanning. -.IP - -.B void yyrestart( FILE *new_file ) -may be called to point -.I yyin -at the new input file. The switch-over to the new file is immediate -(any previously buffered-up input is lost). Note that calling -.B yyrestart() -with -.I yyin -as an argument thus throws away the current input buffer and continues -scanning the same input file. -.IP - -.B FILE *yyout -is the file to which -.B ECHO -actions are done. It can be reassigned by the user. -.IP - -.B YY_CURRENT_BUFFER -returns a -.B YY_BUFFER_STATE -handle to the current buffer. -.IP - -.B YY_START -returns an integer value corresponding to the current start -condition. You can subsequently use this value with -.B BEGIN -to return to that start condition. -.SH MACROS AND FUNCTIONS YOU CAN REDEFINE -.IP - -.B YY_DECL -controls how the scanning routine is declared. -By default, it is "int yylex()", or, if prototypes are being -used, "int yylex(void)". This definition may be changed by redefining -the "YY_DECL" macro. Note that -if you give arguments to the scanning routine using a -K&R-style/non-prototyped function declaration, you must terminate -the definition with a semi-colon (;). -.IP - -The nature of how the scanner -gets its input can be controlled by redefining the -.B YY_INPUT -macro. -YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its -action is to place up to -.I max_size -characters in the character array -.I buf -and return in the integer variable -.I result -either the -number of characters read or the constant YY_NULL (0 on Unix systems) -to indicate EOF. The default YY_INPUT reads from the -global file-pointer "yyin". -A sample redefinition of YY_INPUT (in the definitions -section of the input file): -.nf - - %{ - #undef YY_INPUT - #define YY_INPUT(buf,result,max_size) \\ - { \\ - int c = getchar(); \\ - result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\ - } - %} - -.fi -.IP - -When the scanner receives an end-of-file indication from YY_INPUT, -it then checks the function -.B yywrap() -function. If -.B yywrap() -returns false (zero), then it is assumed that the -function has gone ahead and set up -.I yyin -to point to another input file, and scanning continues. If it returns -true (non-zero), then the scanner terminates, returning 0 to its -caller. -.IP -The default -.B yywrap() -always returns 1. -.IP - -YY_USER_ACTION -can be redefined to provide an action -which is always executed prior to the matched rule's action. -.IP - -The macro -.B YY_USER_INIT -may be redefined to provide an action which is always executed before -the first scan. -.IP - -In the generated scanner, the actions are all gathered in one large -switch statement and separated using -.B YY_BREAK, -which may be redefined. By default, it is simply a "break", to separate -each rule's action from the following rule's. -.SH FILES -.TP -.B \-lfl -library with which to link scanners to obtain the default versions -of -.I yywrap() -and/or -.I main(). -.TP -.I lex.yy.c -generated scanner (called -.I lexyy.c -on some systems). -.TP -.I lex.yy.cc -generated C++ scanner class, when using -.B -+. -.TP -.I <FlexLexer.h> -header file defining the C++ scanner base class, -.B FlexLexer, -and its derived class, -.B yyFlexLexer. -.TP -.I flex.skl -skeleton scanner. This file is only used when building flex, not when -flex executes. -.TP -.I lex.backup -backing-up information for -.B \-b -flag (called -.I lex.bck -on some systems). -.SH "SEE ALSO" -.PP -flexdoc(1), lex(1), yacc(1), sed(1), awk(1). -.PP -M. E. Lesk and E. Schmidt, -.I LEX \- Lexical Analyzer Generator -.SH DIAGNOSTICS -.PP -.I reject_used_but_not_detected undefined -or -.PP -.I yymore_used_but_not_detected undefined - -These errors can occur at compile time. They indicate that the -scanner uses -.B REJECT -or -.B yymore() -but that -.I flex -failed to notice the fact, meaning that -.I flex -scanned the first two sections looking for occurrences of these actions -and failed to find any, but somehow you snuck some in (via a #include -file, for example). Make an explicit reference to the action in your -.I flex -input file. (Note that previously -.I flex -supported a -.B %used/%unused -mechanism for dealing with this problem; this feature is still supported -but now deprecated, and will go away soon unless the author hears from -people who can argue compellingly that they need it.) -.PP -.I flex scanner jammed - -a scanner compiled with -.B \-s -has encountered an input string which wasn't matched by -any of its rules. -.PP -.I warning, rule cannot be matched -indicates that the given rule -cannot be matched because it follows other rules that will -always match the same text as it. See -.I flexdoc(1) -for an example. -.PP -.I warning, -.B \-s -.I -option given but default rule can be matched -means that it is possible (perhaps only in a particular start condition) -that the default rule (match any single character) is the only one -that will match a particular input. Since -.PP -.I scanner input buffer overflowed - -a scanner rule matched more text than the available dynamic memory. -.PP -.I token too large, exceeds YYLMAX - -your scanner uses -.B %array -and one of its rules matched a string longer than the -.B YYLMAX -constant (8K bytes by default). You can increase the value by -#define'ing -.B YYLMAX -in the definitions section of your -.I flex -input. -.PP -.I scanner requires \-8 flag to -.I use the character 'x' - -Your scanner specification includes recognizing the 8-bit character -.I 'x' -and you did not specify the \-8 flag, and your scanner defaulted to 7-bit -because you used the -.B \-Cf -or -.B \-CF -table compression options. -.PP -.I flex scanner push-back overflow - -you used -.B unput() -to push back so much text that the scanner's buffer could not hold -both the pushed-back text and the current token in -.B yytext. -Ideally the scanner should dynamically resize the buffer in this case, but at -present it does not. -.PP -.I -input buffer overflow, can't enlarge buffer because scanner uses REJECT - -the scanner was working on matching an extremely large token and needed -to expand the input buffer. This doesn't work with scanners that use -.B -REJECT. -.PP -.I -fatal flex scanner internal error--end of buffer missed - -This can occur in an scanner which is reentered after a long-jump -has jumped out (or over) the scanner's activation frame. Before -reentering the scanner, use: -.nf - - yyrestart( yyin ); - -.fi -or use C++ scanner classes (the -.B \-+ -option), which are fully reentrant. -.SH AUTHOR -Vern Paxson, with the help of many ideas and much inspiration from -Van Jacobson. Original version by Jef Poskanzer. -.PP -See flexdoc(1) for additional credits and the address to send comments to. -.SH DEFICIENCIES / BUGS -.PP -Some trailing context -patterns cannot be properly matched and generate -warning messages ("dangerous trailing context"). These are -patterns where the ending of the -first part of the rule matches the beginning of the second -part, such as "zx*/xy*", where the 'x*' matches the 'x' at -the beginning of the trailing context. (Note that the POSIX draft -states that the text matched by such patterns is undefined.) -.PP -For some trailing context rules, parts which are actually fixed-length are -not recognized as such, leading to the abovementioned performance loss. -In particular, parts using '|' or {n} (such as "foo{3}") are always -considered variable-length. -.PP -Combining trailing context with the special '|' action can result in -.I fixed -trailing context being turned into the more expensive -.I variable -trailing context. For example, in the following: -.nf - - %% - abc | - xyz/def - -.fi -.PP -Use of -.B unput() -or -.B input() -invalidates yytext and yyleng, unless the -.B %array -directive -or the -.B \-l -option has been used. -.PP -Use of unput() to push back more text than was matched can -result in the pushed-back text matching a beginning-of-line ('^') -rule even though it didn't come at the beginning of the line -(though this is rare!). -.PP -Pattern-matching of NUL's is substantially slower than matching other -characters. -.PP -Dynamic resizing of the input buffer is slow, as it entails rescanning -all the text matched so far by the current (generally huge) token. -.PP -.I flex -does not generate correct #line directives for code internal -to the scanner; thus, bugs in -.I flex.skl -yield bogus line numbers. -.PP -Due to both buffering of input and read-ahead, you cannot intermix -calls to <stdio.h> routines, such as, for example, -.B getchar(), -with -.I flex -rules and expect it to work. Call -.B input() -instead. -.PP -The total table entries listed by the -.B \-v -flag excludes the number of table entries needed to determine -what rule has been matched. The number of entries is equal -to the number of DFA states if the scanner does not use -.B REJECT, -and somewhat greater than the number of states if it does. -.PP -.B REJECT -cannot be used with the -.B \-f -or -.B \-F -options. -.PP -The -.I flex -internal algorithms need documentation. |
