1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
|
TODO file for GNU ptx - last revised 05 November 1993.
Copyright (C) 1992, 1993 Free Software Foundation, Inc.
Francois Pinard <pinard@iro.umontreal.ca>, 1992.
The following are more or less in decreasing order of priority.
* Use rx instead of regex.
* Correct the infinite loop using -S '$' or -S '^'.
* Use mmap for swallowing files (maybe wrong when memory edited).
* Understand and mimic `-t' option, if I can.
* Sort keywords intelligently for Latin-1 code. See how to interface
this character set with various output formats. Also, introduce
options to inverse-sort and possibly to reverse-sort.
* Improve speed for Ignore and Only tables. Consider hashing instead
of sorting. Consider playing with obstacks to digest them.
* Provide better handling of format effectors obtained from input, and
also attempt white space compression on output which would still
maximize full output width usage.
* See how TeX mode could be made more useful, and if a texinfo mode
would mean something to someone.
* Provide multiple language support
Most of the boosting work should go along the line of fast recognition
of multiple and complex boundaries, which define various `languages'.
Each such language has its own rules for words, sentences, paragraphs,
and reporting requests. This is less difficult than I first thought:
. Recognize language modifiers with each option. At least -b, -i, -o,
-W, -S, and also new language switcher options, will have such
modifiers. Modifiers on language switchers will allow or disallow
language transitions.
. Complete the transformation of underlying variables into arrays in
the code.
. Implement a heap of positions in the input file. There is one entry
in the heap for each compiled regexp; it is initialized by a re_search
after each regexp compile. Regexps reschedule themselves in the heap
when their position passes while scanning input. In this way, looking
simultaneously for a lot of regexps should not be too inefficient,
once the scanning starts. If this works ok, maybe consider accepting
regexps in Only and Ignore tables.
. Merge with language processing boundary processing options, really
integrating -S processing as a special case. Maybe, implement several
level of boundaries. See how to implement a stack of languages, for
handling quotations. See if more sophisticated references could be
handled as another special case of a language.
* Tackle other aspects, in a more long term view
. Add options for statistics, frequency lists, referencing, and all
other prescreening tools and subsidiary tasks of concordance
production.
. Develop an interactive mode. Even better, construct a GNU emacs
interface. I'm looking at Gene Myers <gene@cs.arizona.edu> suffix
arrays as a possible implementation along those ideas.
. Implement hooks so word classification and tagging should be merged
in. See how to effectively hook in lemmatisation or other
morphological features. It is far from being clear by now how to
interface this correctly, so some experimentation is mandatory.
. Profile and speed up the whole thing.
. Make it work on small address space machines. Consider three levels
of hugeness for files, and three corresponding algorithms to make
optimal use of memory. The first case is when all the input files and
all the word references fit in memory: this is the case currently
implemented. The second case is when the files cannot fit all together
in memory, but the word references do. The third case is when even
the word references cannot fit in memory.
. There also are subsidiary developments for in-core incremental sort
routines as well as for external sort packages. The need for more
flexible sort packages comes partly from the fact that linguists use
kinds of keys which compare in unusual and more sophisticated ways.
GNU `sort' and `ptx' could evolve together.
Local Variables:
mode: outline
outline-regexp: " *[-+*.] \\| "
eval: (hide-body)
End:
|
