summaryrefslogtreecommitdiffstats
path: root/bin/sh/expand.c
Commit message (Collapse)AuthorAgeFilesLines
* MFC r278818: sh: Add stsavestr(), like savestr() but allocates usingjilles2016-04-091-4/+1
| | | | stalloc().
* MFC r288430: wordexp: Rewrite to make WRDE_NOCMD reliable.jilles2015-10-251-0/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Shell syntax is too complicated to detect command substitution and unquoted operators reliably without implementing much of sh's parser. Therefore, have sh do this detection. While changing sh's support anyway, also read input from a pipe instead of arguments to avoid {ARG_MAX} limits and improve privacy, and output count and length using 16 instead of 8 digits. The basic concept is: execl("/bin/sh", "sh", "-c", "freebsd_wordexp ${1:+\"$1\"} -f "$2", "", flags & WRDE_NOCMD ? "-p" : "", <pipe with words>); The WRDE_BADCHAR error is still implemented in libc. POSIX requires us to fail strings containing unquoted braces with code WRDE_BADCHAR. Since this is normally not a syntax error in sh, there is still a need for checking code in libc, we_check(). The new we_check() is an optimistic check that all the characters <newline> | & ; < > ( ) { } are quoted. To avoid duplicating too much sh logic, such characters are permitted when quoting characters are seen, even if the quoting characters may themselves be quoted. This code reports all WRDE_BADCHAR errors; bad characters that get past it and are a syntax error in sh return WRDE_SYNTAX. Although many implementations of WRDE_NOCMD erroneously allow some command substitutions (and ours even documented this), there appears to be code that relies on its security (codesearch.debian.net shows quite a few uses). Passing untrusted data to wordexp() still exposes a denial of service possibility and a fairly large attack surface. This is also a MFC of r286830 to reduce conflicts. I changed the code somewhat to avoid changes from r286941; in particular, WRDE_BADVAL can still only be returned if WRDE_UNDEF was passed. Relnotes: yes Security: fixes command execution with wordexp(untrusted, WRDE_NOCMD)
* MFC r287148: sh: Fix out of bounds read when there is no ] after a [:class:].jilles2015-09-131-12/+7
| | | | | | | | | The initial check for a matching ] was incorrect if a ] may be consumed by a [:class:]. The subsequent loop assumed that there must be a ]. Remove the initial check and make the loop cope with a missing ]. Found with afl-fuzz.
* MFC r278826,279330: sh: Various cleanups to expand.c:jilles2015-09-131-60/+55
| | | | | | | | * Remove some gotos. * Remove unused parameter. * Remove duplicate code. This MFC is to avoid conflicts in the MFC of r287148.
* MFC r272575: sh: Eliminate some gotos.jilles2015-09-131-10/+5
| | | | This MFC is to avoid conflicts in the MFC of r287148.
* MFC r276037: sh: Remove EXP_REDIR.jilles2014-12-291-12/+6
| | | | | | | | EXP_REDIR was supposed to generate pathnames in redirection if exactly one file matches, as permitted but not required by POSIX in interactive mode. It is unlikely this will be implemented. No functional change is intended.
* MFC r274854: sh: Use DQSYNTAX only while expanding, not SQSYNTAX.jilles2014-12-291-1/+1
| | | | | Quoting during expansion only cares about CCTL, which is the same for DQSYNTAX and SQSYNTAX.
* MFC r273920: sh: Fix corruption of CTL* bytes in positional parameters injilles2014-12-291-1/+1
| | | | | | | redirection. EXP_REDIR was not being checked for while expanding positional parameters in redirection, so CTL* bytes were not being prefixed where they should be.
* MFC r268576: sh: Correctly handle positional parameters beyond INT_MAX onjilles2014-12-141-2/+4
| | | | | | | | | | | | | | | | | 64-bit systems. Currently, there can be no more than INT_MAX positional parameters. Make sure to treat all higher ones as unset to avoid incorrect results and crashes. On 64-bit systems, our atoi() takes the low 32 bits of the strtol() and sign-extends them. On 32-bit systems, the call to atoi() returned INT_MAX for too high values and there is not enough address space for so many positional parameters, so there was no issue. PR: 195918
* MFC r263777: sh: Fix possible memory leaks and double frees with unexpectedjilles2014-04-141-0/+3
| | | | SIGINT.
* MFC r263195: sh: Add some consts.jilles2014-04-051-5/+5
|
* MFC r262755: sh: Make argstr() return where it stopped and simplify expari()jilles2014-04-051-21/+5
| | | | using this.
* Sync sh(1) in stable/10 to head.jmmv2014-03-091-55/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a MFC of all the commits listed below. My original goal of this change was to only merge the move of the tests from tools/regression/bin/ into the new layout (which include tests for sh(1) and other tools as well). However, doing so is tricky due to the ongoing work in sh(1) and, especially, the many changes to its tests since stable/10 was first branched. Merging everything is the simplest way to achieve this goal and, as a bonus point, we get various fixes and miscellaneous improvements into the branch. Per jilles' suggestion, I'm avoiding the merge of a couple of changes (r256850 and r257506) that required depending kernel changes. I'm also avoiding very recent changes that have not had a long enough time to be validated in current. This is "make tinderbox" clean. r256735 sh: Remove one syscall when waiting for a foreground job. r257399 sh: Allow trapping SIGINT/SIGQUIT after ignore because of '&'. r257504 sh: Reorder union node to reduce its size on 64-bit platforms. r257920 sh: Add a test case for would-be assignments that are not due to quoting. r257929 sh: Properly quote alias output from command -v. r258489 sh: Add tests for the </dev/null implicit in a background command. r258533 sh: Add more tests for the </dev/null implicit in a background command. r258535 sh: Make <&0 disable the </dev/null implicit in a background command. r258776 sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro. r259047 sh: Split set -x output into a separate function. r259210 Migrate tools/regression/bin/ tests to the new layout. r259844 sh: Remove an unused variable. r259846 sh: Initialize OPTIND=1 even if it came from the environment. r259874 sh: Simplify code related to PPID variable. r259946 sh: Don't check input for non-whitespace if history is disabled. r260246 sh(1): Discourage use of -e. r260506 Run the sh(1) and test(1) tests as unprivileged. r260586 Mark the bin/pax tests as requiring perl. r260634 Use TAP_TESTS_PERL to register the legacy_test in bin/pax. r260635 Replace hand-crafted Kyuafiles with automatic generation. r260654 sh: Remove SIGWINCH handler and just check for resize before every read. r261121 sh: Add test for nested alias. r261125 sh: Solve the alias recursion problem in a less hackish way. r261141 sh: Do not depend on parse/execute split in new alias test. r261160 sh: Add tests for alias names after another alias. r261192 sh: Allow aliases to force alias substitution on the following word. r262533 sh: Make expari() static. r262565 sh: Do not corrupt internal representation if LINENO inner expansion fails. r262697 sh: Simplify expari(). Reviewed by: jilles
* sh: Fix various compiler warnings.jilles2013-04-011-6/+7
| | | | | | | It now passes WARNS=7 with clang on i386. GCC 4.2.1 does not understand setjmp() properly so will always trigger -Wuninitialized. I will not add the volatile keywords to suppress this.
* sh: Expand here documents in the current process.jilles2013-02-031-13/+0
| | | | | | | | | | | | | | | | | Expand here documents at the same point other redirections are expanded but use a non-fork subshell environment (like simple command substitutions) for compatibility. Substitition errors result in an empty here document like before. As a result, a fork is avoided for short (<4K) expanded here documents. Unexpanded here documents (with quoted end marker after <<) are not affected by this change. They already only forked when >4K. Side effects: * Order of expansion is slightly different. * Slow expansions are not executed in parallel with the redirected command. * A non-fork subshell environment is subtly different from a forked process.
* sh: Make various functions static.jilles2012-01-011-1/+2
|
* sh: Make patmatch() non-recursive.jilles2012-01-011-37/+58
|
* sh: Use dirent.d_type in pathname generation.jilles2011-12-281-0/+4
| | | | | | This improves performance for globs where a slash or another component follows a component with metacharacters by eliminating unnecessary attempts to open directories that are not.
* sh: Cache de->d_namlen in a local variable.jilles2011-12-281-6/+8
|
* sh: Add support for named character classes in bracket expressions.jilles2011-06-151-1/+37
| | | | | Example: case x in [[:alpha:]]) echo yes ;; esac
* sh: Fix duplicate prototypes for builtins.jilles2011-06-131-0/+1
| | | | | | Have mkbuiltins write the prototypes for the *cmd functions to builtins.h instead of builtins.c and include builtins.h in more .c files instead of duplicating prototypes for *cmd functions in other headers.
* sh: Save/restore changed variables in optimized command substitution.jilles2011-06-121-72/+0
| | | | | | | | | | | In optimized command substitution, save and restore any variables changed by expansions (${var=value} and $((var=assigned))), instead of trying to determine if an expansion may cause such changes. If $! is referenced in optimized command substitution, do not cause jobs to be remembered longer. This fixes $(jobs $!) again, simplifies the man page and shortens the code.
* sh: Fix locale-dependent ranges in bracket expressions.jilles2011-06-121-4/+4
| | | | | | | | When I added UTF-8 support in r221646, the LC_COLLATE-based ordering broke because of sign extension of char. Because of libc restrictions, this does not work for UTF-8. For UTF-8 locales, ranges always use character code order.
* sh: Do parameter expansion before printing PS4 (set -x).jilles2011-06-091-0/+1
| | | | | | | | | | | | | | | | The function name expandstr() and the general idea of doing this kind of expansion by treating the text as a here document without end marker is from dash. All variants of parameter expansion and arithmetic expansion also work (the latter is not required by POSIX but it does not take extra code and many other shells also allow it). Command substitution is prevented because I think it causes too much code to be re-entered (for example creating an unbounded recursion of trace lines). Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would be quite useful.
* sh: Fix unquoted $@/$* if IFS=''.jilles2011-05-271-2/+5
| | | | | | If IFS is null, unquoted $@/$* should still expand to separate words. This differs from quoted $@ (which does not depend on IFS) in that pathname generation is performed and empty words are removed.
* sh: Add UTF-8 support to pattern matching.jilles2011-05-081-12/+50
| | | | | | | | | | ?, [...] patterns match codepoints instead of bytes. They do not match invalid sequences. [...] patterns must not contain invalid sequences otherwise they will not match anything. This is so that ${var#?} removes the first codepoint, not the first byte, without putting UTF-8 knowledge into the ${var#pattern} code. However, * continues to match any string and an invalid sequence matches an identical invalid sequence. (This differs from fnmatch(3).)
* sh: Add UTF-8 support to ${#var}.jilles2011-05-071-3/+13
| | | | | If the current locale uses UTF-8, ${#var} counts codepoints (more precisely, bytes b with (b & 0xc0) != 0x80).
* Fix typos - remove duplicate "the".brucec2011-02-211-1/+1
| | | | | | PR: bin/154928 Submitted by: Eitan Adler <lists at eitanadler.com> MFC after: 3 days
* sh: Remove comment mentioning herefd, which is gone.jilles2011-02-021-2/+1
|
* sh: Don't do optimized command substitution if expansions have side effects.jilles2010-12-281-0/+72
| | | | | | | | | | | | | | | | | | | | | | | | | Before considering to execute a command substitution in the same process, check if any of the expansions may have a side effect; if so, execute it in a new process just like happens if it is not a single simple command. Although the check happens at run time, it is a static check that does not depend on current state. It is triggered by: - expanding $! (which may cause the job to be remembered) - ${var=value} default value assignment - assignment operators in arithmetic - parameter substitutions in arithmetic except ${#param}, $$, $# and $? - command substitutions in arithmetic This means that $((v+1)) does not prevent optimized command substitution, whereas $(($v+1)) does, because $v might expand to something containing assignment operators. Scripts should not depend on these exact details for correctness. It is also imaginable to have the shell fork if and when a side effect is encountered or to create a new temporary namespace for variables. Due to the $! change, the construct $(jobs $!) no longer works. The value of $! should be stored in a variable outside command substitution first.
* sh: Allow arbitrary large numbers in CHECKSTRSPACE.jilles2010-12-261-3/+4
| | | | | | Reduce "stack string" API somewhat and simplify code. Add a check for integer overflow of the "stack string" length (probably incomplete).
* Remove dead code.uqs2010-12-181-2/+0
| | | | | | | | c is assigned 0 and *loc is pointing to NULL, so c!=0 cannot be true, and dereferencing loc would be a bad idea anyway. Coverity Prevent: CID 5113 Reviewed by: jilles
* sh: Fix corruption of command substitutions with special chars after newlinejilles2010-12-161-2/+2
| | | | | | | | The CTLESC byte to protect a special character was output before instead of after a newline directly preceding the special character. The special handling of newlines is because command substitutions discard all trailing newlines.
* sh: Remove the herefd hack.jilles2010-12-121-8/+0
| | | | | | | | | | | | | | | | | | The herefd hack wrote out partial here documents while expanding them. It seems unnecessary complication given that other expansions just allocate memory. It causes bugs because the stack is also used for intermediate results such as arithmetic expressions. Such places should disable herefd for the duration but not all of them do, and I prefer removing the need for disabling herefd to disabling it everywhere needed. Here documents larger than 1024 bytes will use a bit more CPU time and memory. Additionally this allows a later change to expand here documents in the current shell environment. (This is faster for small here documents but also changes behaviour.) Obtained from: dash
* sh: Replace some macros and repeated code in expand.c with functions.jilles2010-12-111-31/+32
| | | | | No functional change is intended, but the binary is about 1K smaller on i386.
* sh: Code size optimizations to "stack string" memory allocation:jilles2010-11-231-9/+8
| | | | | | | | * Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC. * Add STPUTS macro (based on function) and use it instead of loops that add nul-terminated strings to the stack string. No functional change is intended, but code size is about 1K less on i386.
* sh: Code size optimizations to buffered output.jilles2010-11-201-4/+2
| | | | | | This is mainly less use of the outc macro. No functional change is intended, but code size is about 2K less on i386.
* sh: Fix some issues with CTL* bytes and ${var#pat}.jilles2010-10-291-14/+13
| | | | | | | | | | | | | | subevalvar() incorrectly assumed that CTLESC bytes were present iff the expansion was quoted. However, they are present iff various processing such as word splitting is to be done later on. Example: v=@$e@$e@$e@ y="${v##*"$e"}" echo "$y" failed if $e contained the magic CTLESC byte. Exp-run done by: pav (with some other sh(1) changes)
* sh: Do IFS splitting on word in ${v+word} and ${v-word}.jilles2010-10-291-10/+27
| | | | | | | | | | | | | | | | | The code is inspired by NetBSD sh somewhat, but different because we preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if v is set, but generates filenames otherwise. Note that this is the only place where we split text literally from the script (the similar ${v=word} assigns to v and then expands $v). The parser must now add additional markers to allow the expansion code to know whether arbitrary characters in substitutions are quoted. Example: for i in ${$+a b c}; do echo $i; done Exp-run done by: pav (with some other sh(1) changes)
* In the spirit of r90111, depend on c89 and remove the "STATIC" macroobrien2010-10-131-35/+35
| | | | and its usage.
* Make DEBUG traces 64-bit clean:jhb2010-10-131-7/+8
| | | | | | | | | - Use %t to print ptrdiff_t values. - Cast a ptrdiff_t value explicitly to int for a field width specifier. While here, sort includes. Submitted by: Garrett Cooper
* Consistently use "STATIC" for all functions in order to be able to setobrien2010-10-131-6/+6
| | | | | | breakpoints with in a debugger. And use naked "static" for variables. Noticed by: bde
* sh: Improve comments in expand.c.jilles2010-09-051-21/+37
|
* sh: Remove remnants of '!!' to negate pattern.jilles2010-08-221-17/+2
| | | | | | | | This Almquist extension was disabled long ago. In pathname generation, components starting with '!!' were treated as containing wildcards, causing unnecessary readdir (which could fail, causing pathname generation to fail while it should not).
* sh: Fix heap-based buffer overflow in pathname generation.jilles2010-08-101-16/+15
| | | | | | | | | | | | | | The buffer for generated pathnames could be too small in some cases. It happened to be always at least PATH_MAX long, so there was never an overflow if the resulting pathnames would be usable. This bug may be abused if a script subjects input from an untrusted source to pathname generation, which a bad idea anyhow. Most shell scripts do not work on untrusted data. secteam@ says no advisory is necessary. PR: bin/148733 Reported by: Changming Sun snnn119 at gmail com MFC after: 10 days
* sh: Forget about terminated background processes sooner.jilles2010-06-291-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | Unless $! has been referenced for a particular job or $! still contains that job's pid, forget about it after it has terminated. If $! has been referenced, remember the job until the wait builtin has reported its completion (either with the pid as parameter or without parameters). In interactive mode, jobs are forgotten after termination has been reported, which happens before primary prompts and through the jobs builtin. Even then, though, remember a job if $! has been referenced. This is similar to what is suggested by POSIX and should fix most memory leaks (which also tend to cause sh to use more CPU time) with long running scripts that start background jobs. Caveats: * Repeatedly referencing $! without ever doing 'wait', like while :; do foo & echo started foo: $!; sleep 60; done will still use a lot of memory and CPU time in the long run. * The jobs and jobid builtins do not cause a job to be remembered for longer like expanding $! does. PR: bin/55346
* sh: Fix pathname expansion with quoted slashes like *\/.jilles2010-05-111-9/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash. Because this is the first code I'm importing from dash to expand.c, add the Herbert Xu copyright notice which is in dash's expand.c. When pathname expanding *\/, the CTLESC representing the quoted state was erroneously taken as part of the * pathname component. This CTLESC was then seen by the pattern matching code as escaping the '\0' terminating the string. The code is slightly different because dash converts the CTLESC characters to backslashes and removes all the other CTL* characters to allow substituting glob(3). The effect of the bug was also slightly different from dash (where nothing matched at all). Because a CTLESC can escape a '\0' in some way, whether files were included despite the bug depended on memory that should not be read. In particular, on many machines /*\/ expanded to a strict subset of what /*/ expanded to. Example: echo /*"/null" This should print /dev/null, not /*/null. PR: bin/146378 Obtained from: dash
* sh: Use stalloc for arith variable names.jilles2010-04-251-1/+3
| | | | | This is simpler than the custom memory tracker I added earlier, and is also needed by the dash arith code I plan to import.
* sh: Do tilde expansion in substitutions.jilles2010-04-031-3/+3
| | | | | | | | | | | | | This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word}, ${v##word} (which does not inherit any quoting). In all cases tilde expansion is only attempted at the start of word, even if word contains spaces. This agrees with POSIX and other shells. This is the last part of the patch tested in the exp-run. Exp-run done by: erwin (with some other sh(1) changes)
* sh: Allow quoting pattern match characters in ${v%pat} and ${v#pat}.jilles2010-04-031-1/+3
| | | | | | | | Note that this depends on r206145 for allowing pattern match characters to have their special meaning inside a double-quoted expansion like "${v%pat}". PR: bin/117748 Exp-run done by: erwin (with some other sh(1) changes)
OpenPOWER on IntegriCloud