| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
| |
stalloc().
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Shell syntax is too complicated to detect command substitution and unquoted
operators reliably without implementing much of sh's parser. Therefore, have
sh do this detection.
While changing sh's support anyway, also read input from a pipe instead of
arguments to avoid {ARG_MAX} limits and improve privacy, and output count
and length using 16 instead of 8 digits.
The basic concept is:
execl("/bin/sh", "sh", "-c", "freebsd_wordexp ${1:+\"$1\"} -f "$2",
"", flags & WRDE_NOCMD ? "-p" : "", <pipe with words>);
The WRDE_BADCHAR error is still implemented in libc. POSIX requires us to
fail strings containing unquoted braces with code WRDE_BADCHAR. Since this
is normally not a syntax error in sh, there is still a need for checking
code in libc, we_check().
The new we_check() is an optimistic check that all the characters
<newline> | & ; < > ( ) { }
are quoted. To avoid duplicating too much sh logic, such characters are
permitted when quoting characters are seen, even if the quoting characters
may themselves be quoted. This code reports all WRDE_BADCHAR errors; bad
characters that get past it and are a syntax error in sh return WRDE_SYNTAX.
Although many implementations of WRDE_NOCMD erroneously allow some command
substitutions (and ours even documented this), there appears to be code that
relies on its security (codesearch.debian.net shows quite a few uses).
Passing untrusted data to wordexp() still exposes a denial of service
possibility and a fairly large attack surface.
This is also a MFC of r286830 to reduce conflicts. I changed the code
somewhat to avoid changes from r286941; in particular, WRDE_BADVAL can still
only be returned if WRDE_UNDEF was passed.
Relnotes: yes
Security: fixes command execution with wordexp(untrusted, WRDE_NOCMD)
|
|
|
|
|
|
|
|
|
| |
The initial check for a matching ] was incorrect if a ] may be consumed by a
[:class:]. The subsequent loop assumed that there must be a ].
Remove the initial check and make the loop cope with a missing ].
Found with afl-fuzz.
|
|
|
|
|
|
|
|
| |
* Remove some gotos.
* Remove unused parameter.
* Remove duplicate code.
This MFC is to avoid conflicts in the MFC of r287148.
|
|
|
|
| |
This MFC is to avoid conflicts in the MFC of r287148.
|
|
|
|
|
|
|
|
| |
EXP_REDIR was supposed to generate pathnames in redirection if exactly one
file matches, as permitted but not required by POSIX in interactive mode. It
is unlikely this will be implemented.
No functional change is intended.
|
|
|
|
|
| |
Quoting during expansion only cares about CCTL, which is the same for
DQSYNTAX and SQSYNTAX.
|
|
|
|
|
|
|
| |
redirection.
EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
64-bit systems.
Currently, there can be no more than INT_MAX positional parameters. Make
sure to treat all higher ones as unset to avoid incorrect results and
crashes.
On 64-bit systems, our atoi() takes the low 32 bits of the strtol() and
sign-extends them.
On 32-bit systems, the call to atoi() returned INT_MAX for too high values
and there is not enough address space for so many positional parameters, so
there was no issue.
PR: 195918
|
|
|
|
| |
SIGINT.
|
| |
|
|
|
|
| |
using this.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a MFC of all the commits listed below.
My original goal of this change was to only merge the move of the tests
from tools/regression/bin/ into the new layout (which include tests for
sh(1) and other tools as well). However, doing so is tricky due to the
ongoing work in sh(1) and, especially, the many changes to its tests
since stable/10 was first branched.
Merging everything is the simplest way to achieve this goal and, as a
bonus point, we get various fixes and miscellaneous improvements into
the branch.
Per jilles' suggestion, I'm avoiding the merge of a couple of changes
(r256850 and r257506) that required depending kernel changes. I'm also
avoiding very recent changes that have not had a long enough time to be
validated in current.
This is "make tinderbox" clean.
r256735 sh: Remove one syscall when waiting for a foreground job.
r257399 sh: Allow trapping SIGINT/SIGQUIT after ignore because of '&'.
r257504 sh: Reorder union node to reduce its size on 64-bit platforms.
r257920 sh: Add a test case for would-be assignments that are not due to quoting.
r257929 sh: Properly quote alias output from command -v.
r258489 sh: Add tests for the </dev/null implicit in a background command.
r258533 sh: Add more tests for the </dev/null implicit in a background command.
r258535 sh: Make <&0 disable the </dev/null implicit in a background command.
r258776 sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.
r259047 sh: Split set -x output into a separate function.
r259210 Migrate tools/regression/bin/ tests to the new layout.
r259844 sh: Remove an unused variable.
r259846 sh: Initialize OPTIND=1 even if it came from the environment.
r259874 sh: Simplify code related to PPID variable.
r259946 sh: Don't check input for non-whitespace if history is disabled.
r260246 sh(1): Discourage use of -e.
r260506 Run the sh(1) and test(1) tests as unprivileged.
r260586 Mark the bin/pax tests as requiring perl.
r260634 Use TAP_TESTS_PERL to register the legacy_test in bin/pax.
r260635 Replace hand-crafted Kyuafiles with automatic generation.
r260654 sh: Remove SIGWINCH handler and just check for resize before every read.
r261121 sh: Add test for nested alias.
r261125 sh: Solve the alias recursion problem in a less hackish way.
r261141 sh: Do not depend on parse/execute split in new alias test.
r261160 sh: Add tests for alias names after another alias.
r261192 sh: Allow aliases to force alias substitution on the following word.
r262533 sh: Make expari() static.
r262565 sh: Do not corrupt internal representation if LINENO inner expansion fails.
r262697 sh: Simplify expari().
Reviewed by: jilles
|
|
|
|
|
|
|
| |
It now passes WARNS=7 with clang on i386.
GCC 4.2.1 does not understand setjmp() properly so will always trigger
-Wuninitialized. I will not add the volatile keywords to suppress this.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Expand here documents at the same point other redirections are expanded but
use a non-fork subshell environment (like simple command substitutions) for
compatibility. Substitition errors result in an empty here document like
before.
As a result, a fork is avoided for short (<4K) expanded here documents.
Unexpanded here documents (with quoted end marker after <<) are not affected
by this change. They already only forked when >4K.
Side effects:
* Order of expansion is slightly different.
* Slow expansions are not executed in parallel with the redirected command.
* A non-fork subshell environment is subtly different from a forked process.
|
| |
|
| |
|
|
|
|
|
|
| |
This improves performance for globs where a slash or another component
follows a component with metacharacters by eliminating unnecessary attempts
to open directories that are not.
|
| |
|
|
|
|
|
| |
Example:
case x in [[:alpha:]]) echo yes ;; esac
|
|
|
|
|
|
| |
Have mkbuiltins write the prototypes for the *cmd functions to builtins.h
instead of builtins.c and include builtins.h in more .c files instead of
duplicating prototypes for *cmd functions in other headers.
|
|
|
|
|
|
|
|
|
|
|
| |
In optimized command substitution, save and restore any variables changed by
expansions (${var=value} and $((var=assigned))), instead of trying to
determine if an expansion may cause such changes.
If $! is referenced in optimized command substitution, do not cause jobs to
be remembered longer.
This fixes $(jobs $!) again, simplifies the man page and shortens the code.
|
|
|
|
|
|
|
|
| |
When I added UTF-8 support in r221646, the LC_COLLATE-based ordering broke
because of sign extension of char.
Because of libc restrictions, this does not work for UTF-8. For UTF-8
locales, ranges always use character code order.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.
All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).
Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).
Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.
|
|
|
|
|
|
| |
If IFS is null, unquoted $@/$* should still expand to separate words.
This differs from quoted $@ (which does not depend on IFS) in that pathname
generation is performed and empty words are removed.
|
|
|
|
|
|
|
|
|
|
| |
?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)
|
|
|
|
|
| |
If the current locale uses UTF-8, ${#var} counts codepoints (more precisely,
bytes b with (b & 0xc0) != 0x80).
|
|
|
|
|
|
| |
PR: bin/154928
Submitted by: Eitan Adler <lists at eitanadler.com>
MFC after: 3 days
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.
Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic
This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.
Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.
Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.
|
|
|
|
|
|
| |
Reduce "stack string" API somewhat and simplify code.
Add a check for integer overflow of the "stack string" length (probably
incomplete).
|
|
|
|
|
|
|
|
| |
c is assigned 0 and *loc is pointing to NULL, so c!=0 cannot be true,
and dereferencing loc would be a bad idea anyway.
Coverity Prevent: CID 5113
Reviewed by: jilles
|
|
|
|
|
|
|
|
| |
The CTLESC byte to protect a special character was output before instead of
after a newline directly preceding the special character.
The special handling of newlines is because command substitutions discard
all trailing newlines.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The herefd hack wrote out partial here documents while expanding them. It
seems unnecessary complication given that other expansions just allocate
memory. It causes bugs because the stack is also used for intermediate
results such as arithmetic expressions. Such places should disable herefd
for the duration but not all of them do, and I prefer removing the need for
disabling herefd to disabling it everywhere needed.
Here documents larger than 1024 bytes will use a bit more CPU time and
memory.
Additionally this allows a later change to expand here documents in the
current shell environment. (This is faster for small here documents but also
changes behaviour.)
Obtained from: dash
|
|
|
|
|
| |
No functional change is intended, but the binary is about 1K smaller on
i386.
|
|
|
|
|
|
|
|
| |
* Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC.
* Add STPUTS macro (based on function) and use it instead of loops that add
nul-terminated strings to the stack string.
No functional change is intended, but code size is about 1K less on i386.
|
|
|
|
|
|
| |
This is mainly less use of the outc macro.
No functional change is intended, but code size is about 2K less on i386.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
subevalvar() incorrectly assumed that CTLESC bytes were present iff the
expansion was quoted. However, they are present iff various processing such
as word splitting is to be done later on.
Example:
v=@$e@$e@$e@
y="${v##*"$e"}"
echo "$y"
failed if $e contained the magic CTLESC byte.
Exp-run done by: pav (with some other sh(1) changes)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.
Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.
Example:
for i in ${$+a b c}; do echo $i; done
Exp-run done by: pav (with some other sh(1) changes)
|
|
|
|
| |
and its usage.
|
|
|
|
|
|
|
|
|
| |
- Use %t to print ptrdiff_t values.
- Cast a ptrdiff_t value explicitly to int for a field width specifier.
While here, sort includes.
Submitted by: Garrett Cooper
|
|
|
|
|
|
| |
breakpoints with in a debugger. And use naked "static" for variables.
Noticed by: bde
|
| |
|
|
|
|
|
|
|
|
| |
This Almquist extension was disabled long ago.
In pathname generation, components starting with '!!' were treated as
containing wildcards, causing unnecessary readdir (which could fail, causing
pathname generation to fail while it should not).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The buffer for generated pathnames could be too small in some cases. It
happened to be always at least PATH_MAX long, so there was never an overflow
if the resulting pathnames would be usable.
This bug may be abused if a script subjects input from an untrusted source
to pathname generation, which a bad idea anyhow. Most shell scripts do not
work on untrusted data. secteam@ says no advisory is necessary.
PR: bin/148733
Reported by: Changming Sun snnn119 at gmail com
MFC after: 10 days
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Unless $! has been referenced for a particular job or $! still contains that
job's pid, forget about it after it has terminated. If $! has been
referenced, remember the job until the wait builtin has reported its
completion (either with the pid as parameter or without parameters).
In interactive mode, jobs are forgotten after termination has been reported,
which happens before primary prompts and through the jobs builtin. Even
then, though, remember a job if $! has been referenced.
This is similar to what is suggested by POSIX and should fix most memory
leaks (which also tend to cause sh to use more CPU time) with long running
scripts that start background jobs.
Caveats:
* Repeatedly referencing $! without ever doing 'wait', like
while :; do foo & echo started foo: $!; sleep 60; done
will still use a lot of memory and CPU time in the long run.
* The jobs and jobid builtins do not cause a job to be remembered for longer
like expanding $! does.
PR: bin/55346
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.
Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.
When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.
The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).
The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.
Example:
echo /*"/null"
This should print /dev/null, not /*/null.
PR: bin/146378
Obtained from: dash
|
|
|
|
|
| |
This is simpler than the custom memory tracker I added earlier, and is also
needed by the dash arith code I plan to import.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which
inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word},
${v##word} (which does not inherit any quoting).
In all cases tilde expansion is only attempted at the start of word, even if
word contains spaces. This agrees with POSIX and other shells.
This is the last part of the patch tested in the exp-run.
Exp-run done by: erwin (with some other sh(1) changes)
|
|
|
|
|
|
|
|
| |
Note that this depends on r206145 for allowing pattern match characters to
have their special meaning inside a double-quoted expansion like "${v%pat}".
PR: bin/117748
Exp-run done by: erwin (with some other sh(1) changes)
|